Multi-task cross-subject electroencephalogram emotion recognition method and device, and storage medium
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- INST OF AUTOMATION CHINESE ACAD OF SCI
- Filing Date
- 2024-05-30
- Publication Date
- 2026-08-07
AI Technical Summary
[0005]本申请实施例提供一种多任务跨被试脑电情绪识别方法、装置及存储介质，用以解决现有技术的情绪识别模型在跨被试的情绪识别中准确性低的技术问题
[0020]本申请实施例还提供一种计算机程序产品，包括计算机程序，所述计算机程序被处理器执行时实现如上述任一种所述多任务跨被试脑电情绪识别方法。
Smart Images
Figure CN118673302B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of electroencephalogram (EEG) signal processing technology, and in particular to a multi-task cross-subject EEG emotion recognition method, device, and storage medium. Background Technology
[0002] Emotion recognition plays a vital role in human-computer interaction, disease assessment, product design, virtual reality, and other fields, and has received increasing attention from the academic community in recent years, becoming an important research topic. Due to the advantages of electroencephalography (EEG) signals, such as high temporal resolution, resistance to spoofing, high reliability, and ease of acquisition, EEG-based emotion recognition is gaining popularity among researchers.
[0003] Early EEG-based emotion recognition models mostly employed machine learning methods, with performance largely dependent on the quality of manually extracted features. With the development of deep learning, more and more research is using deep learning models such as Convolutional Neural Networks (CNNs), Long Short-Term Memory Networks (LSTMs), and Capsule Networks (CapNet). Some works also combine the local perception capabilities of CNNs with the temporal information extraction capabilities of LSTMs to further improve emotion recognition performance.
[0004] However, due to the influence of acquisition equipment, experimental design, and subjects, EEG signals often exhibit significant differences between different subjects and different datasets. This necessitates that EEG-based emotion recognition models possess strong capabilities in extracting emotion-related features and generalizing across different subjects and datasets. While most current work has achieved high performance on subject-dependent datasets (training and testing data come from the same subject), significant challenges remain in cross-subject (using data from some subjects as the training set and data from others as the testing set) and cross-dataset emotion recognition. Summary of the Invention
[0005] This application provides a multi-task cross-subject EEG emotion recognition method, device, and storage medium to solve the technical problem of low accuracy of existing emotion recognition models in cross-subject emotion recognition.
[0006] This application provides a multi-task cross-subject EEG emotion recognition method, including: Four-dimensional input data is acquired, which is determined based on time period, waveband, electrode position and raw EEG signal, which is acquired based on multiple different EEG acquisition devices; Feature extraction is performed on the four-dimensional input data based on a multilayer perceptron, and a general EEG representation is obtained based on self-attention and cross-attention. The general EEG representation is determined based on the primary EEG representation. The target features in the general EEG representation are determined based on the target task, and the target features are used to determine the prediction results of emotion recognition.
[0007] In some embodiments, obtaining the four-dimensional input data includes: The original EEG signal is divided into multiple bands based on the filter, and the differential entropy (DE) feature of each band is obtained to determine the two-dimensional input data. The correspondence between the DE features of each band and the electrode positions is mapped onto the two-dimensional input data to obtain the three-dimensional input data; The three-dimensional input data is arranged chronologically based on time periods to obtain four-dimensional input data.
[0008] In some embodiments, the step of extracting features from the four-dimensional input data based on a multilayer perceptron and obtaining a general EEG representation based on self-attention and cross-attention includes: Based on a multilayer perceptron, feature extraction is performed on the four-dimensional input data to obtain a primary EEG representation. General EEG representations are obtained from the primary EEG representations based on self-attention and cross-attention.
[0009] In some embodiments, the primary EEG representation is used to map the raw EEG signals to the same latent space.
[0010] In some embodiments, the step of obtaining a general EEG representation from the primary EEG representation based on self-attention and cross-attention includes: Obtain learnable embedding vectors; The general EEG representation is obtained based on the self-attention of the learnable embedding vector and the cross-attention between the learnable embedding vector and the primary EEG representation.
[0011] In some embodiments, determining the target features in the general EEG representation based on the target task includes: using a multilayer perceptron corresponding to the target task as a decoder, training it based on minimizing the cross-entropy loss function, and obtaining the target features.
[0012] Secondly, embodiments of this application also provide a multi-task cross-subject EEG emotion recognition device, comprising: The first acquisition module is used to acquire four-dimensional input data, which is determined based on time period, waveband, electrode position and raw EEG signal, and the raw EEG signal is acquired based on multiple different EEG acquisition devices; The second acquisition module is used to extract features from the four-dimensional input data based on a multilayer perceptron, and to acquire a general EEG representation based on self-attention and cross-attention, wherein the general EEG representation is determined based on the primary EEG representation. The first determining module is used to determine the target features in the general EEG representation based on the target task, and the target features are used to determine the prediction result of emotion recognition.
[0013] In some embodiments, the first acquisition module includes: The first acquisition submodule is used to divide the original EEG signal into multiple bands based on a filter, acquire the differential entropy (DE) feature of each band, and determine the two-dimensional input data. The first processing submodule is used to map the correspondence between the DE features of each band and the electrode positions to the two-dimensional input data to obtain three-dimensional input data; The second acquisition submodule is used to arrange the three-dimensional input data in a time sequence based on a time period to acquire four-dimensional input data.
[0014] In some embodiments, the second acquisition module includes: The third acquisition submodule is used to extract features from the four-dimensional input data based on a multilayer perceptron to acquire a primary EEG representation. The fourth acquisition submodule is used to acquire general EEG representations from the primary EEG representations based on self-attention and cross-attention.
[0015] In some embodiments, the primary EEG representation is used to map the raw EEG signals to the same latent space.
[0016] In some embodiments, the fourth acquisition submodule includes: The first acquisition unit is used to acquire learnable embedding vectors; The second acquisition unit is used to acquire the general EEG representation based on the self-attention of the learnable embedding vector and the cross-attention of the learnable embedding vector and the primary EEG representation.
[0017] In some embodiments, the first determining module includes: The fifth acquisition submodule is used to use the multilayer perceptron corresponding to the target task as a decoder, and train it based on minimizing the cross-entropy loss function to acquire the target features.
[0018] This application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the multi-task cross-subject EEG emotion recognition method as described above.
[0019] This application also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the multi-task cross-subject EEG emotion recognition method as described above.
[0020] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the multi-task cross-subject EEG emotion recognition method as described above.
[0021] The multi-task cross-subject EEG emotion recognition method, device, and storage medium provided in this application extract features from four-dimensional input data and obtain general EEG representations based on self-attention and cross-attention. Then, target features in the general EEG representations are determined according to the target task, thereby obtaining prediction results of emotion recognition based on the target features, which can improve the accuracy of cross-subject emotion recognition. Attached Figure Description
[0022] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.
[0023] Figure 1 This is a flowchart illustrating the multi-task cross-subject EEG emotion recognition method provided in the embodiments of this application.
[0024] Figure 2 This is a framework diagram of the multi-task cross-subject EEG emotion recognition method provided in the embodiments of this application.
[0025] Figure 3 This is a diagram illustrating the four-dimensional EEG feature construction process provided in the embodiments of this application.
[0026] Figure 4 This is a model structure diagram of the multi-task cross-subject EEG emotion recognition method provided in the embodiments of this application.
[0027] Figure 5 This is a schematic diagram of the structure of the multi-task cross-subject EEG emotion recognition device provided in the embodiments of this application.
[0028] Figure 6 A schematic diagram of the physical structure of an electronic device is provided. Detailed Implementation
[0029] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.
[0030] This invention focuses on EEG-based emotion recognition tasks, aiming to establish a multi-task, cross-subject EEG emotion recognition method based on an information bottleneck. The information bottleneck refers to constructing a compressed intermediate representation model structure that compresses high-dimensional data to a lower dimension while maximizing useful information relevant to emotion recognition and minimizing noise and redundant information unrelated to emotion recognition. Multi-task emotion recognition refers to different emotion recognition tasks. These tasks can be based on dimensional emotion models, such as classification based on arousal, valence, or dominance; or on discrete emotion models, such as the recognition of happiness, sadness, fear, and anger; or on the classification of positive, negative, or neutral emotions. Cross-subject means using EEG signals from one group of subjects as the training set and EEG signals from another group as the test set, with the training and test sets not simultaneously including data from the same subject.
[0031] Figure 1 This is a flowchart illustrating the multi-task cross-subject EEG emotion recognition method provided in this application embodiment, as shown below. Figure 1 As shown, the method includes the following: Step 101: Obtain four-dimensional input data. The four-dimensional input data is determined based on time period, waveband, electrode position and raw EEG signal. The raw EEG signal is obtained based on multiple different EEG acquisition devices.
[0032] Specifically, the first step is to construct four-dimensional (4D) EEG input features. The raw EEG signal is input, segmented by waveband and time period, and its differential entropy (DE) features are extracted. The obtained DE features from different channels are mapped to a 2D array according to the relative positions of the electrodes on the scalp and arranged accordingly. Then, these 2D features are stacked by waveband to create a 3D format. Finally, the 3D features from different time periods are arranged chronologically to obtain the 4D input data used for emotion recognition.
[0033] Step 102: Feature extraction is performed on the four-dimensional input data based on a multilayer perceptron, and a general EEG representation is obtained based on self-attention and cross-attention. The general EEG representation is determined based on the primary EEG representation.
[0034] Specifically, then primary EEG representation and general feature encoding are performed.
[0035] In this process, a multilayer perceptron (MLP) corresponding to the dataset is used to extract primary representations from the 4D input obtained in S1, thereby obtaining primary EEG representations.
[0036] The general feature encoding employs learnable embeddings, aggregates useful information related to emotion recognition from primary representations through a cross-attention mechanism, and further obtains advanced general EEG representations by combining a Transformer block containing a self-attention mechanism.
[0037] Step 103: Determine the target features in the general EEG representation based on the target task. The target features are used to determine the prediction results of emotion recognition.
[0038] Specifically, task-specific decoding and classification are performed at the end. An MLP corresponding to a specific emotion recognition task is used to further extract useful features related to the specific task from general EEG representations, and the MLP is trained by minimizing the cross-entropy loss function to achieve multi-task, cross-subject emotion recognition.
[0039] The multi-task cross-subject EEG emotion recognition method provided in this application extracts features from four-dimensional input data and obtains general EEG representations based on self-attention and cross-attention. Then, it determines the target features in the general EEG representations based on the target task, thereby obtaining the predicted results of emotion recognition based on the target features, which can improve the accuracy of cross-subject emotion recognition.
[0040] In some embodiments, obtaining the four-dimensional input data includes: The original EEG signal is divided into multiple bands based on the filter, and the differential entropy (DE) feature of each band is obtained to determine the two-dimensional input data. The correspondence between the DE features of each band and the electrode positions is mapped onto the two-dimensional input data to obtain the three-dimensional input data; The three-dimensional input data is arranged chronologically based on time periods to obtain four-dimensional input data.
[0041] Specifically, Figure 2 This is a framework diagram of the multi-task cross-subject EEG emotion recognition method provided in the embodiments of this application, such as... Figure 2 As shown, the raw EEG signal is first divided into five bands—δ (1–3 Hz), θ (4–7 Hz), α (8–13 Hz), β (14–30 Hz), and γ (31–50 Hz)—using a Butterworth filter. Then, the data for each band is segmented according to time intervals (Ts), and the DE features of each segment within each band are calculated. The calculated DE features are mapped to a 2D array based on the spatial position of the electrodes on the scalp, obtaining EEG features containing both spatial and frequency domain information. These 2D EEG signal features are then stacked into a 3D format according to the bands, and every N segments are concatenated in time sequence to form 4D input data, fully utilizing the time-space-frequency domain information contained in the raw EEG signal.
[0042] Furthermore, Figure 3This is a diagram illustrating the four-dimensional EEG feature construction process provided in the embodiments of this application, such as... Figure 3 As shown, for each band, its DE characteristic is calculated using the following formula: h(Z)=-∫ Z f(z)log(f(z))dz Where Z represents the EEG signal, and f(z) represents the probability density function. When Z follows a Gaussian distribution N(μ,σ)... 2 When ), the above formula can be simplified to: Where e is Euler's constant.
[0043] Then, the DE features of each band are mapped into a 2D array according to the spatial location of the EEG electrodes on the scalp, and the 2D array of each band is further smoothed by Gaussian filtering or zscore.
[0044] Finally, the smoothed 2D array is stacked into a 3D format according to the bands, and the data of N segments are concatenated to form an input containing information in the time-space-frequency domains, with the following format: Where B represents the number of EEG signal bands.
[0045] The multi-task cross-subject EEG emotion recognition method provided in this application uses MLP as the EEG representation layer to extract primary EEG feature representations from 4D input data, which helps to map EEG signals recorded by different acquisition devices to the same representation space, laying the foundation for subsequent multi-task emotion recognition.
[0046] In some embodiments, the step of extracting features from the four-dimensional input data based on a multilayer perceptron and obtaining a general EEG representation based on self-attention and cross-attention includes: Based on a multilayer perceptron, feature extraction is performed on the four-dimensional input data to obtain a primary EEG representation. General EEG representations are obtained from the primary EEG representations based on self-attention and cross-attention.
[0047] Specifically, for the obtained 4D EEG input features, i.e., four-dimensional input data, for each time segment X n Spatial and frequency information is extracted, and based on the specific emotion recognition task, an MLP consisting of two linear layers is used to capture EEG feature representations, i.e., primary EEG representations. The calculation formula for this process is as follows: Y n =F RELU (F L2embeddim (F RELU (F L2hiddensize (Freshape (X n ))))) Where F RELU It is the ReLU activation function, F reshape It is the dimension reshaping function, F L2embeddim and F L2hiddensize There are two linear layers that map the data to the dimensions of embed_dim and hidden_size, respectively.
[0048] Then, a set of learnable embedding vectors is initialized. These embeddings, through cross-attention with primary EEG feature representations, aggregate useful information relevant to emotion recognition from the primary EEG feature representations. Simultaneously, because the learnable embedding vectors have a compressed representation dimension compared to the primary EEG features, redundant information or noise irrelevant to emotion recognition in the primary EEG representations can be removed. Furthermore, the learnable embeddings learn their own global dependencies through a self-attention mechanism. After this step, a general EEG representation for emotion recognition can be obtained.
[0049] The multi-task cross-subject EEG emotion recognition method provided in this application adopts learnable embedding and combines Transformer-based self-attention and cross-attention mechanisms to learn high-level general EEG features for emotion recognition. This helps to fully capture useful emotion-related information in multi-task EEG data, remove redundant information unrelated to emotion, and obtain general EEG representations related to emotion.
[0050] In some embodiments, the primary EEG representation is used to map the raw EEG signals to the same latent space.
[0051] Specifically, MLP is used as the EEG representation layer to extract primary EEG feature representations from the 4D input data. Since the data in different EEG datasets are acquired through different EEG acquisition devices, they have different numbers of channels and electrode placements. For different datasets, the 4D input data is fed into different EEG representation layers, which can initially map EEG data of different patterns into the same latent space.
[0052] The multi-task cross-subject EEG emotion recognition method provided in this application uses a multilayer perceptron associated with multiple datasets to extract primary representations of input 4D EEG features, and maps EEG signals with different channel electrode numbers, locations, formats, etc. to the same latent space, laying the foundation for subsequent multi-task emotion recognition.
[0053] In some embodiments, the step of obtaining a general EEG representation from the primary EEG representation based on self-attention and cross-attention includes: Obtain learnable embedding vectors; The general EEG representation is obtained based on the self-attention of the learnable embedding vector and the cross-attention between the learnable embedding vector and the primary EEG representation.
[0054] Specifically, Figure 4 This is a model structure diagram of the multi-task cross-subject EEG emotion recognition method provided in the embodiments of this application, as shown below. Figure 4 As shown, further general feature extraction is performed on the EEG feature representation, and a general feature encoder based on Transformer block is constructed. A set of learnable query embeddings with number L and dimension embed_dim is initialized, with the format Query∈R^(L×embed_dim). The query interacts with itself through the self-attention layer and with the EEG feature representation obtained in S2 through the cross-attention layer, so as to aggregate the effective information related to emotion recognition in the representation and compress redundant information and noise unrelated to emotion recognition.
[0055] Furthermore, for the self-attention layer, the learnable query is first mapped to Q through a linear layer. q K q V q Their dimensions are d q ,d k ,d v The self-attention of this substance is calculated using the following formula: For the cross-attention layer, EEG feature representations are first mapped to K through a linear layer. r V r Their dimensions are d k ,d v Q uses learnable queries q The cross-attention is calculated using the following formula: It is worth mentioning that the frequency of cross-attention layers can be controlled by cross_attention_frequency. For example, when cross_attention_frequency = 2, it means that a cross-attention layer appears once in every two Transformer blocks.
[0056] It is worth mentioning that a multi-head attention mechanism can be used to enable the model to learn various types of information from EEG feature representations. The learnable query and EEG feature representation are divided into h equal-length embeddings, where h represents the number of attention heads. Each embedding is individually fed into a self-attention layer and a cross-attention layer, and their outputs are concatenated. The calculation formula for this process is as follows: Multi_head_self_attention(Q q K q V q ) = [head0, head1, ..., head h-1 ] Multi_head_cross_attention(Q q ,K r V r = [head0, head1, ..., head h-1 ] in,
[0057] After the splicing process, the output MHA(Q, K, V) of the multi-head attention will be passed through a linear layer and a layer normalization function F. layernorm Then, a feedforward layer consisting of two linear layers is added after the attention layer. The calculation formula for this process is as follows: Attention_out(Y) = F layernorm (F L2embeddim (MHA(Q, K, V))) FF_out(Y) = F layernorm (F L2embeddim (F GELU (F L2interdim (Attention_out(Y))))) Among them, F GELU F is the GELU activation function. L2interdim For linear layers, data is mapped to the dimension of inter_dim.
[0058] Extract the last hidden layer of the learnable query and use a linear layer to extract it from... Mapped to The weight of each query is obtained, and then this weight is multiplied by the query itself to obtain the weighted query output UniFeat(Y), which is the general EEG representation. The calculation formula for this process is as follows: Query_weight = F reshape (F softmax (F L2one (Query))) UniFeat(Y)=Query_weight*Query Where F L2one For linear layers, data is mapped to a dimension of 1.
[0059] The multi-task cross-subject EEG emotion recognition method provided in this application adopts learnable embedding and combines Transformer-based self-attention and cross-attention mechanisms to learn high-level general EEG features for emotion recognition. This helps to fully capture useful emotion-related information in multi-task EEG data, remove redundant information unrelated to emotion, and obtain general EEG representations related to emotion.
[0060] In some embodiments, determining the target features in the general EEG representation based on the target task includes: using a multilayer perceptron corresponding to the target task as a decoder, training it based on minimizing the cross-entropy loss function, and obtaining the target features.
[0061] Specifically, an MLP corresponding to a specific emotion recognition task is used as the general EEG representation for decoding and classifying to further extract task-specific features, and the final prediction result is output. The calculation formula for this process is as follows: output(Y) = F L2class (UniFeat(Y)) Where F L2class This is a linear layer that maps data to the number of classes. The MLP is trained by minimizing the cross-entropy loss function.
[0062] Furthermore, to avoid overfitting or forgetting issues that may occur with a particular dataset during multi-task learning, a balanced training strategy is adopted. This involves sampling all data with a probability proportional to the number of training data points in each dataset. Assume there are D datasets, and the number of training samples in each dataset is {S1, S2, ..., S...}. d ,…,S D If}, then the probability p of taking a batch of data from a certain dataset for training is... d for: Where batchsize represents the number of samples in a batch.
[0063] The multi-task cross-subject EEG emotion recognition method provided in this application embodiment can automatically select the corresponding decoder according to the task category of the input data, realize the extraction and classification of specific task-related features, help to add personalized learning space for specific tasks on the basis of general EEG features, and enable the model to adapt to specific tasks on the basis of having general feature extraction capabilities.
[0064] Table 1 shows the comparative experimental results of the multi-task cross-subject EEG emotion recognition method provided in the embodiments of this application on the DEAP dataset.
[0065] Table 1. Comparative experimental results on the DEAP dataset AD-TCN 64.33％±7.06％ 63.25％±4.62％ HSLT 66.51％±8.53％ 65.75％±8.51％ SparseD 61.93％±6.03％ 65.94％±8.82％ RDFKM 64.5％ 63.1％ Our method 72.89％±6.00％ 73.20％±9.48％ The multi-task cross-subject EEG emotion recognition method provided in this application was trained and tested on the publicly available emotion recognition dataset DEAP. The classification accuracy was compared with that of other existing algorithms on the same dataset using leave-one-out cross-validation. The results show that the classification accuracy achieved by our method can be improved by about 6%-11% compared with other algorithms.
[0066] Table 2 shows the comparative experimental results of the multi-task cross-subject EEG emotion recognition method provided in the embodiments of this application on the DREAMER dataset.
[0067] Table 2. Comparative experimental results on the DREAMER dataset AD-TCN 66.56％±10.04％ 63.69％±6.57％ DCNN 64.98％ 63.71％ SparseD 65.67％±8.02％ 66.66％±7.05％ BiSMSM 61.88％ 64.25％ Our method 72.28％±7.84％ 71.66％±9.83％ The multi-task cross-subject EEG emotion recognition method provided in this application embodiment was trained and tested on the publicly available emotion recognition dataset DREAMER, and the classification accuracy was compared with that of other existing algorithms on the same dataset with leave-one-subject cross-validation. The results show that the classification accuracy achieved by our method can be improved by 5% to 11% compared with other algorithms.
[0068] Table 3 shows the comparative experimental results of the multi-task cross-subject EEG emotion recognition method provided in the embodiments of this application on the SEED dataset.
[0069] Table 3. Comparative experimental results on the SEED dataset EEGFuseNet 59.06％ TAS-Net 63.10％ Our method 73.14％ The multi-task cross-subject EEG emotion recognition method provided in this application embodiment was trained and tested on the publicly available emotion recognition dataset SEED, and the classification accuracy was compared with that of other existing algorithms on the same dataset with leave-one-out cross-validation. The results show that the classification accuracy achieved by our method can be improved by 10%-14% compared with other algorithms.
[0070] Table 4 shows the multi-task experimental results of the multi-task cross-subject EEG emotion recognition method provided in the embodiments of this application.
[0071] Table 4. Results of the multi-task experiment Acc 68.0％ 73.0％ 72.14％ 68.21％ 65.22％ 69.31％ Std 4.85％ 6.78％ 6.35％ 3.46％ 7.03％ 1.26％ The multi-task learning framework of the multi-task cross-subject EEG emotion recognition method provided in this application was trained and tested on five tasks (DEAP-V, DEAP-A, DREAMER-V, DREAMER-A, and SEED) on three publicly available emotion recognition datasets: DEAP, DREAMER, and SEED. One subject was randomly selected from each dataset as the test set, and the process was repeated five times. The average accuracy was calculated as the final result and compared with the algorithm's results on a single task independently for each subject. The results show that the classification performance achieved by the algorithm in multi-task learning is also quite excellent, with little difference in accuracy compared to single-task learning, demonstrating the model's effectiveness and potential in cross-subject, multi-task applications.
[0072] Experimental results show that the proposed multi-task, cross-subject EEG emotion recognition model based on information bottleneck can provide a joint learning framework for multiple EEG emotion recognition tasks, and achieve high accuracy cross-subject EEG-based emotion recognition on multiple datasets.
[0073] Figure 5 This is a schematic diagram of the structure of the multi-task cross-subject EEG emotion recognition device provided in the embodiments of this application, as shown below. Figure 5 As shown, the multi-task cross-subject EEG emotion recognition device provided in this application embodiment includes a first acquisition module 501, a second acquisition module 502, and a first determination module 503, wherein: The first acquisition module is used to acquire four-dimensional input data, which is determined based on time period, waveband, electrode position and raw EEG signal, and the raw EEG signal is acquired based on multiple different EEG acquisition devices; The second acquisition module is used to extract features from the four-dimensional input data based on a multilayer perceptron, and to acquire a general EEG representation based on self-attention and cross-attention, wherein the general EEG representation is determined based on the primary EEG representation. The first determining module is used to determine the target features in the general EEG representation based on the target task, and the target features are used to determine the prediction result of emotion recognition.
[0074] In some embodiments, the first acquisition module includes: The first acquisition submodule is used to divide the original EEG signal into multiple bands based on a filter, acquire the differential entropy (DE) feature of each band, and determine the two-dimensional input data. The first processing submodule is used to map the correspondence between the DE features of each band and the electrode positions to the two-dimensional input data to obtain three-dimensional input data; The second acquisition submodule is used to arrange the three-dimensional input data in a time sequence based on a time period to acquire four-dimensional input data.
[0075] In some embodiments, the second acquisition module includes: The third acquisition submodule is used to extract features from the four-dimensional input data based on a multilayer perceptron to acquire a primary EEG representation. The fourth acquisition submodule is used to acquire general EEG representations from the primary EEG representations based on self-attention and cross-attention.
[0076] In some embodiments, the primary EEG representation is used to map the raw EEG signals to the same latent space.
[0077] In some embodiments, the fourth acquisition submodule includes: The first acquisition unit is used to acquire learnable embedding vectors; The second acquisition unit is used to acquire the general EEG representation based on the self-attention of the learnable embedding vector and the cross-attention of the learnable embedding vector and the primary EEG representation.
[0078] In some embodiments, the first determining module includes: The fifth acquisition submodule is used to use the multilayer perceptron corresponding to the target task as a decoder, and train it based on minimizing the cross-entropy loss function to acquire the target features.
[0079] Specifically, the multi-task cross-subject EEG emotion recognition device provided in this application embodiment can implement all the method steps implemented in the above-mentioned multi-task cross-subject EEG emotion recognition method embodiment, and can achieve the same technical effect. Here, the parts that are the same as those in the method embodiment and the beneficial effects will not be described in detail.
[0080] Figure 6 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 6 As shown, the electronic device may include: a processor 610, a communications interface 620, a memory 630, and a communication bus 640, wherein the processor 610, the communications interface 620, and the memory 630 communicate with each other via the communication bus 640. The processor 610 can call logical instructions in the memory 630 to execute a multi-task cross-subject EEG emotion recognition method, which includes: Four-dimensional input data is acquired, which is determined based on time period, waveband, electrode position and raw EEG signal, which is acquired based on multiple different EEG acquisition devices; Feature extraction is performed on the four-dimensional input data based on a multilayer perceptron, and a general EEG representation is obtained based on self-attention and cross-attention. The general EEG representation is determined based on the primary EEG representation. The target features in the general EEG representation are determined based on the target task, and the target features are used to determine the prediction results of emotion recognition.
[0081] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0082] In some embodiments, obtaining the four-dimensional input data includes: The original EEG signal is divided into multiple bands based on the filter, and the differential entropy (DE) feature of each band is obtained to determine the two-dimensional input data. The correspondence between the DE features of each band and the electrode positions is mapped onto the two-dimensional input data to obtain the three-dimensional input data; The three-dimensional input data is arranged chronologically based on time periods to obtain four-dimensional input data.
[0083] In some embodiments, the step of extracting features from the four-dimensional input data based on a multilayer perceptron and obtaining a general EEG representation based on self-attention and cross-attention includes: Based on a multilayer perceptron, feature extraction is performed on the four-dimensional input data to obtain a primary EEG representation. General EEG representations are obtained from the primary EEG representations based on self-attention and cross-attention.
[0084] In some embodiments, the primary EEG representation is used to map the raw EEG signals to the same latent space.
[0085] In some embodiments, the step of obtaining a general EEG representation from the primary EEG representation based on self-attention and cross-attention includes: Obtain learnable embedding vectors; The general EEG representation is obtained based on the self-attention of the learnable embedding vector and the cross-attention between the learnable embedding vector and the primary EEG representation.
[0086] In some embodiments, determining the target features in the general EEG representation based on the target task includes: using a multilayer perceptron corresponding to the target task as a decoder, training it based on minimizing the cross-entropy loss function, and obtaining the target features.
[0087] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program that can be stored on a non-transitory computer-readable storage medium, wherein when the computer program is executed by a processor, the computer is capable of executing the multi-task cross-subject EEG emotion recognition method provided by the above methods, the method comprising: Four-dimensional input data is acquired, which is determined based on time period, waveband, electrode position and raw EEG signal, which is acquired based on multiple different EEG acquisition devices; Feature extraction is performed on the four-dimensional input data based on a multilayer perceptron, and a general EEG representation is obtained based on self-attention and cross-attention. The general EEG representation is determined based on the primary EEG representation. The target features in the general EEG representation are determined based on the target task, and the target features are used to determine the prediction results of emotion recognition.
[0088] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the multi-task cross-subject EEG emotion recognition method provided by the methods described above, the method comprising: Four-dimensional input data is acquired, which is determined based on time period, waveband, electrode position and raw EEG signal, which is acquired based on multiple different EEG acquisition devices; Feature extraction is performed on the four-dimensional input data based on a multilayer perceptron, and a general EEG representation is obtained based on self-attention and cross-attention. The general EEG representation is determined based on the primary EEG representation. The target features in the general EEG representation are determined based on the target task, and the target features are used to determine the prediction results of emotion recognition.
[0089] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.
[0090] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.
[0091] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A multi-task cross-subject EEG emotion recognition method, characterized in that, include: Four-dimensional input data is acquired, which is determined based on time period, waveband, electrode position and raw EEG signal, which is acquired based on multiple different EEG acquisition devices; Feature extraction is performed on the four-dimensional input data based on a multilayer perceptron, and a general EEG representation is obtained based on self-attention and cross-attention. The general EEG representation is determined based on the primary EEG representation. The target features in the general EEG representation are determined based on the target task, and the target features are used to determine the prediction results of emotion recognition; The process of extracting features from the four-dimensional input data based on a multilayer perceptron and obtaining a general EEG representation based on self-attention and cross-attention includes: Based on a multilayer perceptron, feature extraction is performed on the four-dimensional input data to obtain a primary EEG representation. A general EEG representation is obtained from the primary EEG representation based on self-attention and cross-attention; The acquisition of general EEG representations from the primary EEG representations based on self-attention and cross-attention includes: Obtain learnable embedding vectors; The general EEG representation is obtained based on the self-attention of the learnable embedding vector and the cross-attention of the learnable embedding vector and the primary EEG representation; the learnable embedding vector has a compressed representation dimension compared to the primary EEG representation.
2. The multi-task cross-subject EEG emotion recognition method according to claim 1, characterized in that, The acquisition of four-dimensional input data includes: The original EEG signal is divided into multiple bands based on the filter, and the differential entropy (DE) feature of each band is obtained to determine the two-dimensional input data. The correspondence between the DE features of each band and the electrode positions is mapped onto the two-dimensional input data to obtain the three-dimensional input data; The three-dimensional input data is arranged chronologically based on time periods to obtain four-dimensional input data.
3. The multi-task cross-subject EEG emotion recognition method according to claim 1, characterized in that, The primary EEG representation is used to map the raw EEG signals to the same latent space.
4. The multi-task cross-subject EEG emotion recognition method according to claim 1, characterized in that, The determination of target features in the general EEG representation based on the target task includes: The multilayer perceptron corresponding to the target task is used as a decoder, and the target features are obtained by training based on minimizing the cross-entropy loss function.
5. A multi-task cross-subject EEG emotion recognition device, characterized in that, include: The first acquisition module is used to acquire four-dimensional input data, which is determined based on time period, waveband, electrode position and raw EEG signal, and the raw EEG signal is acquired based on multiple different EEG acquisition devices; The second acquisition module is used to extract features from the four-dimensional input data based on a multilayer perceptron, and to acquire a general EEG representation based on self-attention and cross-attention, wherein the general EEG representation is determined based on the primary EEG representation. The first determining module is used to determine the target features in the general EEG representation based on the target task, and the target features are used to determine the prediction result of emotion recognition; The process of extracting features from the four-dimensional input data based on a multilayer perceptron and obtaining a general EEG representation based on self-attention and cross-attention includes: Based on a multilayer perceptron, feature extraction is performed on the four-dimensional input data to obtain a primary EEG representation. A general EEG representation is obtained from the primary EEG representation based on self-attention and cross-attention; The acquisition of general EEG representations from the primary EEG representations based on self-attention and cross-attention includes: Obtain learnable embedding vectors; The general EEG representation is obtained based on the self-attention of the learnable embedding vector and the cross-attention of the learnable embedding vector and the primary EEG representation; the learnable embedding vector has a compressed representation dimension compared to the primary EEG representation.
6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the multi-task cross-subject EEG emotion recognition method as described in any one of claims 1 to 4.
7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the multi-task cross-subject EEG emotion recognition method as described in any one of claims 1 to 4.
8. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the multi-task cross-subject EEG emotion recognition method as described in any one of claims 1 to 4.