Multi-task EEG emotional state classification method, system and equipment based on residual cross attention

Abstract

The invention discloses a multi-task EEG emotional state classification method, system and device based on residual cross attention, and relates to the field of electroencephalogram signal emotional state recognition, and the method comprises the steps: obtaining to-be-classified EEG data; performing de-noising processing on the EEG data to be classified to obtain de-noised EEG data to be classified; performing time sequence feature extraction on the de-noised to-be-classified EEG data to obtain time sequence features of the to-be-classified EEG data; inputting the time sequence characteristics of the to-be-classified EEG data into a trained EEG emotional state classification model, and determining an emotional state; the EEG emotional state classification model comprises a feature extraction module, a residual multi-head cross attention module, a gating multi-embedding module and a task tower classification module. The EEG emotional state classification method and device can improve the accuracy of EEG emotional state classification.

Description

Technical Field

[0001] The present application relates to the field of EEG signal emotion state recognition, and in particular to a multi-task EEG emotion state classification method, system and device based on residual cross-attention. Background Art

[0002] Emotions are complex psychological and physiological responses of humans to internal and external environmental stimuli, and they play an important role in human-computer interaction and affective computing. Neuroscience research shows that physiological signals can more directly reflect the source of emotions. Compared with behavioral signals such as facial expressions and voice, EEG signals (Electroencephalogram), as a non-invasive neurophysiological technology, are particularly difficult to disguise and hide. With the rapid development of non-invasive and cost-effective EEG recording equipment, the application of EEG-based emotion recognition in deep learning has attracted more and more attention.

[0003] At present, EEG-based emotion recognition models can be mainly divided into discrete models and dimensional models. Discrete models usually divide emotions into specific discrete states, such as happiness, sadness, anger, etc., while dimensional models analyze emotions based on different dimensions, such as the valence (positive or negative) and arousal (emotional intensity) of emotions. In recent years, emotion recognition models based on emotion dimensions have received extensive research attention and practical application promotion. However, most current studies only focus on a single task, such as only identifying the valence of emotions or only identifying the arousal, and fail to fully consider the correlation between different tasks, ignoring the potential information sharing between tasks, resulting in low accuracy in EEG emotional state classification. Summary of the invention

[0004] The purpose of this application is to provide a multi-task EEG emotional state classification method, system and device based on residual cross-attention, which can improve the accuracy of EEG emotional state classification.

[0005] To achieve the above objectives, this application provides the following solutions:

[0006] In a first aspect, the present application provides a multi-task EEG emotional state classification method based on residual cross attention, comprising:

[0007] Obtain EEG data to be classified;

[0008] Performing denoising on the EEG data to be classified to obtain denoised EEG data to be classified;

[0009] Extracting time series features from the denoised EEG data to be classified to obtain time series features of the EEG data to be classified;

[0010] The temporal features of the EEG data to be classified are input into a trained EEG emotional state classification model to determine the emotional state; the EEG emotional state classification model includes a feature extraction module, a residual multi-head cross attention module, a gated multi-embedding module and a task tower classification module.

[0011] Optionally, the training process of the EEG emotional state classification model is:

[0012] Constructing a training set based on the DEAP database; the training set includes the temporal features of the sample EEG data and the emotional state corresponding to the sample EEG data;

[0013] Construct an EEG emotional state classification model;

[0014] Using the loss function, the temporal features of the sample EEG data are taken as input and the emotional state corresponding to the sample EEG data is taken as output to train the EEG emotional state classification model, thereby obtaining a trained EEG emotional state classification model.

[0015] Optionally, construct a training set, including:

[0016] Get sample EEG data from the DEAP database;

[0017] Performing denoising processing on the sample EEG data to obtain denoised sample EEG data;

[0018] Extracting time series features from the denoised sample EEG data to obtain time series features of the sample EEG data;

[0019] A training set is constructed based on the temporal features of the sample EEG data and the emotional state corresponding to the sample EEG data.

[0020] Optionally, inputting the temporal features of the EEG data to be classified into a trained EEG emotional state classification model to determine the emotional state specifically includes:

[0021] Inputting the time series features of the EEG data to be classified into the feature extraction module to obtain a feature set; the feature set includes arousal features, valence features and shared features;

[0022] Inputting the feature set into the residual multi-head cross attention module to obtain specific features; the specific features include arousal specific features and valence specific features;

[0023] Inputting the specific features into the gated multi-embedding module to obtain the specific features after removing useless information;

[0024] The specific features after removing useless information are input into the task tower classification module to determine the emotional state.

[0025] Optionally, the feature extraction module includes: an arousal task expert unit, a valence task expert unit and a sharing task expert unit;

[0026] The time series features of the EEG data to be classified are input into the feature extraction module to obtain a feature set, which specifically includes:

[0027] Inputting the time series features of the EEG data to be classified into the arousal task expert unit to obtain arousal features;

[0028] Inputting the time series features of the EEG data to be classified into the valence task expert unit to obtain valence features;

[0029] The time series features of the EEG data to be classified are input into the shared task expert unit to obtain shared features.

[0030] Optionally, the feature set is input into the residual multi-head cross attention module to obtain specific features, specifically including:

[0031] The valence feature and the arousal feature are respectively concatenated with the shared feature to obtain related features; the related features include a first related feature and a second related feature;

[0032] A multi-head attention value is obtained based on the arousal feature, the valence feature and the related feature; the multi-head attention value includes a first multi-head attention value and a second multi-head attention value;

[0033] Normalizing a result of adding the arousal feature and the first multi-head attention value to obtain an arousal specific feature;

[0034] A result of adding the valence feature and the second multi-head attention value is normalized to obtain a valence specific feature.

[0035] Optionally, obtaining a multi-head attention value based on the arousal feature, the valence feature, and the correlation feature specifically includes:

[0036] Using multiple projection matrices, perform multiple linear transformations on the arousal feature and the first related feature to obtain a first linear vector set;

[0037] Using multiple projection matrices, multiple linear transformations are performed on the valence feature and the second correlation feature to obtain a second linear vector set;

[0038] A cross-task multi-head attention value calculation is performed on the first linear vector set to obtain a first attention value set, and the first attention value set is spliced ​​to obtain a first multi-head attention value.

[0039] Optionally, the specific features are input into the gated multi-embedding module to obtain the specific features after removing useless information, specifically including:

[0040] Based on the multi-embedding paradigm, the specific features are linearly transformed to obtain an embedding result;

[0041] Using a sigmoid activation function to activate the embedding result to obtain an activated embedding result;

[0042] The embedding result and the activated embedding result are point-multiplied and linearly transformed to obtain specific features after removing useless information.

[0043] In a second aspect, the present application provides a multi-task EEG emotional state classification system based on residual cross-attention, which is used to implement any of the multi-task EEG emotional state classification methods based on residual cross-attention described above, and the multi-task EEG emotional state classification system based on residual cross-attention includes:

[0044] An acquisition module is used to obtain EEG data to be classified;

[0045] A denoising module is used to perform denoising on the EEG data to be classified to obtain denoised EEG data to be classified;

[0046] A time series feature extraction module is used to extract time series features from the denoised EEG data to be classified, so as to obtain time series features of the EEG data to be classified;

[0047] The emotional state classification module is used to input the temporal features of the EEG data to be classified into the trained EEG emotional state classification model to determine the emotional state; the EEG emotional state classification model includes a feature extraction module, a residual multi-head cross-attention module, a gated multi-embedding module and a task tower classification module.

[0048] In a third aspect, the present application provides a computer device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement any of the steps of the multi-task EEG emotional state classification method based on residual cross-attention described above.

[0049] According to the specific embodiments provided in this application, this application has the following technical effects:

[0050] The present application provides a multi-task EEG emotional state classification method, system and device based on residual cross-attention, which classifies the EEG data to be classified through a trained EEG emotional state classification model, thereby determining the emotional state. The EEG emotional state classification model includes a feature extraction module, a residual multi-head cross-attention module, a gated multi-embedding module and a task tower classification module; the feature extraction module can extract shared features from the EEG data to be classified; the residual multi-head cross-attention module can learn the correlation between different tasks to dynamically adjust the attention weight, explicitly capture the shared and specific information between tasks, and promote complementarity and sharing between tasks; the gated multi-embedding module can reduce the impact of useless information, thereby reducing the negative impact on task performance; therefore, the present application can improve the accuracy of EEG emotional state classification through the EEG emotional state classification model. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0052] Figure 1 A flowchart of a multi-task EEG emotional state classification method based on residual cross-attention provided in one embodiment of the present application;

[0053] Figure 2 A schematic diagram of a framework of a multi-task EEG emotional state classification method based on residual cross-attention provided in one embodiment of the present application;

[0054] Figure 3 A schematic diagram of the structure of an EEG emotional state classification model provided in an embodiment of the present application;

[0055] Figure 4 A schematic diagram of the structure of a residual multi-head cross attention module provided in an embodiment of the present application;

[0056] Figure 5 A schematic diagram of the structure of a gated multi-embedded module provided in one embodiment of the present application;

[0057] Figure 6 A schematic diagram of the structure of a computer device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0058] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0059] At present, there are still the following problems in building a general multi-task model: 1. Existing multi-task models usually rely on a unified shared model, but when the correlation between tasks is weak or the tasks are complex, this over-sharing may lead to improper knowledge transfer, thereby affecting task performance. 2. The task sharing feature extraction in the existing models is often implicit, and does not fully consider the explicit correlation between tasks, ignoring the dependencies between tasks or feature interactions, which limits the effectiveness of feature learning. 3. Existing multi-task models usually do not filter harmful features after feature fusion, resulting in the fused features may contain features that are useless for specific tasks, thereby affecting the adaptability and performance of the model. The above problems will ultimately lead to low accuracy in emotional state classification. Therefore, the present application provides a multi-task EEG emotional state classification method, system and device based on residual cross-attention, which can improve the accuracy of EEG emotional state classification.

[0060] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0061] In an exemplary embodiment, Figure 1-Figure 2 As shown, a multi-task EEG emotional state classification method based on residual cross attention is provided, and the method comprises the following steps:

[0062] Step 1: Obtain the EEG data to be classified.

[0063] Step 2: De-noise the EEG data to be classified to obtain the denoised EEG data to be classified.

[0064] Step 3: Extract the time series features of the denoised EEG data to be classified to obtain the time series features of the EEG data to be classified.

[0065] Step 4: Input the temporal features of the EEG data to be classified into the trained EEG emotion state classification model to determine the emotional state; the EEG emotion state classification model includes a feature extraction module, a residual multi-head cross attention module, a gated multi-embedding module and a task tower classification module.

[0066] In a specific embodiment, the training process of the EEG emotional state classification model specifically includes:

[0067] (1) Construct a training set based on the DEAP database; the training set includes the temporal features of the sample EEG data and the emotional state corresponding to the sample EEG data.

[0068] (2) Construct an EEG emotional state classification model.

[0069] (3) Using the loss function, taking the temporal features of each sample EEG data as input and the emotional state corresponding to the sample EEG data as output, the EEG emotional state classification model is trained to obtain a trained EEG emotional state classification model, specifically including:

[0070] Initialize the EEG emotional state classification model;

[0071] The EEG emotion state classification model is trained multiple times using the training set to obtain a trained EEG emotion state classification model; wherein the training process in any current round includes:

[0072] The temporal features of the sample EEG data are input into the EEG emotional state classification model under the current round to obtain the predicted emotional state under the current round.

[0073] Using the loss function, the loss for the current round is calculated based on the predicted emotional state for the current round and the emotional state corresponding to the sample EEG data.

[0074] Determine whether a stopping condition is met; the stopping condition is that a preset training round is reached, the loss in the current round is less than a preset loss, or the absolute value of the difference between the loss in the current round and the loss in the previous round is less than a preset absolute value.

[0075] If not, update the parameters of the EEG emotional state classification model in the current round, and return to "input the temporal features of the sample EEG data into the EEG emotional state classification model in the current round to obtain the predicted emotional state in the current round".

[0076] If so, the EEG emotion state classification model in the current round is determined as the trained EEG emotion state classification model.

[0077] Specifically, due to the differences in difficulty and progress of training each task, the overall target loss function is usually optimized to ensure that not only the performance of a single task is improved but each task can achieve good results. The overall target loss function is usually defined as a weighted linear combination of the losses (Loss, L) of each task. In order to cope with the different variances and offsets between single-task losses, this application introduces an automatic weighted loss function, and the final overall target loss function of the model is The specific formula is as follows:

[0078]

[0079] Where x is a sample; σ is adaptive learning to weight the different losses of each task; is the predicted label; y T ={y a ,y b} is the true label; w T is the category weight; is the loss function of arousal; is the loss function of the valence task; σ a for The relative weight that should be learned for adaptation; σ b for The relative weights that the adaptation should learn; is the regularization term.

[0080] The loss function for a single task can use any loss function, such as cross-entropy, KL-div, focalloss, etc.

[0081] The overall objective loss function defined above is used when training the EEG emotional state classification model. Where x is the sample, y T is the corresponding multi-task label, is the multi-task result predicted by the model, w T is the class weight. The training process uses an adaptive time estimation method to perform gradient descent. Specifically, batch training is used to adjust the model parameters through continuous iterations so that the overall loss function It decreases until the loss converges or the preset number of iterations is reached.

[0082] In a specific embodiment, constructing a training set specifically includes:

[0083] 1) Obtain the original EEG signal (i.e., sample EEG data) in the DEAP database.

[0084] Specifically, the data set used in this application for training model performance is derived from DEAP, and specific reference may be made to the paper Koelstra S, Mühl C, 0001S M, et al. DEAP: A Database for Emotion Analysis; Using Physiological Signals. [J]. IEEE Trans. Affective Computing, 2012, 3(1): 18-31. .

[0085] 2) Performing denoising processing on the sample EEG data to obtain denoised sample EEG data.

[0086] Specifically, the raw EEG signals collected in the dataset were preprocessed to reduce the sampling rate from 512 Hz to 128 Hz, and a 4.0 Hz–45.0 Hz bandpass frequency filter was used to remove artifacts.

[0087] 3) Extracting time series features from the denoised sample EEG data to obtain time series features of the sample EEG data.

[0088] Specifically, the denoised EEG data was subjected to temporal feature extraction, and each signal was segmented into 60 segments using a 1-second sliding window for each subject, with each EEG signal segment having 128 sampling points. Since each subject conducted 40 experiments, approximately 2,400 samples (40 experiments × 60 segments) could be obtained for each subject, and a total of 76,800 temporal feature extractions were obtained for 32 subjects.

[0089] 4) Construct a training set based on the temporal characteristics of the sample EEG data and the emotional state corresponding to the sample EEG data.

[0090] Specifically, suppose there are N subjects, the set of N subjects is represented as U, and the data set of each subject is represented as Where s represents the subject, n represents the number of subjects in a sample, It represents the i-th sample of subject s, and the corresponding labels of task a and task b are

[0091] This application adopts the cross-subject method and divides the data set by the leave-one-out method. Specifically, first take the data of all emotional tests of the first subject and assume that its emotional state label is unknown as the test set; take the data of all emotional tests of the remaining N-1 subjects and assume that their emotional state labels are known as the training set. Then take the emotional test of the i-th subject as the test set i=2, 3,..., N in turn, and take the data of all emotional tests of the remaining N-1 subjects as the training set.

[0092] For the cross-subject experimental scenario: the test set data input is in represents the i-th sample of subject t, with a total of n samples; the training set input of the model is s∈Ut, where represents the i-th sample of subject s, represents the arousal label of the i-th sample of subject s, Represents the valence label of the i-th sample of subject s. Each subject has n samples in total. Ut represents the set of all subjects except t. The model does not obtain any information about the samples from the test set. Label information.

[0093] In a specific embodiment, step 4 specifically includes:

[0094] Step 41: Input the extracted EEG data to be classified into a feature extraction module to obtain a feature set; the feature set includes arousal features, valence features and shared features.

[0095] Step 42: Input the feature set into the residual multi-head cross attention module to obtain specific features; the specific features include arousal specific features and valence specific features.

[0096] Step 43: Input the specific features into the gated multi-embedding module to obtain the specific features after removing useless information.

[0097] Step 44: Input the specific features after eliminating useless information into the task tower classification module to determine the emotional state.

[0098] In a specific embodiment, the feature extraction module includes: an arousal task expert unit, a valence task expert unit and a sharing task expert unit. Step 41 specifically includes:

[0099] Step 411: input the time series features of the EEG data to be classified into the arousal task expert unit to obtain arousal features;

[0100] Step 412: input the time series features of the EEG data to be classified into the valence task expert unit to obtain the valence features;

[0101] Step 413: Input the time series features of the EEG data to be classified into the shared task expert unit to obtain shared features.

[0102] Specifically, the feature extraction module mainly learns task-specific features through their own specific task experts and learns shared features through shared task experts. The feature extraction module includes: arousal task expert unit (f a ), valence task expert unit (f b ) and shared task expert units (f s The three models use the same network model structure, but can be any network model (such as dynamic graph convolutional neural network, shallow convolutional neural network, deep convolutional neural network, etc.). Through the three feature extraction expert models, the training sets of the models are extracted to obtain specific arousal features (F a ), potency characteristics (F b ) and shared features (F s ). Figure 3 As shown on the left, the process can be described as:

[0103] F a =fa (x),F b =f b (x),F s =f s (x)(2)

[0104] Where x∈R n×c×t As the input of the task model, n is the number of samples, c is the number of channels, and t is the time.

[0105] In a specific embodiment, step 42 specifically includes:

[0106] Step 421: concatenate the valence feature and the arousal feature with the shared feature respectively to obtain related features; the related features include the first related feature and the second related feature.

[0107] Step 422: Obtain a multi-head attention value based on the arousal feature, the valence feature, and the related features; the multi-head attention value includes a first multi-head attention value and a second multi-head attention value.

[0108] Step 423: normalize the result of adding the arousal feature and the first multi-head attention value to obtain an arousal specific feature.

[0109] Step 424: Normalize the result of adding the valence feature and the second multi-head attention value to obtain a valence-specific feature.

[0110] Specifically, the Residual Multi-head Cross Attention module (RMCA module) explicitly captures the task features (F a ,F b ,F s ), dig out the correlation between the main features and other features, and then extract the beneficial features from other features to supplement the deficiencies of the main features and improve the effectiveness of the main features in their classification tasks. Finally, the two sets of task features {F a ,(F b ,F s )},{F b ,(F a ,F s )} Through the residual multi-head cross attention module, the corresponding task-specific features are obtained.

[0111] Specific characteristics by potency For example, Figure 4 As shown in Figure 2, taking the valence task (B) as the key task, the key feature of the module is F b , other features are Fa and F s . b As X in the residual multi-head cross attention module b , F a and F s Perform concatenation and serve as X in the residual multi-head cross attention module c , the specific formula is as follows:

[0112] X b =F b ,X c =Concat(F a ,F s )(3)

[0113] X b , X c The features are linearly transformed to generate the corresponding query, key, and value vectors. Specifically, the feature X of key task B is transformed into b Generate query vector Q b , the features X of other tasks C c Generate key vectors K respectively c , value vector V c The specific formula is as follows:

[0114] Q b =W q,b X b ,K c =W k,c X c ,V c =W v,c X c (4)

[0115] Among them, W q,b is the query projection matrix for the key task B, W k,c is the key projection matrix for other tasks C; W v,c is the value projection matrix used for other tasks C.

[0116] Calculate the query vector Q of key task B b The key vector K with other tasks C c The cross-task attention value between them adds meaningful information from other tasks C to the key task B. The specific formula is as follows:

[0117]

[0118] Among them, d k The dimension of the key vector, used to scale the dot product to avoid the dot product value being too large as the dimension increases.

[0119] In order to extract diverse features from different subspaces, multi-head attention is used to focus on different feature subspaces. For feature X b ,X c Projection is performed to obtain h sets of different query, key, and value vectors Then, each group is subjected to parallel cross-task attention operations to obtain h groups of independent attention heads. Finally, the outputs of all heads are concatenated and mapped back to the original dimensional space. The specific formula is as follows:

[0120]

[0121] MultiHead(Q b ,K c ,V c )=W o Concat(head1,head2,…,head h )(7)

[0122] Among them, head i is the i-th attention head; W o is a projection matrix used to map the spliced ​​result back to the original dimensional space; MultiHead(Q b ,K c ,V c ) is the multi-head attention value.

[0123] In order to preserve the original information of the input features, allow effective learning of deep features and alleviate the gradient vanishing problem, the input feature X of task B is b Add the output of the multi-head attention to form a residual connection, and normalize the result to ensure that the mean value is 0 and the variance is 1. The specific formula is as follows:

[0124]

[0125] Similarly, when the arousal task (A) is taken as the key task, the key feature is F a , other features are F b and F s , F b and F s Splice to get F d , and for F d , F a Conduct with The same operation is used to obtain

[0126]

[0127] Where X a is the input feature of task A, is the arousal-specific feature, Q a is the query vector of task A, K d is the key vector of other tasks, V d is the value vector of other tasks.

[0128] In a specific embodiment, step 422 specifically includes:

[0129] Step 4221: Use multiple projection matrices to perform multiple linear transformations on the wake-up feature and the first related feature to obtain a first linear vector set.

[0130] Step 4222: Use multiple projection matrices to perform multiple linear transformations on the valence feature and the second correlation feature to obtain a second linear vector set.

[0131] Step 4223: Perform cross-task multi-head attention value calculation on the first linear vector set to obtain a first attention value set, and splice the first attention value set to obtain a first multi-head attention value.

[0132] In a specific embodiment, step 43 specifically includes:

[0133] Step 431: Based on the multi-embedding paradigm, linearly transform the specific features to obtain an embedding result.

[0134] Step 432: Activate the embedding result using a sigmoid activation function to obtain an activated embedding result.

[0135] Step 433: perform a point multiplication on the embedding result and the activated embedding result, and perform a linear transformation to obtain specific features after removing useless information.

[0136] Specifically, the Gated Multi Embedding Module (GME module) removes features through a gating mechanism. The information that is useless for a specific task is removed, thereby reducing the negative impact of this information on task performance. As the embedding dimension increases, the amount of embedded information is not significantly improved, and the embedding matrix still exhibits low-rank characteristics. Therefore, simply expanding the model size will not effectively improve the ability to express information, but may lead to information redundancy. Therefore, using multiple embeddings instead of single embeddings, combined with a gating mechanism, can obtain richer feature information from multiple dimensions. Finally, Through the respective GME modules, harmful features are removed and the

[0137] The valence-specific features after removing useless information For example, Figure 5 As shown, the valence task (B) is the key task. The GME module contains two Multi-Embedding components. Unlike simply increasing the size of the embedding, Multi-Embedding adds multiple independent and complete embedding layers to obtain richer feature information from multiple dimensions. Perform multiple linear transformations, each of which generates an independent embedding feature, and then concatenate all the embedded features. The specific formula is as follows:

[0138]

[0139] in, is the weight matrix of the i-th embedding layer; b i is the bias value of the i-th embedding layer; is the output of the i-th embedding layer; Multi-Embedding is the final embedding result, E b ∈R c×r , Where c is The dimension of the feature, m is the preset number of embedding layers, and r is the preset expansion ratio.

[0140] In order to better filter out useless information, a multi-embedding is processed by a sigmoid function to highlight useful features, and then the processed embedding layer is further filtered out by dot multiplication with another multi-embedding layer. Finally, the linear layer is used to obtain The specific formula is as follows:

[0141]

[0142] Among them, W E,b Output the weight matrix of the linear layer for task B; b E,b Output the bias value of the linear layer for task B, represents the matrix dot multiplication operation, and σ represents the sigmoid activation function.

[0143] Similarly, when we take the arousal task (A) as the key task, the specific formula is as follows:

[0144]

[0145] in, It is the specific feature of arousal after removing useless information.

[0146] In a specific embodiment, step 44 specifically includes:

[0147] The task tower classification module consists of two independent task towers, each responsible for predicting a specific task: arousal task and valence task. and input into their respective task towers for classification to obtain prediction results The prediction results include the arousal prediction results and valence prediction results Right now

[0148] like Figure 3 As shown on the right, this application will extract features from GME Input into the respective task tower for classification to obtain arousal and valence prediction results

[0149]

[0150] Among them, classfiers a is the arousal classifier; classfiers b is the valence classifier; Predict results for arousal; The result of potency prediction.

[0151] In a specific embodiment, after the EEG emotion state classification model is trained, the test set used when constructing the training set is used to verify the accuracy of the trained EEG emotion state classification model. Specifically, the test set is input into the trained model, and the prediction result of the test set is obtained. And the true state y shown by the label in the test set T Compare and obtain the comparison results to evaluate the model performance. The accuracy is the number of correctly classified samples in the model test to the total number of test samples. The calculation formula for the accuracy of each subject in the test set is as follows:

[0152]

[0153] Among them, TP is the positive sample predicted by the model as positive, TN is the negative sample predicted by the model as negative, FP is the negative sample predicted by the model as positive, and FN is the positive sample predicted by the model as negative. The DEAP data includes 32 subjects, and the average accuracy is as follows:

[0154]

[0155] The resulting mean square error formula is as follows:

[0156]

[0157] The basic models were tested, including: Dynamic Graph Convolutional Neural Networks (DGCNN), Shallow Convolutional Neural Network (ShallowConvNet), Deep Convolutional Neural Network (DeepConvNet), A Compact Convolutional Neural Network for EEG-based Brain-Computer Interfaces (EEGNet). The final test results show that the average accuracy rate of the basic model performance is improved as shown in Table 1:

[0158] Table 1 Performance improvement of basic model

[0159]

[0160] As can be seen from Table 1, all basic models (EEGNet, DGCNN, DeepConvNet and ShallowConvNet) have improved in valence and arousal tasks, indicating that the present application can significantly improve the performance of the model in these tasks.

[0161] In order to compare the advantages of the EEG emotion state classification model of the present application, the present application conducted similar experiments with other methods and compared them with the existing most advanced technologies, including: Hybrid Unsupervised Deep Feature Characterization and Fusion for High-Dimensional EEG With an Application to Emotion Recognition (EEGFuseNet for short), WTCNN from the paper "EEG emotion recognition based on the attention mechanism and pre-trained convolution capsule network", HLST from the paper "Transformers for EEG-Based Emotion Recognition: A Hierarchical Spatial Information Learning Model", STFG-CAP from the paper "Capsule neural networkson spatio-temporal EEG frames for cross-subject emotion recognition", TMLP+SRDANN from the paper "TMLP+SRDANN: A domain adaptation method for EEG-based emotion recognition", and EEGFuseNet from the paper "EEG emotion recognitionbased on attention mechanism and pre-trained convolution capsule network". The comparison is shown in Table 2:

[0162] Table 2 Comparison of the accuracy of existing state-of-the-art technologies

[0163]

[0164] From the results in Table 2 above, it can be seen that the method proposed in this application achieves higher accuracy and smaller variance, showing obvious advantages in multi-task emotion classification.

[0165] This application is not only applicable to multi-task research on emotional state recognition, but also to any EEG-based multi-task classification prediction. It solves the problem of individual differences in EEG to a certain extent and has the advantages of low time complexity, high computational efficiency, and strong generalization ability.

[0166] This application proposes a multi-task EEG emotional state classification method based on residual cross-attention mechanism, which uses specific and shared expert models to learn task-specific and shared features to reduce the negative transfer between shared and specific knowledge; uses residual cross-attention module (RMCA) to learn the correlation between different tasks to dynamically adjust the attention weight, explicitly capture the shared and specific information between tasks, and promote the complementarity and sharing between tasks; adopts gated multi-head embedding module (GME) to reduce the influence of harmful or useless features, and combines the task tower classification module with the loss weight self-adjustment mechanism to balance the loss of different tasks to achieve emotion classification. In addition, MT-RCAF has improved the performance of all basic models (EEGNet, DGCNN, DeepConvNet and ShallowConvNet) to varying degrees. At the same time, MT-RCAF (EEGNet) is compared with the most advanced models and significantly outperforms the emotion classification performance of other models.

[0167] Based on the same inventive concept, the embodiment of the present application also provides a multi-task EEG emotional state classification system based on residual cross-attention. The implementation scheme for solving the problem provided by the system is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more embodiments of the multi-task EEG emotional state classification system based on residual cross-attention provided below can refer to the limitations of the multi-task EEG emotional state classification method based on residual cross-attention above, and will not be repeated here.

[0168] In an exemplary embodiment, a multi-task EEG emotional state classification system based on residual cross-attention is provided, comprising:

[0169] The acquisition module is used to obtain the EEG data to be classified.

[0170] The denoising module is used to denoise the EEG data to be classified to obtain denoised EEG data to be classified.

[0171] The time series feature extraction module extracts the time series features of the EEG data to be classified after denoising, and obtains the time series features of the EEG data to be classified.

[0172] The emotional state classification module is used to input the time series features of the EEG data to be classified into the trained EEG emotional state classification model to determine the emotional state; the EEG emotional state classification model includes a feature extraction module, a residual multi-head cross-attention module, a gated multi-embedding module and a task tower classification module.

[0173] In an exemplary embodiment, a computer device is provided. The computer device may be a server or a terminal. The internal structure diagram thereof may be as follows: Figure 6 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store EEG data to be classified. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a multi-task EEG emotional state classification method based on residual cross attention is implemented.

[0174] Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components. In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0175] In an exemplary embodiment, a computer-readable storage medium is provided, storing a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0176] In an exemplary embodiment, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0177] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0178] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0179] The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. The non-relational database may include a distributed database based on blockchain, etc., but is not limited thereto. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but is not limited thereto.

[0180] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0181] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A multi-task EEG emotional state classification method based on residual cross-attention, characterized in that: The multi-task EEG emotional state classification method based on residual cross attention includes: Obtain EEG data to be classified; Performing denoising on the EEG data to be classified to obtain denoised EEG data to be classified; Extracting time series features from the denoised EEG data to be classified to obtain time series features of the EEG data to be classified; The temporal features of the EEG data to be classified are input into a trained EEG emotional state classification model to determine the emotional state; the EEG emotional state classification model includes a feature extraction module, a residual multi-head cross attention module, a gated multi-embedding module and a task tower classification module.

2. The multi-task EEG emotional state classification method based on residual cross-attention according to claim 1 is characterized in that: The training process of the EEG emotional state classification model is: Constructing a training set based on the DEAP database; the training set includes the temporal features of the sample EEG data and the emotional state corresponding to the sample EEG data; Construct an EEG emotional state classification model; Using the loss function, the temporal features of the sample EEG data are taken as input and the emotional state corresponding to the sample EEG data is taken as output to train the EEG emotional state classification model, thereby obtaining a trained EEG emotional state classification model.

3. The multi-task EEG emotional state classification method based on residual cross-attention according to claim 2 is characterized in that: Construct a training set, including: Get sample EEG data from the DEAP database; Performing denoising processing on the sample EEG data to obtain denoised sample EEG data; Extracting time series features from the denoised sample EEG data to obtain time series features of the sample EEG data; A training set is constructed based on the temporal features of the sample EEG data and the emotional state corresponding to the sample EEG data.

4. The multi-task EEG emotional state classification method based on residual cross-attention according to claim 1 is characterized in that: Inputting the temporal features of the EEG data to be classified into the trained EEG emotional state classification model to determine the emotional state, specifically including: Inputting the time series features of the EEG data to be classified into the feature extraction module to obtain a feature set; the feature set includes arousal features, valence features and shared features; Inputting the feature set into the residual multi-head cross attention module to obtain specific features; the specific features include arousal specific features and valence specific features; Inputting the specific features into the gated multi-embedding module to obtain the specific features after removing useless information; The specific features after removing useless information are input into the task tower classification module to determine the emotional state.

5. The multi-task EEG emotional state classification method based on residual cross-attention according to claim 4 is characterized in that: The feature extraction module includes: an arousal task expert unit, a valence task expert unit and a sharing task expert unit; The time series features of the EEG data to be classified are input into the feature extraction module to obtain a feature set, which specifically includes: Inputting the time series features of the EEG data to be classified into the arousal task expert unit to obtain arousal features; Inputting the time series features of the EEG data to be classified into the valence task expert unit to obtain valence features; The time series features of the EEG data to be classified are input into the shared task expert unit to obtain shared features.

6. The multi-task EEG emotional state classification method based on residual cross-attention according to claim 4 is characterized in that: The feature set is input into the residual multi-head cross attention module to obtain specific features, including: The valence feature and the arousal feature are respectively concatenated with the shared feature to obtain related features; the related features include a first related feature and a second related feature; A multi-head attention value is obtained based on the arousal feature, the valence feature and the related feature; the multi-head attention value includes a first multi-head attention value and a second multi-head attention value; Normalizing a result of adding the arousal feature and the first multi-head attention value to obtain an arousal specific feature; A result of adding the valence feature and the second multi-head attention value is normalized to obtain a valence specific feature.

7. The multi-task EEG emotional state classification method based on residual cross-attention according to claim 6 is characterized in that: The multi-head attention value is obtained based on the arousal feature, the valence feature and the related feature, specifically including: Using multiple projection matrices, perform multiple linear transformations on the arousal feature and the first related feature to obtain a first linear vector set; Using multiple projection matrices, multiple linear transformations are performed on the valence feature and the second correlation feature to obtain a second linear vector set; A cross-task multi-head attention value calculation is performed on the first linear vector set to obtain a first attention value set, and the first attention value set is spliced ​​to obtain a first multi-head attention value.

8. The multi-task EEG emotional state classification method based on residual cross-attention according to claim 4 is characterized in that: The specific features are input into the gated multi-embedding module to obtain the specific features after removing useless information, specifically including: Based on the multi-embedding paradigm, the specific features are linearly transformed to obtain an embedding result; Using a sigmoid activation function to activate the embedding result to obtain an activated embedding result; The embedding result and the activated embedding result are point-multiplied and linearly transformed to obtain specific features after removing useless information.

9. A multi-task EEG emotional state classification system based on residual cross-attention, characterized in that: The multi-task EEG emotional state classification method based on residual cross-attention according to any one of claims 1 to 8, wherein the multi-task EEG emotional state classification system based on residual cross-attention comprises: An acquisition module is used to obtain EEG data to be classified; A denoising module is used to perform denoising on the EEG data to be classified to obtain denoised EEG data to be classified; A time series feature extraction module is used to extract time series features from the denoised EEG data to be classified, so as to obtain time series features of the EEG data to be classified; The emotional state classification module is used to input the temporal features of the EEG data to be classified into the trained EEG emotional state classification model to determine the emotional state; the EEG emotional state classification model includes a feature extraction module, a residual multi-head cross-attention module, a gated multi-embedding module and a task tower classification module.

10. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement any of the multi-task EEG emotional state classification methods based on residual cross-attention according to claims 1-8.

Images