A multi-task EEG emotional state classification method, system and device based on residual cross attention

Abstract

This application discloses a multi-task EEG emotion state classification method, system, and device based on residual cross-attention, relating to the field of EEG signal emotion state recognition. The method includes: acquiring EEG data to be classified; denoising the EEG data to be classified to obtain denoised EEG data to be classified; extracting temporal features from the denoised EEG data to obtain temporal features of the EEG data to be classified; and inputting the temporal features of the EEG data to be classified into a trained EEG emotion state classification model to determine the emotion state. The EEG emotion state classification model includes a feature extraction module, a residual multi-head cross-attention module, a gated multi-embedding module, and a task tower classification module. This application can improve the accuracy of EEG emotion state classification.

Description

Technical Field

[0001] This application relates to the field of EEG signal emotion state recognition, and in particular to a multi-task EEG emotion state classification method, system and device based on residual cross-attention. Background Technology

[0002] Emotions are complex psychological and physiological responses to internal and external environmental stimuli, playing a crucial role in human-computer interaction and affective computing. Neuroscience research indicates that physiological signals can more directly reflect the source of emotions. Compared to behavioral signals such as facial expressions and speech, electroencephalograms (EEGs), as a non-invasive neurophysiological technique, have a particularly prominent advantage in that they are difficult to fake or hide. With the rapid development of non-invasive, cost-effective EEG recording devices, the application of EEG-based emotion recognition in deep learning is attracting increasing attention.

[0003] Currently, EEG-based emotion recognition models can be mainly divided into discrete models and dimensional models. Discrete models typically classify emotions into specific discrete states, such as happiness, sadness, and anger, while dimensional models analyze emotions based on different dimensions, such as valence (positive or negative) and arousal (emotional intensity). In recent years, emotion recognition models based on emotion dimensions have received widespread research attention and practical application. However, most current research focuses only on a single task, such as identifying only emotional valence or only arousal, failing to fully consider the correlation between different tasks and ignoring the potential information sharing between tasks, resulting in low accuracy in EEG emotion state classification. Summary of the Invention

[0004] The purpose of this application is to provide a multi-task EEG emotion state classification method, system, and device based on residual cross-attention, which can improve the accuracy of EEG emotion state classification.

[0005] To achieve the above objectives, this application provides the following solution:

[0006] Firstly, this application provides a multi-task EEG emotion state classification method based on residual cross-attention, including:

[0007] Obtain the EEG data to be classified;

[0008] The EEG data to be classified is denoised to obtain denoised EEG data to be classified.

[0009] Temporal features are extracted from the denoised EEG data to be classified to obtain the temporal features of the EEG data to be classified.

[0010] The temporal features of the EEG data to be classified are input into the trained EEG emotion state classification model to determine the emotion state; the EEG emotion state classification model includes a feature extraction module, a residual multi-head cross-attention module, a gated multi-embedding module, and a task tower classification module.

[0011] Optionally, the training process of the EEG emotion state classification model is as follows:

[0012] A training set is constructed based on the DEAP database; the training set includes the temporal features of the sample EEG data and the emotional states corresponding to the sample EEG data.

[0013] Construct an EEG emotion state classification model;

[0014] Using a loss function, the EEG emotion state classification model is trained by taking the temporal features of the sample EEG data as input and the corresponding emotion state of the sample EEG data as output.

[0015] Optionally, a training set is constructed, specifically including:

[0016] Retrieve sample EEG data from the DEAP database;

[0017] The sample EEG data is denoised to obtain denoised sample EEG data;

[0018] Temporal features are extracted from the denoised sample EEG data to obtain the temporal features of the sample EEG data;

[0019] A training set is constructed based on the temporal features of the sample EEG data and the corresponding emotional states of the sample EEG data.

[0020] Optionally, the temporal features of the EEG data to be classified are input into a trained EEG emotion state classification model to determine the emotion state, specifically including:

[0021] The temporal features of the EEG data to be classified are input into the feature extraction module to obtain a feature set; the feature set includes arousal features, valence features, and shared features.

[0022] The feature set is input into the residual multi-head cross-attention module to obtain specific features; the specific features include arousal-specific features and valence-specific features.

[0023] The specific feature is input into the gated multi-embedding module to obtain the specific feature after removing useless information;

[0024] The specific features after removing useless information are input into the task tower classification module to determine the emotional state.

[0025] Optionally, the feature extraction module includes: an arousal task expert unit, a valence task expert unit, and a sharing task expert unit;

[0026] Specifically, the time-series features of the EEG data to be classified are input into the feature extraction module to obtain a feature set, which includes:

[0027] The temporal features of the EEG data to be classified are input into the arousal task expert unit to obtain arousal features;

[0028] The temporal features of the EEG data to be classified are input into the valence task expert unit to obtain the valence features;

[0029] The temporal features of the EEG data to be classified are input into the shared task expert unit to obtain shared features.

[0030] Optionally, the feature set is input into the residual multi-head cross-attention module to obtain specific features, including:

[0031] The valence feature and the arousal feature are concatenated with the shared feature to obtain the relevant feature; the relevant feature includes a first relevant feature and a second relevant feature;

[0032] A multi-head attention value is obtained based on the arousal feature, the valence feature, and the correlation feature; the multi-head attention value includes a first multi-head attention value and a second multi-head attention value.

[0033] The result of adding the arousal feature and the first multi-head attention value is normalized to obtain the arousal-specific feature;

[0034] The result of adding the valence feature and the second multi-head attention value is normalized to obtain the valence-specific feature.

[0035] Optionally, a multi-head attention value is obtained based on the arousal feature, the valence feature, and the correlation feature, specifically including:

[0036] Multiple projection matrices are used to perform multiple linear transformations on the arousal feature and the first related feature to obtain a first set of linear vectors.

[0037] Multiple projection matrices are used to perform multiple linear transformations on the valence feature and the second correlation feature to obtain a second set of linear vectors.

[0038] Perform cross-task multi-head attention value calculation on the first linear vector set to obtain a first attention value set, and concatenate the first attention value set to obtain a first multi-head attention value.

[0039] Optionally, the specific features are input into the gated multi-embedding module to obtain the specific features after removing useless information, specifically including:

[0040] Based on multiple embedding paradigms, a linear transformation is performed on the specific features to obtain the embedding result;

[0041] The embedding result is activated using the sigmoid activation function to obtain the activated embedding result;

[0042] The embedding result and the activated embedding result are multiplied by a dot product and then linearly transformed to obtain specific features after removing useless information.

[0043] Secondly, this application provides a multi-task EEG emotion state classification system based on residual cross-attention, used to implement the multi-task EEG emotion state classification method based on residual cross-attention described above. The multi-task EEG emotion state classification system based on residual cross-attention includes:

[0044] Get the module, used to acquire the EEG data to be classified;

[0045] A denoising module is used to denoise the EEG data to be classified, so as to obtain denoised EEG data to be classified.

[0046] The temporal feature extraction module extracts temporal features from the denoised EEG data to be classified, thereby obtaining the temporal features of the EEG data to be classified.

[0047] An emotion state classification module is used to input the temporal features of the EEG data to be classified into a trained EEG emotion state classification model to determine the emotion state; the EEG emotion state classification model includes a feature extraction module, a residual multi-head cross-attention module, a gated multi-embedding module, and a task tower classification module.

[0048] Thirdly, this application provides a computer device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of any of the above-described multi-task EEG emotion state classification methods based on residual cross-attention.

[0049] According to the specific embodiments provided in this application, this application has the following technical effects:

[0050] This application provides a multi-task EEG emotion state classification method, system, and device based on residual cross-attention. The method classifies EEG data to be classified using a trained EEG emotion state classification model to determine the emotion state. The EEG emotion state classification model includes a feature extraction module, a residual multi-head cross-attention module, a gated multi-embedding module, and a task tower classification module. The feature extraction module extracts shared features from the EEG data to be classified. The residual multi-head cross-attention module learns the correlation between different tasks and dynamically adjusts the attention weights, explicitly capturing shared and specific information between tasks, promoting complementarity and sharing between tasks. The gated multi-embedding module reduces the influence of useless information, thereby reducing the negative impact on task performance. Therefore, this application improves the accuracy of EEG emotion state classification through its EEG emotion state classification model. Attached Figure Description

[0051] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0052] Figure 1 A flowchart illustrating a multi-task EEG emotion state classification method based on residual cross-attention, provided as an embodiment of this application;

[0053] Figure 2 A schematic diagram illustrating the framework of a multi-task EEG emotion state classification method based on residual cross-attention, provided in an embodiment of this application;

[0054] Figure 3 This is a schematic diagram of the structure of an EEG emotion state classification model provided in an embodiment of this application;

[0055] Figure 4 This is a schematic diagram of the structure of a residual multi-head cross-attention module provided in an embodiment of this application;

[0056] Figure 5 This is a schematic diagram of the structure of a gated multi-embedded module provided in an embodiment of this application;

[0057] Figure 6 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0058] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0059] Currently, constructing a general multi-task model still faces the following problems: 1. Existing multi-task models typically rely on a unified shared model. However, when the correlation between tasks is weak or the tasks are complex, this excessive sharing may lead to improper knowledge transfer, thereby affecting task performance. 2. Task-shared feature extraction in existing models is often implicit, failing to fully consider the explicit correlation between tasks, ignoring dependencies or feature interactions between tasks, and limiting the effectiveness of feature learning. 3. Existing multi-task models typically do not perform harmful feature filtering after feature fusion, resulting in the fused features potentially containing features useless for specific tasks, thus affecting the model's adaptability and performance. These problems ultimately lead to low accuracy in emotion state classification. Therefore, this application provides a multi-task EEG emotion state classification method, system, and device based on residual cross-attention, which can improve the accuracy of EEG emotion state classification.

[0060] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0061] In one exemplary embodiment, such as Figure 1-Figure 2 As shown, a multi-task EEG emotion state classification method based on residual cross-attention is provided. This method includes the following steps:

[0062] Step 1: Obtain the EEG data to be classified.

[0063] Step 2: Denoise the EEG data to be classified to obtain the denoised EEG data to be classified.

[0064] Step 3: Extract temporal features from the denoised EEG data to be classified to obtain the temporal features of the EEG data to be classified.

[0065] Step 4: Input the temporal features of the EEG data to be classified into the trained EEG emotion state classification model to determine the emotion state; the EEG emotion state classification model includes a feature extraction module, a residual multi-head cross-attention module, a gated multi-embedding module, and a task tower classification module.

[0066] In a specific embodiment, the training process of the EEG emotion state classification model specifically includes:

[0067] (1) A training set was constructed based on the DEAP database. The training set includes the temporal features of the sample EEG data and the emotional state corresponding to the sample EEG data.

[0068] (2) Construct an EEG emotion state classification model.

[0069] (3) Using a loss function, with the temporal features of each sample's EEG data as input and the emotional state corresponding to the sample's EEG data as output, the EEG emotional state classification model is trained to obtain a well-trained EEG emotional state classification model, specifically including:

[0070] Initialize the EEG sentiment state classification model;

[0071] The EEG sentiment state classification model is trained multiple times using the training set to obtain a trained EEG sentiment state classification model; wherein the training process in any current round includes:

[0072] The temporal features of the sample EEG data are input into the EEG sentiment state classification model for the current round to obtain the predicted sentiment state for the current round.

[0073] Using a loss function, the loss for the current round is calculated based on the predicted sentiment state and the sentiment state corresponding to the sample EEG data.

[0074] Determine whether the stopping condition is met; the stopping condition is that a preset training round has been reached, the loss in the current round is less than the preset loss, or the absolute value of the difference between the loss in the current round and the loss in the previous round is less than a preset absolute value.

[0075] If not, update the parameters of the EEG sentiment classification model for the current round and return "Input the temporal features of the sample EEG data into the EEG sentiment classification model for the current round to obtain the predicted sentiment state for the current round".

[0076] If so, the EEG sentiment classification model for the current round will be determined as the trained EEG sentiment classification model.

[0077] Specifically, due to the varying difficulty and pace of training different tasks, the overall objective loss function is typically optimized to ensure that not only the performance of a single task is improved, but that good results are achieved across all tasks. The overall objective loss function is usually defined as a weighted linear combination of the losses (Loss, L) of each task. To address the different variances and biases between individual task losses, this application introduces an automatically weighted loss function, resulting in the following overall objective loss function for the model: The specific formula is as follows:

[0078]

[0079] Where x is the sample; σ is the different losses used by adaptive learning to weight each task; For predicting labels; y T ={y a ,y b} represents the actual label; w T Category weights; The loss function for wakefulness; The loss function for the valence task; σ a for The relative weights to be learned adaptively; σ b for The relative weights that should be learned adaptively; This is a regularization term.

[0080] The loss function for a single task can be any type of loss function, such as cross-entropy, KL-div, focal loss, etc.

[0081] The overall objective loss function defined above is used when training the EEG sentiment state classification model. Where x is the sample, y T For the corresponding multitasking tags, For the multi-task results predicted by the model, w T These are the class weights. The training process uses an adaptive time estimation method to perform gradient descent. Specifically, batch training is used, iteratively adjusting the model parameters to optimize the overall loss function. Decrease until the loss converges or the preset number of iterations is reached.

[0082] In one specific embodiment, constructing the training set specifically includes:

[0083] 1) Obtain the raw EEG signal (i.e. sample EEG data) from the DEAP database.

[0084] Specifically, the dataset used to train the model performance in this application comes from DEAP, and can be found in the paper Koelstra S, Mühl C, 0001S M, et al. DEAP: A Database for Emotion Analysis; Using Physiological Signals. [J]. IEEE Trans. Affective Computing, 2012, 3(1): 18-31.

[0085] 2) The sample EEG data is denoised to obtain denoised sample EEG data.

[0086] Specifically, the sampling rate of the raw EEG signals acquired from the dataset was reduced from 512Hz to 128Hz through preprocessing, and artifacts were removed using a bandpass frequency filter of 4.0Hz–45.0Hz.

[0087] 3) Extract time-series features from the denoised sample EEG data to obtain the time-series features of the sample EEG data.

[0088] Specifically, temporal features were extracted from the denoised EEG data. Each subject was segmented using a 1-second sliding window, dividing each signal into 60 segments, with 128 sampling points per EEG segment. Since each subject underwent 40 experiments, approximately 2400 samples (40 experiments × 60 segments) were obtained per subject. With 32 subjects, a total of 76,800 temporal features were extracted.

[0089] 4) Construct a training set based on the temporal features of the sample EEG data and the emotional states corresponding to the sample EEG data.

[0090] Specifically, suppose there are N subjects, denoted by U, and the data set of each subject is denoted by... Where s represents the subjects, and n represents the number of subjects in a sample. Let represent the i-th sample of subject s, with labels for tasks a and b respectively.

[0091] This application employs a cross-subject approach and uses a leave-one-out method for dataset partitioning. Specifically, first, all emotional test data of the first subject are taken, assuming their emotional state labels are unknown, as the test set; then, all emotional test data of the remaining N-1 subjects are taken, assuming their emotional state labels are known, as the training set. Next, the emotional test data of the i-th subject is sequentially taken as the test set (i = 2, 3, ..., N), and the emotional test data of the remaining N-1 subjects are taken as the training set.

[0092] For experimental scenarios involving multiple participants: the test set data input is... in Let represent the i-th sample of subject t, and there are n samples in total; the training set input of the model is... s∈Ut, where, Let represent the i-th sample of subject s. The arousal label represents the i-th sample of subject s. U represents the valence label of the i-th sample of subject s. Each subject has n samples. Ut represents the set of all subjects except t. The model does not obtain any information about the samples from the test set. Tag information.

[0093] In one specific embodiment, step 4 specifically includes:

[0094] Step 41: Input the extracted EEG data to be classified into the feature extraction module to obtain the feature set; the feature set includes arousal features, valence features and shared features.

[0095] Step 42: Input the feature set into the residual multi-head cross-attention module to obtain specific features; the specific features include arousal-specific features and valence-specific features.

[0096] Step 43: Input the specific features into the gated multi-embedding module to obtain the specific features after removing useless information.

[0097] Step 44: Input the specific features after removing useless information into the task tower classification module to determine the emotional state.

[0098] In one specific embodiment, the feature extraction module includes: an arousal task expert unit, a valence task expert unit, and a sharing task expert unit. Step 41 specifically includes:

[0099] Step 411: Input the temporal features of the EEG data to be classified into the arousal task expert unit to obtain arousal features;

[0100] Step 412: Input the time-series features of the EEG data to be classified into the valence task expert unit to obtain the valence features;

[0101] Step 413: Input the time-series features of the EEG data to be classified into the shared task expert unit to obtain the shared features.

[0102] Specifically, the feature extraction module primarily learns task-specific features through its own task-specific experts and learns shared features through shared task experts. The feature extraction module includes: an arousal task expert unit (f... a ), Valence Task Expert Unit (f b ) and shared task expert unit (f s The three models use the same network model structure, but can be any type of network model (such as dynamic graph convolutional neural networks, shallow convolutional neural networks, deep convolutional neural networks, etc.). Specific arousal features (F) are extracted from the training set of each model using three expert feature extraction models. a ), valence characteristics (F) b ) and shared features (F s ). Specifically, as follows Figure 3 As shown on the left, its process can be described as follows:

[0103] F a =fa (x),F b =f b (x),F s =f s (x)(2)

[0104] Where, x∈R n×c×t As input to the task model, n is the number of samples, c is the number of channels, and t is time.

[0105] In one specific embodiment, step 42 specifically includes:

[0106] Step 421: Concatenate the valence feature and arousal feature with the shared feature to obtain the relevant features; the relevant features include the first relevant feature and the second relevant feature.

[0107] Step 422: Obtain multi-head attention values ​​based on arousal features, valence features, and related features; the multi-head attention values ​​include the first multi-head attention value and the second multi-head attention value.

[0108] Step 423: Normalize the sum of the arousal feature and the first multi-head attention value to obtain the arousal-specific feature.

[0109] Step 424: Normalize the result of adding the valence feature and the second multi-head attention value to obtain the valence-specific feature.

[0110] Specifically, the Residual Multi-head Cross Attention module (RMCA module) explicitly captures task features (F... a ,F b ,F s By leveraging the shared and unique information between the two sets of task features {F}, we can uncover the correlations between the main features and other features, and then extract beneficial features from the other features to supplement the deficiencies of the main features, thereby improving the effectiveness of the main features in their classification tasks. Finally, we will combine the two sets of task features {F} a ,(F b ,F s )},{F b ,(F a ,F s The corresponding task-specific features are obtained by fusing features through residual multi-head cross-attention modules.

[0111] Based on specific characteristics of valence For example, specifically as follows Figure 4 As shown, if the valence task (B) is taken as the key task, then the key feature of the module is F. b Other features are Fa and F s F b X, as a residual multi-head cross-attention module b , will F a and F s Perform the splicing operation and use it as X in the residual multi-head cross-attention module. c The specific formula is as follows:

[0112] X b =F b ,X c =Concat(F a ,F s (3)

[0113] X b X c The features are linearly transformed to generate corresponding query, key, and value vectors. Specifically, the features X of key task B are transformed... b Generate query vector Q b The features X of other tasks C c Generate key vector K respectively c Value vector V c The specific formula is as follows:

[0114] Q b =W q,b X b ,K c =W k,c X c V c =W v,c X c (4)

[0115] Among them, W q,b To query the projection matrix for mission-critical task B, W k,c W is the key projection matrix used for other tasks C; v,c The projection matrix is ​​used for the values ​​of C in other tasks.

[0116] Calculate the query vector Q for critical task B. b Key vector K of other tasks C c The cross-task attention value is used to incorporate meaningful information from other tasks C into the critical task B. The specific formula is as follows:

[0117]

[0118] Where, d k The dimension of the key vector is used to scale the dot product to avoid the dot product value becoming too large as the dimension increases.

[0119] To extract diverse features from different subspaces, multi-head attention is used to focus on different feature subspaces. This is achieved through h different projection matrices. For feature X b ,X c Projection is performed to obtain h distinct query, key, and value vectors. Then, parallel cross-task attention operations are performed on each group to obtain h independent attention heads. Finally, the outputs of all heads are concatenated and mapped back to the original dimensional space, as shown in the following formula:

[0120]

[0121] MultiHead(Q b ,K c V c ) = W o Concat(head1,head2,…,head h (7)

[0122] Among them, head i W is the i-th attention head; o MultiHead(Q) is a projection matrix used to map the stitched result back to the original dimensional space. b ,K c V c () represents the multi-head attention value.

[0123] To preserve the original information of the input features, and to allow for effective learning of deep features while mitigating the gradient vanishing problem, the input features X of task B are... b The result is added to the output of the multi-head attention algorithm to form a residual connection, and then normalized to ensure that the mean is 0 and the variance is 1. The specific formula is as follows:

[0124]

[0125] Similarly, when the arousal task (A) is taken as the key task, the key feature is F. a Other features are F b and F s , will F b and F s F is obtained by splicing. d and for F d F a Conduct and The same operation to obtain

[0126]

[0127] where X a For the input features of task A, Q is a specific feature for arousal. a Let K be the query vector for task A. d V is the key vector for other tasks. d For other tasks, the value vector is used.

[0128] In one specific embodiment, step 422 specifically includes:

[0129] Step 4221: Using multiple projection matrices, perform multiple linear transformations on the arousal feature and the first related feature to obtain a first set of linear vectors.

[0130] Step 4222: Using multiple projection matrices, perform multiple linear transformations on the valence feature and the second correlation feature to obtain a second set of linear vectors.

[0131] Step 4223: Perform cross-task multi-head attention value calculation on the first linear vector set to obtain the first attention value set, and concatenate the first attention value set to obtain the first multi-head attention value.

[0132] In one specific embodiment, step 43 specifically includes:

[0133] Step 431: Based on the multiple embedding paradigm, perform a linear transformation on the specific feature to obtain the embedding result.

[0134] Step 432: Activate the embedding result using the sigmoid activation function to obtain the activated embedding result.

[0135] Step 433: Perform a dot product on the embedding result and the activated embedding result, and then perform a linear transformation to obtain the specific features after removing useless information.

[0136] Specifically, the Gated Multi Embedding Module (GME module) removes features through a gating mechanism. Information useless for specific tasks is extracted, thus reducing the negative impact of this information on task performance. As the embedding dimension increases, the amount of information embedded does not significantly improve; the embedding matrix still exhibits low-rank characteristics. Therefore, simply increasing the model size does not effectively improve information representation ability and may even lead to information redundancy. Therefore, using multiple embeddings instead of single embeddings, combined with a gating mechanism, can obtain richer feature information from multiple dimensions. Ultimately, this will... Harmful features are removed using their respective GME modules to obtain...

[0137] Valence-specific features after removing useless information For example, Figure 5 As shown, the valence task (B) is the key task. The input to the GME module is used, which contains two Multi-Embedding components. Unlike simply increasing the embedding size, Multi-Embedding extracts richer feature information from multiple dimensions by adding multiple independent and complete embedding layers. In Multi-Embedding, the first step is to... Multiple linear transformations are performed, each generating an independent embedding feature. All embedding features are then concatenated. The specific formula is as follows:

[0138]

[0139] in, b is the weight matrix of the i-th embedding layer; i This is the bias value for the i-th embedding layer; Let E be the output of the i-th embedding layer; Multi-Embedding represents the final embedding result. b ∈R c×r , Where c is The dimension of the feature is m, which is the preset number of embedding layers, and r is the preset expansion ratio.

[0140] To better filter out useless information, a multi-embedding layer is processed using a sigmoid function to highlight useful features. Then, the processed embedding layer is multiplied by another multi-embedding layer to further filter out key information. Finally, a linear layer is applied to obtain... The specific formula is as follows:

[0141]

[0142] Among them, W E,b Output the weight matrix of the linear layer for task B; b E,b Output the bias values ​​for the linear layer for task B. σ represents the matrix dot product operation, and σ represents the sigmoid activation function.

[0143] Similarly, when we consider wakefulness task (A) as the key task, the specific formula is as follows:

[0144]

[0145] in, A specific feature of arousal after removing useless information.

[0146] In one specific embodiment, step 44 specifically includes:

[0147] The task tower classification module comprises two independent task towers, each responsible for predicting a specific task: arousal and valence. Features extracted from GME will be used. The data is input into their respective task towers for classification to obtain prediction results. The prediction results include arousal prediction results. and valence prediction results Right now

[0148] like Figure 3 As shown on the right, this application extracts features from GME. The data is input into their respective task towers for classification to obtain arousal and valence prediction results.

[0149]

[0150] Among them, classfiers a For wakefulness classifiers; b For valence classifier; This is the result of the arousal prediction. This represents the valence prediction results.

[0151] In one specific embodiment, after training the EEG emotion state classification model, this application uses the test set used during the construction of the training set to verify the accuracy of the trained EEG emotion state classification model. Specifically, this includes: inputting the test set into the trained model and obtaining the prediction results of the test set. The actual state y displayed by the labels in the test set T Compare the results to evaluate the model's performance. Accuracy is the percentage of correctly classified samples out of the total test samples. The formula for calculating the accuracy for each subject in the test set is as follows:

[0152]

[0153] Where TP represents positive samples predicted as positive by the model, TN represents negative samples predicted as negative by the model, FP represents negative samples predicted as positive by the model, and FN represents positive samples predicted as negative by the model. The DEAP dataset includes 32 participants, and the average accuracy is shown below:

[0154]

[0155] The formula for the mean square error of the results is shown below:

[0156]

[0157] The base models were tested, including: Dynamic Graph Convolutional Neural Networks (DGCNN), Shallow Convolutional Neural Network (ShallowConvNet), Deep Convolutional Neural Network (DeepConvNet), and A Compact Convolutional Neural Network for EEG-based Brain-Computer Interfaces (EEGNet). The final test results show the performance improvement of the base models based on the average accuracy.

[0158] Table 1 shows the performance improvement of the basic model.

[0159]

[0160] As can be seen from Table 1, all the base models (EEGNet, DGCNN, DeepConvNet and ShallowConvNet) have improved in valence and arousal tasks, indicating that the present application can significantly improve the performance of the models in these tasks.

[0161] To compare the advantages of the EEG emotion state classification model in this application, similar experiments were conducted with other methods and compared with state-of-the-art technologies, including: Hybrid Unsupervised Deep Feature Characterization and Fusion for High-Dimensional EEG With an Application to Emotion Recognition (EEGFuseNet); WTCNN from the paper "EEG emotion recognition based on the attention mechanism and pre-trained convolution capsule network"; HLST from the paper "Transformers for EEG-Based Emotion Recognition: A Hierarchical Spatial Information Learning Model"; STFG-CAP from the paper "Capsule neural networks on spatio-temporal EEG frames for cross-subject emotion recognition"; TMLP+SRDANN from the paper "TMLP+SRDANN: A domain adaptation method for EEG-based emotion recognition"; and EEG emotion recognition based on ontheattention mechanism and pre-trained convolution capsule network. The comparison is shown in Table 2: AP-CapsNet from the paper "EEG-Based Emotion Recognition Using Trainable AdjacencyRelation Driven Graph Convolutional Network".

[0162] Table 2 Comparison of the accuracy of the most advanced technologies currently available

[0163]

[0164] As can be seen from the results in Table 2 above, the method proposed in this application achieves higher accuracy and lower variance, demonstrating a significant advantage in multi-task emotion classification.

[0165] This application is not only applicable to multi-task research on emotion state recognition, but also to any multi-task classification prediction based on EEG. It solves the problem of individual differences in EEG to a certain extent and has the advantages of low time complexity, high computational efficiency and strong generalization ability.

[0166] This application proposes a multi-task EEG emotion classification method based on a residual cross-attention mechanism. This method utilizes specific and shared expert models to learn task-specific and shared features to mitigate negative transfer between shared and specific knowledge. It employs a residual cross-attention module (RMCA) to learn the correlation between different tasks and dynamically adjust attention weights, explicitly capturing shared and specific information between tasks and promoting task complementarity and sharing. A gated multi-head embedding module (GME) is used to reduce the influence of harmful or useless features, and a task pyramid classification module combined with a loss weight self-adjustment mechanism balances the losses of different tasks to achieve emotion classification. Furthermore, MT-RCAF improves the performance of all base models (EEGNet, DGCNN, DeepConvNet, and ShallowConvNet) to varying degrees. Compared with state-of-the-art models, MT-RCAF (EEGNet) significantly outperforms other models in emotion classification performance.

[0167] Based on the same inventive concept, this application also provides a multi-task EEG emotion state classification system based on residual cross-attention. The solution provided by this system is similar to the solution described in the above method. Therefore, the specific limitations of one or more embodiments of the multi-task EEG emotion state classification system based on residual cross-attention provided below can be found in the limitations of the multi-task EEG emotion state classification method based on residual cross-attention described above, and will not be repeated here.

[0168] In one exemplary embodiment, a multi-task EEG emotion state classification system based on residual cross-attention is provided, comprising:

[0169] The module is used to acquire the EEG data to be classified.

[0170] The denoising module is used to denoise the EEG data to be classified, resulting in denoised EEG data to be classified.

[0171] The temporal feature extraction module extracts temporal features from the denoised EEG data to be classified, thus obtaining the temporal features of the EEG data to be classified.

[0172] The emotion state classification module is used to input the temporal features of the EEG data to be classified into the trained EEG emotion state classification model to determine the emotion state. The EEG emotion state classification model includes a feature extraction module, a residual multi-head cross-attention module, a gated multi-embedding module, and a task tower classification module.

[0173] In one exemplary embodiment, a computer device is provided, which may be a server or a terminal, and its internal structure diagram may be as follows. Figure 6 As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs in the non-volatile storage media to run. The database stores EEG data to be classified. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When executed by the processor, the computer program implements a multi-task EEG emotion state classification method based on residual cross-attention.

[0174] Those skilled in the art will understand that Figure 6 The structures shown are merely block diagrams of some structures related to the present application and do not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than shown in the figures, or combine certain components, or have different component arrangements. In an exemplary embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0175] In one exemplary embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0176] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0177] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0178] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0179] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0180] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0181] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A multi-task EEG emotion state classification method based on residual cross-attention, characterized in that, The multi-task EEG emotion state classification method based on residual cross-attention includes: Obtain the EEG data to be classified; The EEG data to be classified is denoised to obtain denoised EEG data to be classified. Temporal features are extracted from the denoised EEG data to be classified to obtain the temporal features of the EEG data to be classified. The temporal features of the EEG data to be classified are input into a trained EEG emotion state classification model to determine the emotion state; the EEG emotion state classification model includes a feature extraction module, a residual multi-head cross-attention module, a gated multi-embedding module, and a task tower classification module; The temporal features of the EEG data to be classified are input into a trained EEG emotion state classification model to determine the emotion state, specifically including: The temporal features of the EEG data to be classified are input into the feature extraction module to obtain a feature set; the feature set includes arousal features, valence features, and shared features. The feature set is input into the residual multi-head cross-attention module to obtain specific features; the specific features include arousal-specific features and valence-specific features. The specific feature is input into the gated multi-embedding module to obtain the specific feature after removing useless information; The specific features after removing useless information are input into the task tower classification module to determine the emotional state; The feature extraction module includes: an arousal task expert unit, a valence task expert unit, and a sharing task expert unit; Specifically, the time-series features of the EEG data to be classified are input into the feature extraction module to obtain a feature set, which includes: The temporal features of the EEG data to be classified are input into the arousal task expert unit to obtain arousal features; The temporal features of the EEG data to be classified are input into the valence task expert unit to obtain the valence features; The temporal features of the EEG data to be classified are input into the shared task expert unit to obtain shared features; Specifically, the feature set is input into the residual multi-head cross-attention module to obtain specific features, including: The valence feature and the arousal feature are concatenated with the shared feature to obtain the relevant feature; the relevant feature includes a first relevant feature and a second relevant feature; A multi-head attention value is obtained based on the arousal feature, the valence feature, and the correlation feature; the multi-head attention value includes a first multi-head attention value and a second multi-head attention value. The result of adding the arousal feature and the first multi-head attention value is normalized to obtain the arousal-specific feature; The result of adding the valence feature and the second multi-head attention value is normalized to obtain the valence-specific feature; Specifically, the multi-head attention value is obtained based on the arousal feature, the valence feature, and the correlation feature, including: Multiple projection matrices are used to perform multiple linear transformations on the arousal feature and the first related feature to obtain a first set of linear vectors. Multiple projection matrices are used to perform multiple linear transformations on the valence feature and the second correlation feature to obtain a second set of linear vectors. Perform cross-task multi-head attention value calculation on the first linear vector set to obtain a first attention value set, and concatenate the first attention value set to obtain a first multi-head attention value; Specifically, the specific features are input into the gated multi-embedding module to obtain the specific features after removing useless information, including: Based on multiple embedding paradigms, a linear transformation is performed on the specific features to obtain the embedding result; The embedding result is activated using the sigmoid activation function to obtain the activated embedding result; The embedding result and the activated embedding result are multiplied by a dot product and then linearly transformed to obtain specific features after removing useless information; The task tower classification module includes task towers for arousal tasks and task towers for valence tasks: specific features extracted from the gated multi-embedding module after removing useless information are input into their respective task towers for classification to obtain prediction results and determine emotional states.

2. The multi-task EEG emotion state classification method based on residual cross-attention according to claim 1, characterized in that, The training process of the EEG emotion state classification model is as follows: A training set is constructed based on the DEAP database; the training set includes the temporal features of the sample EEG data and the emotional states corresponding to the sample EEG data. Construct an EEG emotion state classification model; Using a loss function, the EEG emotion state classification model is trained by taking the temporal features of the sample EEG data as input and the corresponding emotion state of the sample EEG data as output.

3. The multi-task EEG emotion state classification method based on residual cross-attention according to claim 2, characterized in that, Constructing the training set specifically includes: Retrieve sample EEG data from the DEAP database; The sample EEG data is denoised to obtain denoised sample EEG data; Temporal features are extracted from the denoised sample EEG data to obtain the temporal features of the sample EEG data; A training set is constructed based on the temporal features of the sample EEG data and the corresponding emotional states of the sample EEG data.

4. A multi-task EEG emotion state classification system based on residual cross-attention, characterized in that, The multi-task EEG emotion state classification method based on residual cross-attention, applied to any one of claims 1-3, wherein the multi-task EEG emotion state classification system based on residual cross-attention comprises: Get the module, used to acquire the EEG data to be classified; A denoising module is used to denoise the EEG data to be classified, so as to obtain denoised EEG data to be classified. The temporal feature extraction module extracts temporal features from the denoised EEG data to be classified, thereby obtaining the temporal features of the EEG data to be classified. An emotion state classification module is used to input the temporal features of the EEG data to be classified into a trained EEG emotion state classification model to determine the emotion state; the EEG emotion state classification model includes a feature extraction module, a residual multi-head cross-attention module, a gated multi-embedding module, and a task tower classification module.

5. A computer device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement any one of the multi-task EEG emotion state classification methods based on residual cross-attention as claimed in claims 1-3.

Images