A dual-stream adaptive convolutional recurrent hybrid EEG emotion recognition method combined with attention mechanism

Abstract

The present invention relates to the technical field of electroencephalogram emotion recognition, and discloses a dual-stream adaptive convolutional cyclic hybrid electroencephalogram emotion recognition method combined with an attention mechanism. The method assigns corresponding weights to electroencephalogram channels through a CSAM module in a dual-stream adaptive convolutional cyclic hybrid network model combined with an attention mechanism; then uses an MHSA-CRNN module and an SA-CRNN module to extract local spatiotemporal features; then splices the local features extracted by the MHSA-CRNN module and the SA-CRNN module and inputs them into an MHSA-TCN module to extract global information, and uses the extracted electroencephalogram information for emotion classification. The present invention performs binary and ternary classification experiments on a DEAP dataset. For the arousal dimension and the valence dimension, the binary classification accuracy is 99.26% and 99.15%, respectively, and the ternary classification accuracy is 97.69% and 98.05%, respectively, which is better than related algorithms.

Description

Technical Field

[0001] The present invention relates to the technical field of electroencephalogram (EEG) emotion recognition, and in particular to a dual-stream adaptive convolutional cycle hybrid EEG emotion recognition method combined with an attention mechanism. Background Art

[0002] Emotion is a complex state of the human brain when it is stimulated by various sensations, behaviors, and thoughts, and it plays a vital role in shaping the present mental state and guiding behavioral decisions. Advances in emotion recognition systems have become a prominent focus of disciplines such as psychology, artificial intelligence, computer vision, consumer behavior analysis, and medical treatment. Understanding human emotions is a complex task that involves analyzing non-physiological signals and physiological signals. Facial expressions, intonation, and text messages are non-physiological signals used to interpret emotions. Signals such as brain activity, muscle movement, and skin reactions are physiological signals that can provide a deeper understanding of the present emotional state. People can mask their true emotions, which makes it tricky to accurately predict their emotions based solely on the non-physiological signals they emit. In contrast, physiological signals such as electroencephalogram (EEG) can provide a more reliable understanding of a person's emotions. EEG excels in capturing emotions in real time with high accuracy and reasonable cost, and it has become a common method for interpreting emotions without invading others' privacy;

[0003] However, the existing methods only use a single network model, and the recognition accuracy needs to be improved. In order to further improve the accuracy of EEG emotion recognition, researchers have been trying to combine CNN, RNN, LSTM, GCNN and attention mechanisms in a hybrid network model in an innovative way. Zong et al. introduced the FCAN-XGBoost algorithm, which mixed FCAN and XGBoost algorithms to process DEAP and dream data sets with an accuracy of 95.26% and 94.05%, respectively. Chakravarthi et al. developed an automated system based on the ResNet-152 algorithm using a combination of CNN and LSTM, with an accuracy of 98%. Although the above methods have achieved a series of satisfactory results, there is still a problem of information loss in the process of extracting features using deep neural network models, which also means that the accuracy of EEG emotion recognition still has room for improvement; in addition, there is a large amount of redundant information in EEG signals that is not related to emotions, which leads to the inefficiency of emotion recognition models. Summary of the invention

[0004] 1. Technical issues to be resolved

[0005] In view of the shortcomings of the prior art, the present invention provides a dual-stream adaptive convolutional cyclic hybrid EEG emotion recognition method combined with an attention mechanism, which has the advantages of accurate recognition and solves the above-mentioned technical problems.

[0006] (II) Technical solution

[0007] To achieve the above object, the present invention provides the following technical solution: a dual-stream adaptive convolution cycle hybrid EEG emotion recognition method combined with an attention mechanism, comprising the following steps:

[0008] S1. Collect multimodal emotion datasets and use them as input data, and preprocess the input data;

[0009] S2. Construct a two-stream adaptive convolutional cyclic hybrid network model combined with an attention mechanism, wherein the two-stream adaptive convolutional cyclic hybrid network model combined with an attention mechanism includes a CSAM module, a SA-CRNN module, a MHSA-CRNN module, and a MHSA-TCN module;

[0010] S3, dividing the input data in step S1 into a training set and a test set in a ratio of 8:2, and training the two-stream adaptive convolutional recurrent hybrid network model combined with the attention mechanism in step S2;

[0011] S4. Identify EEG emotions through the dual-stream adaptive convolutional recurrent hybrid network model combined with the attention mechanism trained in step S3.

[0012] As a preferred technical solution of the present invention, the specific steps of data preprocessing in step S1 include:

[0013] S1.1, removing the baseline of the EEG signal in the multimodal emotion dataset to obtain the EEG signal after removing the baseline;

[0014] S1.2. Process the emotion labels in the multimodal emotion dataset.

[0015] As a preferred technical solution of the present invention, the calculation expression for removing the baseline in step S1.1 is as follows:

[0016]

[0017] s=s original -S baseline

[0018] Among them, s baseline is the average baseline signal, S original is the original EEG signal, s is the signal after removing the baseline, and x i represents the baseline signal at the i-th second.

[0019] As a preferred technical solution of the present invention, the CSAM module is used to allocate weights in the channel dimension, and includes a global pooling layer, a linear layer, a convolutional layer and an activation layer;

[0020] The SA-CRNN module is used to extract EEG emotion information;

[0021] The MHSA-CRNN module is composed of a convolutional recurrent neural network CRNN and a multi-head self-attention mechanism MHSA. The convolutional recurrent neural network CRNN in the MHSA-CRNN module first extracts information of the spatial dimension through the convolutional recurrent neural network CRNN, and then uses LSTM to extract the temporal dependency of the signal;

[0022] The MHSA-TCN module consists of a multi-head self-attention mechanism MHSA fused into a temporal convolutional network TCN;

[0023] The specific workflow of the two-stream adaptive convolutional recurrent hybrid network model combined with the attention mechanism is as follows:

[0024] S2.1. For the input data, the CSAM module assigns corresponding weights to the EEG channels, and sends the data to the MHSA-CRNN module and the SA-CRNN module respectively;

[0025] S2.2, extract local spatiotemporal features using MHSA-CRNN module and SA-CRNN module;

[0026] S2.3, then the local features extracted by the MHSA-CRNN module and the SA-CRNN module are concatenated and input into the MHSA-TCN module to extract the global information;

[0027] S2.4. Use the information extracted in step S2.3 for sentiment classification.

[0028] As a preferred technical solution of the present invention, the specific expression of the convolutional recurrent neural network CRNN in the MHSA-CRNN module is as follows:

[0029] v st =LSTM(Conv(x))

[0030] Where x is the input signal, v st is the feature vector output by the CRNN network, LSTM(*) indicates the use of LSTM to extract signals, and Conv(x) indicates the convolution operation on the input signal x;

[0031] The specific calculation expression of the multi-head self-attention mechanism MHSA is as follows:

[0032]

[0033] MHSA(Q,K,V)=Concat(h1,...,h8)W O

[0034] Among them, Q, K, V represent the features of query, key and value respectively, SA(Q, K, V) represents the output of self-attention, It means that the product of the Q, K, V values ​​of the nth self-attention head and its corresponding transformation matrix is ​​used to perform the operation of the first formula, h n represents the nth head attention output, MHSA(Q,K,V) represents the output of the multi-head self-attention mechanism MHSA, are the transformation matrices of the query, key, and value of the nth head, respectively. Concat(*) represents the concatenation of different feature vectors. h1,…,h8 represent the attention outputs of 8 different heads of the multi-head self-attention mechanism MHSA. W O Represents the output transformation matrix.

[0035] As a preferred technical solution of the present invention, the convolution step size and the pooling step size of the SA-CRNN module are adjusted by the following calculation formula when performing the convolution operation:

[0036]

[0037] Among them, s c represents the convolution step size, k c [-1] represents the element of the last dimension of the convolution kernel, k p represents the size of the pooling kernel, s p Represents the pooling stride.

[0038] Compared with the prior art, the present invention provides a dual-stream adaptive convolution cycle hybrid EEG emotion recognition method combined with an attention mechanism, which has the following beneficial effects:

[0039] The present invention assigns corresponding weights to EEG channels by combining the CSAM module in the dual-stream adaptive convolutional recurrent hybrid network model with an attention mechanism; then uses the MHSA-CRNN module and the SA-CRNN module to extract local spatiotemporal features; then splices the local features extracted by the MHSA-CRNN module and the SA-CRNN module and inputs them into the MHSA-TCN module to extract global information, and uses the extracted EEG information for emotion classification. The present invention performs binary and ternary classification experiments on the DEAP dataset. For the arousal dimension and the valence dimension, the binary classification accuracy rates are 99.26% and 99.15% respectively, and the ternary classification accuracy rates are 97.69% and 98.05% respectively, which are better than related algorithms. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 This is a schematic diagram of the model framework of the present invention;

[0041] Figure 2 It is a schematic diagram of the CSAM structure of the present invention;

[0042] Figure 3 It is a schematic diagram of the structure of MHSA of the present invention;

[0043] Figure 4 This is a schematic diagram of the structure of MHSA-TCN of the present invention;

[0044] Figure 5 This is a schematic diagram of the relationship between accuracy and number of iterations of the present invention;

[0045] Figure 6 This is a schematic diagram of the average accuracy of arousal and valence for each subject using CSA-SA-CRTNN on the DEAP dataset of the present invention;

[0046] Figure 7 It is a schematic diagram of the confusion matrix of the present invention;

[0047] Figure 8 Schematic diagram for comparing different channel attention mechanisms of the present invention;

[0048] Fig. 9 It is a schematic diagram of the process of the present invention. DETAILED DESCRIPTION

[0049] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0050] See also Figure 1 - Fig. 9 , a dual-stream adaptive convolutional cycle hybrid EEG emotion recognition method combined with attention mechanism, comprising the following steps:

[0051] S1. Collect multimodal emotion datasets and use them as input data, and preprocess the input data;

[0052] The multimodal emotion dataset is specifically the DEAP dataset (Database for Emotion Analysis using Physiological Signals), which is a multimodal emotion dataset designed by Queen Mary University of London and is one of the most widely used datasets in affective computing research. The dataset includes peripheral physiological signals and EEGs of 32 participants (16 males and 16 females, with an average age of 26.9 years old). In addition, frontal facial videos of 22 participants were recorded. In order to stimulate the emotions of the participants, the researchers played a one-minute music video and recorded the data in real time. Each participant watched 40 different videos, and the preprocessed EEG data for each trial contained 60 seconds of trial data and 3 seconds of baseline data. In the experiment, each participant used the Self-Assessment Mannequin (SAM) to rate the videos according to their emotional arousal, valence, degree of liking or disliking, dominance, and familiarity. The ratings were based on a two-dimensional arousal valence model, with a rating range of 1 to 9. The DEAP dataset contains 32 EEG signals and 8 peripheral physiological signals, with a sampling rate of 512Hz;

[0053] S1.1. Remove the baseline of the EEG signal in the multimodal emotion dataset to obtain the EEG signal after baseline removal. Specifically, the signal is downsampled to 128Hz; then the 63-second data is segmented into 1-second windows, the first three 1-second baseline segments are arithmetic averaged, and then the corresponding average baselines are subtracted from the last 60 segments to achieve the purpose of baseline removal. The calculation process is as follows;

[0054]

[0055] s=s original -S baseline

[0056] Among them, s baseline is the average baseline signal, s original is the original EEG signal, s is the signal after removing the baseline, and x i represents the baseline signal at the i-th second.

[0057] S1.2, process the emotion labels in the multimodal emotion dataset;

[0058] Two schemes are adopted based on the valence dimension and the arousal dimension on the DEAP dataset. The first scheme uses 5 as the threshold, and the arousal and valence SAM scores of 1 to 5 are low scores, and 6 to 9 are high scores. In the arousal dimension, it is divided into low arousal (LA) and high arousal (HA); in the valence dimension, it is divided into low valence (LV) and high valence (HV). The second scheme uses 4 and 6 as thresholds, and the arousal and valence SAM scores of 1 to 4 are low scores, 5 to 6 are medium scores, and 7 to 9 are high scores. In the arousal dimension, it is divided into low arousal (LA), medium arousal (MA) and high arousal (HA); in the valence dimension, it is divided into low valence (LV), medium valence (MV) and high valence (HV), see Table 1 below for details:

[0059] Table 1

[0060]

[0061] S2. Construct a two-stream adaptive convolutional recurrent hybrid network model combined with an attention mechanism. The two-stream adaptive convolutional recurrent hybrid network model combined with an attention mechanism includes a channel attention (CSAM) module, an adaptive convolutional recurrent (SA-CRNN) module, a multi-head self-attention convolutional recurrent (MHSA-CRNN) module, and a multi-head self-attention temporal convolution (MHSA-TCN) module. The CSAM module receives the preprocessed EEG signal and allocates attention to the channel dimension to enhance the effect of the attention mechanism on the performance of the model. The proposed CSAM module is as follows: Figure 2 As shown, it includes a global pooling layer, a linear layer, a convolutional layer and an activation layer. The specific parameter settings of the module are shown in Table 2;

[0062] Table 2

[0063]

[0064] The CSAM module first performs average pooling on the input, then uses a linear layer to map the channel information of the obtained vector to a higher dimensional space, and further extracts features through a convolutional layer; after the convolution operation, a nonlinear transformation is performed using the tanh activation function, and then a linear transformation is performed again to restore the convolution and activated feature map to its original shape; finally, the Softmax function is applied to the linearly transformed feature map in the channel dimension to generate an attention map, convert the feature value of each channel into a probability distribution, that is, the weight of each channel, and perform element-by-element multiplication of the attention map with the original input to generate a weighted output. In this way, each channel is weighted according to its importance, thereby achieving weight distribution.

[0065] The SA-CRNN module is used to extract EEG emotion information; the module is composed of a three-layer CNN and a two-layer LSTM in series. The module first extracts information of the spatial dimension through a convolutional neural network CNN, and then uses LSTM to extract the temporal dependency of the signal; the CNN includes a convolutional layer, an ELU activation layer, and a maximum pooling layer, and the CNN can adaptively match the convolution kernel with the appropriate convolution step size and pooling operation parameters according to the provided convolution kernel size; the input dimension of the LSTM unit is 720, and the hidden layer dimension is 64.

[0066] When the SA-CRNN module performs convolution operations, the convolution step size and pooling step size are adjusted by the following calculation formula:

[0067]

[0068] Among them, s c represents the convolution step size, k c [-1] represents the element of the last dimension of the convolution kernel, k p represents the size of the pooling kernel, s p represents the pooling step size;

[0069] The MHSA-CRNN module consists of a convolutional recurrent neural network (CRNN) and a multi-head self-attention mechanism (MHSA). The convolutional recurrent neural network (CRNN) is composed of a three-layer CNN and a two-layer LSTM in series; and the multi-head self-attention mechanism (MHSA) is added between CNN and LSTM. The module first extracts information about the spatial dimension through the convolutional neural network (CNN), and then uses LSTM to extract the temporal dependency of the signal.

[0070] CNN includes convolutional layers, activation (ELU) layers, and maximum pooling layers. After the convolution operation, the exponential linear unit (ELU) function is used as the activation function, which is better than the commonly used rectified linear unit (ReLU) function. The convolution kernel size is (32, 40) and the stride is 1; the pooling kernel size is (1, 75) and the stride is 10. LSTM is composed of two stacked layers. Each layer of LSTM takes the hidden state of the previous layer as input to learn more advanced feature representations. Increasing the number of layers can increase the expressive power of the model, but it will also increase the training time and computational cost of the model. The input dimension of the LSTM unit is 80, and the hidden layer dimension is 64.

[0071] The self-attention mechanism is the core component of MHSA. First, the input features are mapped to query (Q), key (K), and value (V) features through linear transformation, and divided into three parts: Q, K, and V. Then, the attention score is calculated by calculating the inner product of the query and the key and multiplying it by a scaling factor (Scale). Then, the attention score is masked according to the input mask (if any) to ignore invalid positions, and the masked attention score is Softmax normalized. Finally, the value is weighted and summed to get the output, and the output is mapped back to the original feature dimension using a linear layer and Dropout is performed.

[0072] The specific expression of the convolutional recurrent neural network CRNN in the MHSA-CRNN module is as follows:

[0073] v st =LSTM(Conv(x))

[0074] Where x is the input signal, v st is the feature vector output by the CRNN network, LSTM(*) indicates the use of LSTM to extract signals, and Conv(x) indicates the convolution operation on the input signal x;

[0075] The specific calculation expression of the multi-head self-attention mechanism MHSA is as follows:

[0076]

[0077] MHSA(Q,K,V)=Concat(h1,...,h8)W O

[0078] Among them, Q, K, V represent the features of query, key and value respectively, SA(Q, K, V) represents the output of self-attention, It means that the product of the Q, K, V values ​​of the nth self-attention head and its corresponding transformation matrix is ​​used to perform the operation of the first formula, h n represents the nth head attention output, MHSA(Q,K,V) represents the output of the multi-head self-attention mechanism MHSA, are the transformation matrices of the query, key, and value of the nth head, respectively. Concat(*) represents the concatenation of different feature vectors. h1,…,h8 represents the attention output of the multi-head self-attention mechanism MHSA from 8 different heads. W O Represents the output transformation matrix;

[0079] The MHSA-TCN module consists of a multi-head self-attention mechanism MHSA and a temporal convolutional network TCN; the multi-head self-attention mechanism MHSA is added between the first hidden layer and the second hidden layer of TCN, enabling the model to focus on different aspects of the input sequence.

[0080] TCN can achieve efficient parallel computing and is suitable for large-scale data processing. In addition, it has many advantages, such as: extracting features of different scales by stacking multiple convolutional layers; effectively capturing local dependencies in sequence data by stacking convolutional layers and increasing the receptive field of the convolution kernel; and avoiding the gradient vanishing and gradient exploding problems to a large extent.

[0081] The main reason why TCN can be widely used is that it can increase the receptive field. The calculation method of the receptive field of the first layer is as follows:

[0082]

[0083] F l is the receptive field of the lth layer, k is the convolution kernel size, L is the number of convolution layers in each residual block, and d l is the expansion factor of the lth layer;

[0084] The MHSA-TCN module accepts the local features extracted by MHSA-CRNN and SA-CRNN as input to extract global EEG features.

[0085] This module involves knowledge of dilated causal convolution, residual connection, etc., where convolution refers to convolution in CNN, which refers to a sliding operation performed by the convolution kernel on the data; dilation refers to allowing interval sampling of the input during convolution; causality refers to the data at time t in the current layer, which only depends on the influence of the values ​​at time t and before in the previous layer. The model first performs interval sampling on the input, where the scaling factor indicates how many time steps are sampled at each layer; then the sampled data is weighted normalized and Dropout is performed; finally, the processed signal is added to the original signal as the output. In this experiment, TCN contains four hidden layers, each hidden layer contains two overlapping operations mentioned above, the number of channels is 25, kernel size = 2, and scaling factor d = [1,2,4,8].

[0086] The specific workflow of the two-stream adaptive convolutional recurrent hybrid network model combined with the attention mechanism is as follows:

[0087] S2.1. For the input data, the CSAM module assigns corresponding weights to the EEG channels, and sends the data to the MHSA-CRNN module and the SA-CRNN module respectively;

[0088] S2.2, extract local spatiotemporal features using MHSA-CRNN module and SA-CRNN module;

[0089] S2.3, then the local features extracted by the MHSA-CRNN module and the SA-CRNN module are concatenated and input into the MHSA-TCN module to extract the global information;

[0090] S2.4. Use the information extracted in step S2.3 for sentiment classification

[0091] S3, dividing the input data in step S1 into a training set and a test set in a ratio of 8:2, and training the two-stream adaptive convolutional recurrent hybrid network model combined with the attention mechanism in step S2;

[0092] S4, recognizing EEG emotions through the dual-stream adaptive convolutional recurrent hybrid network model combined with the attention mechanism trained in step S3;

[0093] The following is the experiment and results section of the present invention:

[0094] In this experiment, eight peripheral channels are removed and only EEG signals are used for emotion recognition. The experimental data of the DEAP dataset consists of 32 files. Corresponding to the 32 experimental subjects, each participant file contains two arrays. The data format after simple preprocessing is shown in Table 3.

[0095] Table 3 DEAP dataset

[0096]

[0097] The present invention uses the DEAP public dataset and uses the Pytorch framework on the GeForce RTX 2080Ti GPU to train and verify the model. The experiment selects Adam as the optimizer, the learning rate is set to 0.001, the batch size is set to 128, the loss function uses the cross entropy loss function, and after experimental verification, the epoch is set to 30. In this study, the data is divided into a training set and a test set in a ratio of 8:2, and a 10-fold cross-validation is used to evaluate the dual-stream adaptive convolutional recurrent hybrid network model (CSA-SA-CRTNN model) combined with the attention mechanism of the present invention;

[0098] The accuracy of the two-class and three-class classification on the DEAP dataset changes with the training process as shown below: Figure 5As shown in the figure, the accuracy gradually increases and finally stabilizes, indicating that the training process of the model is stable and can converge within a certain number of iterations. It can also be seen from the figure that the best number of iterations for the proposed model is 30. When it is iterated to 30 times, whether it is binary classification or ternary classification, the model achieves the best accuracy with the least resource consumption, so this experiment sets the epoch to 30.

[0099] from Figure 5 It can be seen that the CSA-SA-CRTNN model is stable and effective on the DEAP dataset. Figure 6 The accuracy of the model for each subject on the DEAP dataset is shown when the epoch is set to the optimal 30.

[0100] In the two-classification experiment, except for s05 and s22, the average accuracy of the valence dimension reached 99.15%, and for the arousal dimension, except for s02 and s22, the average accuracy was 99.26%. In the three-classification experiment, except for s05 and s22, the average accuracy of the valence dimension reached 98.05%, and in the arousal dimension, except for s02 and s22, the average accuracy was 97.69%. It is worth noting that subject No. 22 showed the same characteristics in the two-classification and three-classification experiments, and its recognition accuracy was significantly lower than that of other subjects. The two-classification valence and arousal were 90.27% and 93.75%, respectively, and the three-classification valence and arousal were 87.40% and 80.75%, respectively. This may be due to the subject's lack of concentration in the experiment or subjective factors during scoring.

[0101] The confusion matrix is Figure 7 The confusion matrix shows the excellent classification ability of CSA-SA-CRTNN for each emotion category.

[0102] To further verify the effectiveness of the model, CSA-SA-CRTNN was compared with some related EEG emotion recognition studies on the DEAP dataset. The comparison method is as follows:

[0103] 1. Tsception (Yi et al, 2021): Tsception consists of a dynamic temporal layer, an asymmetric spatial layer, and an advanced fusion layer, which can simultaneously learn discriminative representations in the temporal and channel dimensions. The dynamic temporal layer consists of multi-scale one-dimensional convolutional kernels to learn the dynamic temporal representation and frequency representation of EEG signals. The length of the multi-scale one-dimensional convolutional kernel is related to the sampling rate of the EEG signal. The asymmetric spatial layer exploits the asymmetric neural activations behind emotional responses and learns discriminative global and hemispheric representations. The learned multiple features will be fused by an advanced fusion layer.

[0104] 2.DCRNN (Li et al, 2022): DCRNN uses a deep sparse autoencoder network (DSAE) to remove redundant information from EEG signals and reconstruct the underlying features of EEG signals. It combines convolutional neural networks (CNN) with long short-term memory (LSTM) to mine the correlation between EEG signal channels and integrate contextual information from EEG signal frames.

[0105] 3.4D-CRNN (Shen et al, 2020): 4D-CRNN converts the differential entropy features of different channels into a four-dimensional structure to train deep models. Convolutional neural network (CNN) and long short-term memory (LSTM) network are introduced, CNN is used to learn frequency and spatial information from each time slice of 4D input, and LSTM is used to extract time dependency from CNN output.

[0106] 4. BiSMSM (Li et al, 2023): BiSMSM includes two streams: spatial stream and temporal stream. It captures useful information from temporal, spatial, local and global perspectives. The structures of these two streams are similar, both of which contain a multi-layer perceptron (MLP)-based module to extract regional intra-channel and cross-channel information. The MLP-based module is followed by a self-attention mechanism module to explore global signal correlation.

[0107] 5.AP-CapsNet (Liu et al, 2023): AP-CapsNet uses the coordinate attention mechanism to give relative spatial information weights to the input signal, and then maps the EEG signal to a high-dimensional space. The pre-trained model is used as a feature extractor to construct a two-layer capsule network for emotion recognition.

[0108] 6. ATDD-LSTM (Du et al, 2021): ATDD-LSTM applies an attention mechanism to the feature vectors generated by LSTM to automatically select appropriate EEG channels for emotion recognition; it uses a domain discriminator to modify the representation space of the data and generate domain-invariant features.

[0109] 7. ICaps-ResLSTM (Fan et al, 2024): ICaps-ResLSTM uses an improved capsule network as a spatial module to learn specific EEG spatial representations. ResLSTM, as a temporal module, inherits the information flow of the upper spatial module and complementarily learns the spatiotemporal dual module features through residual connections, thereby obtaining more discriminative EEG features.

[0110] 8. CADD-DCCNN (Li et al, 2024): CADD-DCCNN uses short-time Fourier transform (STFT) to extract differential entropy features from raw EEG signals; then each channel of the DE feature is regarded as a view, and the attention mechanism is used on different views to aggregate and discriminate emotional information at the EEG time frame level. Then, an extended causal convolutional neural network is used to extract nonlinear relationships between different time frames. The feature-level fusion method is used to fuse multi-channel features to mine potential complementary information between different perspectives; a domain discriminator is used to generate domain-invariant features, and data from different domains are projected into the same data representation space to minimize individual differences.

[0111] The CSA-SA-CRTNN model proposed in this paper has an accuracy of 99.26% and 99.15% for arousal and valence respectively on the DEAP dataset, which is more competitive than existing methods. Table 4 shows the comparison of the results of these methods.

[0112] Table 4 Performance comparison of different methods on the DEAP dataset

[0113]

[0114] The model proposed in the present invention consists of four modules, namely, CSAM module, SA-CRNN module, MHSA-CRNN module and MHSA-TCN module. In order to evaluate the contribution of each module in the model, the present invention conducted 4 types of ablation experiments on the DEAP dataset.

[0115] Specifically, four ablation models are constructed: SA-CRTNN, CSA-CRTNN, CA-SA-CRTNN and CSA-SA-CRNN. ​​Among them, SA-CRTNN is a model without CSAM module; CSA-CRTNN is a model without SA-CRNN module; CA-SA-CRTNN is a model without MHSA-CRNN module; CSA-SA-CRNN is a model without MHSA-TCN module. The detailed composition of each model is shown in Table 5.

[0116] Table 5 Ablation model

[0117]

[0118] Each model is tested on the DEAP dataset for binary and ternary classification, and the results are shown in Table 6.

[0119] Table 6 Ablation results of CSA-SA-CRTNN on the DEAP dataset

[0120]

[0121] At present, the model proposed by the present invention has the best accuracy, which is 99.26% and 99.15% for the second classification, and 97.69% and 98.05% for the third classification. Compared with the proposed model, the accuracy of SA-CRTNN decreases the most, which is 13.01% and 14.15% for the second classification, and 13.95% and 19.30% for the third classification, respectively. This shows that the proposed channel attention module CSAM plays a key role in the model. Without CSAM, the model may focus on some useless information, resulting in a decrease in accuracy. The accuracy gap between CSA-CRTNN and the proposed model is second, with a decrease of 6.76% and 4.16% for the second classification, and a decrease of 3.94% and 11.80% for the third classification, respectively. This shows that the proposed SA-CRNN module can well solve the problem of information omission in the feature extraction process. The accuracy gap between CSA-SA-CRNN and the proposed model ranks third, with a decrease of 4.47% and 6.24% in the second classification and a larger decrease of 10.61% and 9.51% in the third classification, indicating the necessity of global feature extraction. CA-SA-CRTNN has the least decrease in accuracy, with a decrease of 2.11% and 2.90% in the second classification and 1.15% and 3.06% in the third classification, indicating that MHSA-CRNN has a certain contribution in the model.

[0122] Comparison of CSAM with other channel attention mechanisms:

[0123] In order to verify the advantages of the channel attention mechanism CSAM proposed in this paper, this paper compares it with the traditional channel attention mechanisms SENet and ECANet, and the results are as follows: Figure 8 As shown:

[0124] SENet obtains the importance of each channel of the feature map through automatic learning, and then uses this importance to assign a weight value to each feature, so that the neural network can focus on certain feature channels. Its key operations are squeeze and excitation.

[0125] ECANet uses a 1x1 convolution layer directly after the global average pooling layer, removes the fully connected layer, and avoids dimensionality reduction. One-dimensional convolution is used to complete cross-channel information interaction, and the size of the convolution kernel changes adaptively through a function, so that layers with a larger number of channels can interact more across channels.

[0126] The experimental results show that the channel attention mechanism CSAM proposed in this paper achieves better performance than SENet and ECANet, which indicates that in the model of the present invention, compared with the commonly used channel attention, CSAM can better select the key channels of EEG signals and improve the model recognition accuracy;

[0127] Evaluation of model efficiency:

[0128] The original intention of this paper is to improve the model accuracy while reducing the computational cost. In order to prove that CSA-SA-CRTNN can reduce the model computational cost, the present invention lists the parameters of the proposed model, as shown in Table 7.

[0129] Table 7 Parameters of CSA-SA-CRTNN model

[0130]

[0131] In order to prove the efficiency advantage of the proposed model, the parameters and accuracy of this model are compared with those of the other four models. The results are shown in Table 8

[0132] Table 8 Comparison of efficiency of different models

[0133]

[0134] The results show that the model of the present invention has higher efficiency, the model parameter is 2.90M, and the model can achieve higher accuracy with lower resource consumption.

[0135] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A dual-stream adaptive convolutional cyclic hybrid EEG emotion recognition method combined with an attention mechanism, characterized by: The following steps are involved: S1. Collect multimodal emotion datasets and use them as input data, and preprocess the input data; S2. Construct a two-stream adaptive convolutional cycle hybrid network model combined with an attention mechanism, wherein the two-stream adaptive convolutional cycle hybrid network model combined with an attention mechanism includes a channel attention CSAM module, an adaptive convolutional cycle SA-CRNN module, a multi-head self-attention convolutional cycle MHSA-CRNN module, and a multi-head self-attention temporal convolution MHSA-TCN module; The channel attention CSAM module is used to allocate weights in the channel dimension and includes a global pooling layer, a linear layer, a convolutional layer and an activation layer; The adaptive convolution cycle SA-CRNN module is used to extract EEG emotion information. It is composed of a three-layer CNN and a two-layer LSTM in series. The module first extracts information of the spatial dimension through the convolutional neural network CNN, and then uses LSTM to extract the time dependency of the signal; The multi-head self-attention convolutional cycle MHSA-CRNN module is composed of a convolutional recurrent neural network CRNN and a multi-head self-attention mechanism MHSA. The convolutional recurrent neural network CRNN in the multi-head self-attention convolutional cycle MHSA-CRNN module first extracts information of the spatial dimension through the convolutional recurrent neural network CRNN, and then uses LSTM to extract the temporal dependency of the signal; The self-attention temporal convolution MHSA-TCN module is composed of a multi-head self-attention mechanism MHSA fused into a temporal convolutional network TCN; The specific workflow of the two-stream adaptive convolutional recurrent hybrid network model combined with the attention mechanism is as follows: S2.

1. For the input data, the channel attention CSAM module assigns corresponding weights to the EEG channels, and sends the data to the multi-head self-attention convolutional cycle MHSA-CRNN module and SA-CRNN module respectively; S2.2, use the multi-head self-attention convolutional recurrent MHSA-CRNN module and the adaptive convolutional recurrent SA-CRNN module to extract local spatiotemporal features; S2.3, then the local features extracted by the multi-head self-attention convolution cycle MHSA-CRNN module and the adaptive convolution cycle SA-CRNN module are spliced ​​and input into the self-attention temporal convolution MHSA-TCN module to extract global information; S2.4, using the information extracted in step S2.3 for sentiment classification; S3, dividing the input data in step S1 into a training set and a test set in a ratio of 8:2, and training the two-stream adaptive convolutional recurrent hybrid network model combined with the attention mechanism in step S2; S4. Identify EEG emotions through the dual-stream adaptive convolutional recurrent hybrid network model combined with the attention mechanism trained in step S3.

2. According to claim 1, a dual-stream adaptive convolutional cycle hybrid EEG emotion recognition method combined with an attention mechanism is characterized by: The specific steps of data preprocessing in step S1 include: S1.1, removing the baseline of the EEG signal in the multimodal emotion dataset to obtain the EEG signal after removing the baseline; S1.

2. Process the emotion labels in the multimodal emotion dataset.

3. According to claim 2, a dual-stream adaptive convolutional cycle hybrid EEG emotion recognition method combined with an attention mechanism is characterized by: The calculation expression for removing the baseline in step S1.1 is as follows: s=s original -S baseline Among them, s baseline is the average baseline signal, s original is the original EEG signal, s is the signal after removing the baseline, and x i represents the baseline signal at the i-th second.

4. According to claim 1, a dual-stream adaptive convolutional cycle hybrid EEG emotion recognition method combined with an attention mechanism is characterized by: The specific expression of the convolutional recurrent neural network CRNN in the multi-head self-attention convolutional cycle MHSA-CRNN module is as follows: v st =LSTM(Conv(x)) Where x is the input signal, v st is the feature vector output by the CRNN network, LSTM(*) indicates the use of LSTM to extract signals, and Conv(x) indicates the convolution operation on the input signal x; The specific calculation expression of the multi-head self-attention mechanism MHSA is as follows: MHSA(Q,K,V)=Concat(h1,……,h8)W O Among them, Q, K, V represent the features of query, key and value respectively, SA(Q, K, V) represents the output of self-attention, represents the operation of multiplying the Q, K, V values ​​of the nth self-attention head by the corresponding transformation matrix, h n represents the nth head attention output, MHSA(Q,K,V) represents the output of the multi-head self-attention mechanism MHSA, are the transformation matrices of the query, key, and value of the nth head, respectively. Concat(*) represents the concatenation of different feature vectors. h1,…,h8 represent the attention outputs of 8 different heads of the multi-head self-attention mechanism MHSA. W O Represents the output transformation matrix.

5. According to claim 4, a dual-stream adaptive convolutional cycle hybrid EEG emotion recognition method combined with an attention mechanism is characterized in that: The convolution step size and pooling step size of the adaptive convolution cycle SA-CRNN module are adjusted by the following calculation formula when performing convolution operation: Among them, s c represents the convolution step size, k c [-1] represents the element of the last dimension of the convolution kernel, k p represents the size of the pooling kernel, s p Represents the pooling stride.

Images