Emotion recognition method based on multi-domain attention fusion network

Abstract

The invention discloses an emotion recognition method based on a multi-domain attention fusion network, and the method comprises the steps: firstly obtaining an EEG signal, carrying out the preprocessing of the EEG signal, carrying out the channel feature extraction through a space-frequency domain mixed channel attention module, and capturing a weighted attention feature; secondly, weighting attention features, and extracting space and frequency domain features of EEG signals through a CNN module; and combining the improved multi-dimensional self-attention with the BiSRU, and capturing comprehensive features of space and frequency domain features. And finally, classifying the comprehensive features through a Softmax function, and outputting an emotion recognition result. According to the method, the capability of capturing the long-term dependency relationship of the electroencephalogram signals is enhanced, and the emotion recognition performance and efficiency are effectively improved.

Description

Technical Field

[0001] The present invention relates to an electroencephalogram (EEG) signal processing and emotion recognition method, and in particular to an EEG signal emotion recognition method based on a multi-domain attention fusion network. Background Art

[0002] Emotions play a vital role in our daily lives. The presence of negative emotions increases the risk of mental health problems such as depression and schizophrenia. Therefore, the recognition and identification of emotions are crucial for emotion regulation. Electroencephalogram (EEG) signals, as a non-invasive method for monitoring brain activity, have been widely used in fields such as emotion recognition, medical diagnosis, and human-computer interaction.

[0003] EEG signals have multi-dimensional features such as time, space and spectrum, which must be fully considered in emotion analysis. However, current methods fail to fully utilize these different EEG features and the local patterns between emotion features. In EEG-based emotion recognition, the introduction of deep learning models has significantly improved classification performance and generalization ability.

[0004] Some researchers have also extracted spectra from different frequency bands and input them into convolutional neural networks for emotion analysis. However, convolutional neural networks (CNNs) ignore global information, while recurrent neural networks (RNNs) fail to capture spatial information and have low parallel efficiency. To address these problems, researchers began to combine convolutional neural networks with recurrent neural networks for emotion recognition. At the same time, the attention mechanism has also attracted the attention of researchers. Although the self-attention mechanism in the transformer can mine the relationship between temporal information in the EEG, it does not take into account the crucial spatial information in the EEG.

[0005] Moreover, it should be noted that most of the existing models only utilize a single feature or a combination of two features, and fail to fully utilize the complementarity between space-frequency-time features. Therefore, these models are limited to a certain extent. In addition, how to capture the local patterns of features of different dimensions also brings challenges to EEG emotion recognition. The commonly used long short-term memory network (LSTM) and gated recurrent unit (GRU) have relatively more complex structures, resulting in higher computational costs for training and inference. They usually have more parameters, which increases the complexity of the model. The pooling methods currently used for EEG emotion recognition tasks are mainly maximum pooling and average pooling. Maximum pooling tends to capture the most significant local features, while average pooling pays more attention to global information. However, in the emotion recognition task, it is necessary to capture local features and consider global patterns, which may cause the pooling operation to fail to fully meet the task requirements. In addition, the processing method of EEG signals can also be further optimized to improve the accuracy and efficiency of emotion recognition. Summary of the invention

[0006] The present invention proposes an emotion recognition method of a multi-domain attention fusion network, aiming to solve the problems of insufficient feature extraction, limited classification accuracy, information loss in the pooling process, etc. in the current EEG emotion recognition method. Different from the traditional method, the present invention comprehensively considers the frequency domain, spatial domain and time domain features. The differential entropy features of each channel of the EEG signal are extracted and combined with the channel information, and the importance of each channel is dynamically weighted through the space-frequency domain hybrid channel attention mechanism, thereby more accurately extracting the spatial information of the EEG signal. Compared with the traditional pooling method, the present invention adopts a bilinear downsampling pooling (BD pooling) operation, which can retain more detailed information while highlighting important features, thereby improving the performance of the model. In addition, the present invention introduces a bidirectional simple recurrent unit (BiSRU) structure instead of the traditional RNN structure, uses its bidirectional characteristics to capture the long-term dependencies in the EEG signal, and considers the intrinsic correlation between samples by improving the multidimensional self-attention mechanism, thereby further improving the performance of the model, making it have higher accuracy and efficiency in the EEG emotion recognition task.

[0007] According to the invention, the present invention specifically comprises the following steps:

[0008] Step 1: Obtain EEG signals and preprocess the EEG signals.

[0009] Step 2: The preprocessed EEG signal is passed through the spatial-frequency domain hybrid channel attention module to extract channel features and capture weighted attention features. Specifically:

[0010] The preprocessed EEG signal is subjected to channel feature extraction, and differential entropy features are extracted for each channel. The channel features and differential entropy features are used to calculate the weight of attention of each channel respectively, and then the two weights are added and weighted to the original EEG signal after normalization, so as to realize the dynamic allocation of the importance of each channel in emotion recognition. The spatial and frequency domain hybrid channel attention module can capture the spatial and frequency domain features of the EEG signal at the same time. Existing methods usually only focus on the features of a single dimension for EEG signal channels, without fully considering the comprehensive feature information of the spatial and frequency domains. The method of the present invention can more comprehensively reflect the complexity of the EEG signal by fusing multi-dimensional features, thereby improving the accuracy of emotion recognition. This fusion method can better retain and utilize important information in the signal and improve the performance of the model.

[0011] Step 3: The weighted attention features are passed through the CNN module to extract the spatial and frequency domain features of the EEG signal.

[0012] The CNN module further extracts the spatial and frequency domain information of the EEG signal. The BD pooling method proposed in the present invention calculates the corresponding value by considering the data in the entire pooling window, which can highlight the most important features while retaining some detailed features, thereby improving the classification performance and efficiency. Existing commonly used pooling methods such as maximum pooling may lose some detailed information, while BD pooling retains important details in a selective way, thereby improving the representativeness of features and classification accuracy.

[0013] Step 4: Combine the improved multi-dimensional self-attention with BiSRU to capture the comprehensive features of spatial and frequency domain features.

[0014] In order to fully consider the temporal information of the EEG signal, the improved multi-dimensional self-attention is combined with BiSRU to capture the long-term dependencies of the features in the output of step 3. Although general RNN models such as LSTM and GRU can process time-dependent information, they have limitations in capturing complex multi-dimensional features. The introduction of the self-attention mechanism allows the model to pay attention to the importance of each position in the input sequence, thereby better capturing timing information. In addition, BiSRU further improves the model's ability to capture time-dependent features through a bidirectional information flow. In theory, the combination of the self-attention mechanism and BiSRU can significantly improve the model's ability to understand temporal features without significantly increasing the computational cost.

[0015] Step 5: Classify the comprehensive features through the Softmax function and output the emotion recognition results.

[0016] The beneficial effects of the present invention are as follows:

[0017] By introducing the multi-domain attention mechanism and the CNN-BSA architecture, the frequency domain, spatial and temporal features of the EEG signal are effectively integrated. Through the novel spatial-frequency domain hybrid channel attention mechanism, the present invention realizes the dynamic allocation of weights to each channel to more accurately extract the key information of the EEG signal. The introduction of the BiSRU module combined with the improved multi-dimensional self-attention not only enhances the ability to capture the long-term dependencies of EEG signals, but also effectively improves the performance and efficiency of emotion recognition. In addition, the BD pooling method proposed in the present invention effectively reduces the parameter calculation amount of the model, can better highlight important features and retain key detail information, thereby providing a more reliable feature representation for emotion recognition and further improving the classification accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is the principle diagram of the present invention;

[0019] Figure 2 Schematic diagram of spatial-frequency domain mixed channel attention;

[0020] Figure 3 This is the convolutional layer structure and BD pooling principle diagram;

[0021] Figure 4 It is the schematic diagram of the SRU unit;

[0022] Figure 5 Schematic diagram of the improved multi-dimensional self-attention mechanism;

[0023] Figure 6 It is the classification result diagram of each subject in the dataset;

[0024] Figure 7 This is the result diagram of the channel weights of the spatial-frequency domain mixed channel attention on the dataset;

[0025] Figure 8 This is a comparison chart of the BD pooling method, maximum pooling, and average pooling. DETAILED DESCRIPTION

[0026] The present invention is further described below in conjunction with specific embodiments. The following description is only for demonstration and explanation and does not limit the present invention in any form.

[0027] like Figure 1 As shown, the implementation steps of the embodiment of the present invention are as follows:

[0028] Step 1: Obtain EEG signals and preprocess the signals, including downsampling to 128 Hz and removing EOG artifacts, and subtracting 1 second baseline signals from the experimental brain point signals. In addition, the EEG signals were divided into 3 second intervals for subsequent analysis.

[0029] Step 2: Apply the space-frequency domain mixed channel attention module to the preprocessed EEG signal to perform feature weighting, and perform space-frequency domain mixed channel attention weighting on the preprocessed EEG signal, such as Figure 2 As shown, the importance of each channel in emotion recognition can be dynamically adjusted, and both spatial domain and frequency domain information are considered at the same time, so as to more accurately determine the weight of the key channel. The specific steps are as follows:

[0030] Step 2-1: Perform Fourier transform on the EEG signal and use the differential entropy formula to extract the frequency domain feature H(x) of a single channel:

[0031] H(x)=-∫p(x)log[p(x)]dx (1)

[0032] Where x represents the EEG signal, and p(x) represents the probability density function of x. The frequency domain features of the c channels of each sample are concatenated to obtain the frequency domain features of each sample.

[0033] Step 2-2: Perform mean pooling on each channel of the preprocessed EEG sample to obtain the following channel statistics:

[0034]

[0035] in is the average value of the jth channel, Obtaining spatial domain features through dimensionality conversion

[0036] Step 2-3: Use the extracted frequency domain features and spatial domain features to weight them through the weight matrix. The mechanism of spatial-frequency domain mixed channel attention is expressed as:

[0037] p = sigmoid(W s2 ·(tanh(W s1 ·F s +b s1 )+b s2 )+W f2 ·(tanh(W f1 ·F f +b f1 )+b f2 )+b) (3)

[0038] Where W s1 and W f1 Represents the dimension reduction (dimension reduction ratio is R) weighted matrix of spatial domain and frequency domain features, W s2 and W f2 Represents the dimension-increasing weighted matrix of spatial domain and frequency domain features, b s1 and b f1 is the bias term of the dimension reduction layer, b s2 and b f2 is the bias term of the dimension increase layer, and b is the bias term of the mixed feature weight. The tanh function is used for nonlinear activation of the dimension reduction layer. The Sigmoid function converts the importance of the channel into different probability distributions p = [p1, p2, ..., p c ], which represents the importance of different channels.

[0039] Step 2-4: Finally, the EEG sample S i =[s1,s2,...,s c ], using probability as weight for re-encoding. Therefore, the attention channel feature of the jth (j=1,2,...,c) channel extracted by spatial-frequency domain mixed channel attention can be expressed as:

[0040] c j =p j ·s j (4)

[0041] Therefore, C={C1,C2,...,C n} represents the extracted channel attention feature, the i-th extracted feature C i =[c1,c2,...,c c ] can be obtained by S i =[s1,s2,...,s c ] corresponds to the channel and p=[p1,p2,...,p c ] is obtained by multiplying the corresponding elements in .

[0042] Step 3: Use convolutional neural network to extract features from the attention-weighted signal, such as Figure 3 As shown, bilinear downsampling pooling (BD pooling) is used to further extract features. The specific steps are as follows:

[0043] Step 3-1: The number of convolution kernels is K, and its height is the same as the number of channels. It is mainly used to capture spatial features, while the width of the kernel is designed to explore temporal information. The present invention selects the exponential linear unit (ELU) as the activation function of the convolution operation. Therefore, the i-th convolution feature C′ i (i=1,2,...,n) is the i-th weighted attention feature C i Obtained through convolution and activation operations.

[0044] Step 3-2: Use the BD Pooling method to pool the convolutional features. BD Pooling first performs a bilinear interpolation-like calculation operation on the input within each step size and window size, calculates n corresponding interpolations, and then represents the features in the pooling window by calculating the interpolation average. Each feature encoding Q i It is expressed as the result of interpolation and averaging within the pooling window, and the formula is:

[0045]

[0046] Among them, P i,j is the value obtained after bilinear interpolation within the pooling window i. Specifically, the pooling window size is [h,w] and the step size is s. Bilinear interpolation calculation is performed inside each pooling window to obtain the summarized feature representation.

[0047] Step 3-3: Finally, the feature dimension is transformed into a feature that conforms to the input shape of the subsequent bidirectional simple recurrent unit, and a dropout operation is applied to prevent overfitting and improve the generalization ability of the model to obtain spatial and frequency domain features.

[0048] Step 4: Use the bidirectional simple recurrent unit (BiSRU) combined with the improved multi-dimensional self-attention mechanism to extract temporal features from the extracted features. The specific steps are as follows:

[0049] Step 4-1: Simple loop unit, such as Figure 4 As shown, it receives two inputs, namely the spatial and frequency domain features x of the current time step t , the input c of the previous time step t-1 The simple recurrent unit outputs two results: the output c of the current time step t and the hidden state h t , where state c t It can be regarded as a feature extracted from the current time step. The SRU unit updates the state and calculates the output in the following way:

[0050]

[0051] h t =r t ⊙tanh(c t )+(1-r t )⊙x t (7)

[0052] where f t is the forget gate, which is used to control the degree of mixing between the current state and the previous state. is the candidate state for the current time step. t is the reset gate, which is used to control the influence of the previous state in the current state update.

[0053] The bidirectional simple recurrent unit is an extension of the simple recurrent unit. By processing the forward and reverse information of the sequence simultaneously, the final i-th output of the bidirectional simple recurrent unit network is {h′ i |h′ i =BiSRU(Q i ),i=1,2,...,s};

[0054] Step 4-2: Through the improved multi-dimensional self-attention mechanism, such as Figure 5 As shown in the figure, the comprehensive features of spatial and frequency domain features are extracted from the output features of the bidirectional simple cycle unit. The details are as follows:

[0055] Step 4-2-1: The improved multi-dimensional self-attention mechanism calculates the similarity within each sample from different points. Feature score vector z i It can be regarded as the i-th feature h′ i The improved multi-dimensional self-attention mechanism adds two bias terms inside and outside the activation function, where z i It is expressed as:

[0056] z i =f(h′ i )=W T σ(W1h i ′+b1)+b (8)

[0057] Among them, z i Represents the intrinsic similarity of the i-th encoded EEG sample. The activation function σ adopts ELU, W and b are the weight term and bias term of the function respectively, W1 is the weight parameter, and b1 is the bias term;

[0058] Step 4-3: Then, calculate the probability matrix P for each sample ki =[p1,p2,...p i ,...,p s ]:

[0059] The probability of the i-th EEG sample can be expressed as:

[0060]

[0061] Step 4-4: Finally, the comprehensive feature extracted by the improved multi-dimensional self-attention mechanism is F att ={F1,F2,...,F n}, the i-th attention feature extracted by the improved multi-dimensional self-attention mechanism can be expressed as:

[0062] F i =p i ·h′ i (10)

[0063] Step 5: Use SoftMax classifier to extract the feature F att ={F1,F2,...,F n} to classify:

[0064] P emo =SoftMax(W·F att +b) (11)

[0065] Where P i (i=1,2,....,n) represents the predicted probability of the i-th EEG sample, P emo = {P1,P2,...,P n} represents the set of these samples, W and b represent the weight and bias of SoftMax respectively.

[0066] Finally, the cross entropy loss is calculated for all samples:

[0067]

[0068] Among them, Y iis the true label of the i-th EEG sample, with a lower cross entropy loss Corresponding to higher recognition accuracy.

[0069] The present invention conducts experiments on the Database of Emotion Analysis of Physiological Signals (DEAP), which is a multimodal dataset designed to study human emotional states. It records the EEG and peripheral physiological signals of 32 participants while they watch 40 one-minute video clips. These signals are recorded through 32 electrodes according to the international 10-20 system. Each participant scores the emotional experience of arousal, valence, like / dislike, dominance, and familiarity, ranging from 1 to 9. Each video trial contains 60 seconds of experimental signals and 3 seconds of baseline signals. The dataset is preprocessed, including downsampling to 128Hz and removing electrooculographic artifacts. In this embodiment, only EEG signals are used, and the threshold is set to 5. The experimental data are divided into high and low categories according to the Arousal and Valence levels.

[0070] In the dataset, 1 second of baseline signal was subtracted from each second of the subject's experimental signal. In the recognition task, the EEG signal was segmented into 3-second segments. For the DEAP dataset, the 60-second experimental signal of each trial was divided into 20 segments. Since the DEAP dataset has 40 trials, 800 EEG samples were generated for each subject. The sampling rate of the dataset is 128Hz, so the size of each sample in the DEAP dataset is 32×384. In the experiment, 10-fold cross validation was used to evaluate the model in a subject-dependent manner. In each cross validation, the DEAP dataset includes 720 training samples and 80 test samples. The average accuracy through 10 cross validations is used as the result of a single subject, and the average result of all subjects is used as the final accuracy.

[0071] Table 1 Average accuracy (%) and standard deviation of different methods for Arousal and Valence classification tasks in DEAP

[0072]

[0073] Table 1 shows the average accuracy and standard deviation of the present invention and other methods on the DEAP dataset. Compared with the three traditional methods (SVM, DT and MLP), the present invention shows the best performance in emotion recognition of EEG signals. Specifically, the average accuracy of the present invention in Arousal and Valence reached 99.78% and 99.81% respectively, exceeding the seven latest methods (DGCNN, Conti-CNN, CRAM, ACRNN, gcForest, FP-CapsNet and TNAS). Among them, TNAS, which followed closely, recorded the second highest average accuracy, but there was still a gap of 1.1% compared with the present invention. Figure 6 The classification accuracy of the proposed method in terms of arousal and valence for the first 20 subjects in the DEAP dataset is shown in detail. In addition, from the perspective of standard deviation, the standard deviation of the proposed method is less than 1%, which indicates that the results of the proposed method fluctuate less between different subjects and have higher consistency and reliability.

[0074] Table 2 Ablation experiments on DEAP

[0075]

[0076] In order to further demonstrate the effectiveness of the method of the present invention, an ablation experiment was conducted on the DEAP dataset to illustrate the influence of each module in the model. The results of the ablation experiment are shown in Table 2. First, the impact of BD Pooling was evaluated. Considering the necessity of the pooling layer, the commonly used maximum pooling was used instead. The classification accuracy of the maximum pooling was 99.04% and 99.03% on Arousal and Valence, respectively, which was lower than the 99.78% and 99.81% of this method. The results show that BDPooling is more effective in extracting features. Secondly, the classification accuracy of using traditional LSTM to extract temporal features reached 87.28% and 83.73%, respectively, which was 11.76% and 15.3% lower than that of the BiSRU self-attention method (BSA), indicating that BiSRU combined with the improved multi-dimensional self-attention mechanism can enable the model to more effectively capture the temporal feature information of EEG signals and fully consider the intrinsic connection between EEG signals. In addition, when the spatial and frequency domain information of EEG is not considered through the spatial-frequency domain mixed channel attention mechanism (SSDFCA), the accuracy is reduced by 34.69% and 41.28% respectively compared with the proposed method. The final experimental results also show that each module has a significant impact on the improvement of emotion recognition classification performance.

[0077] In order to further explore the impact of mixed channel attention in space-frequency domain on EEG signals, an experiment was conducted to calculate the channel weights of EEG signals. Figure 7The average channel weights of all subjects in the DEAP dataset are shown. In the DEAP dataset, there are 32 channels. Figure 7 , it can be found that in the DEAP dataset, the weights of FC5, C4, P8 and O2 channels in two dimensions are significantly higher than other channels. The larger the channel weight, the more important the channel is in capturing emotion-related signals. This shows that the spatial-frequency domain hybrid channel attention mechanism can effectively identify EEG signal channels that are closely related to emotions, which is of great significance for further improving the accuracy of EEG signal emotion recognition.

[0078] The advantages of the proposed BD pooling method over the maximum pooling and average pooling commonly used for emotion recognition of EEG signals are further demonstrated. Figure 8 As shown in the figure, the average test accuracy of 32 subjects in the Arousal dimension on the DEAP dataset at each epoch of the 10-fold cross-validation experiment is shown. As can be seen from the figure, the proposed BD pooling method can achieve high accuracy in the 20th epoch, which shows that the method can highlight important features while retaining more detailed information, thereby improving classification performance and efficiency.

Claims

1. An emotion recognition method based on a multi-domain attention fusion network, characterized in that: The following steps are involved: Step 1: Obtain EEG signals and preprocess the EEG signals; Step 2: The preprocessed EEG signal is passed through the spatial-frequency domain hybrid channel attention module to extract channel features and capture weighted attention features; Step 3: The weighted attention features are passed through the CNN module to extract the spatial and frequency domain features of the EEG signal; Step 4: Combine the improved multi-dimensional self-attention with BiSRU to capture the comprehensive features of spatial and frequency domain features; Step 5: Classify the comprehensive features through the Softmax function and output the emotion recognition results.

2. The emotion recognition method based on multi-domain attention fusion network according to claim 1 is characterized in that: The specific implementation process of step 2 is: Step 2-1: Perform Fourier transform on the EEG signal and use the differential entropy formula to extract the frequency domain feature H(x) of a single channel: H(x)=-∫p(x)log[p(x)]dx Where x represents the EEG signal, and p(x) represents the probability density function of x. The frequency domain features of the c channels of each sample are concatenated to obtain the frequency domain features of each sample. Step 2-2: Perform mean pooling on each channel of the preprocessed EEG to obtain channel statistics: in is the average value of the jth channel, Obtaining spatial domain features through dimensionality conversion Step 2-3: Use the extracted frequency domain features and spatial domain features to weight them through the weight matrix; the mechanism of spatial-frequency domain mixed channel attention is expressed as: p=sigmoid(W s2 ·(tanh(W s1 ·F s +b s1 )+b s2 )+W f2 ·(tanh(W f1 ·F f +b f1 )+b f2 )+b) Where W s1 and W f1 Represents the dimension reduction weighting matrix of spatial domain and frequency domain features, W s2 and W f2 Represents the dimension-increasing weighted matrix of spatial domain and frequency domain features, b s1 and b f1 is the bias term of the dimension reduction layer, b s2 and b f2 is the bias term of the dimension-increasing layer, and b is the bias term of the mixed feature weight; The tanh function is used for nonlinear activation of the dimension reduction layer; the sigmoid function converts the importance of the channel into different probability distributions p = [p1, p2, ..., p c ], which represents the importance of different channels; Step 2-4: Preprocessed EEG signal S i =[s1,s2,...,s c ], using probability as weight for re-encoding, the attention channel feature of the jth channel extracted by spatial-frequency domain mixed channel attention is expressed as: c j =p j ·s j C={C1,C2,...,C n } represents the extracted channel attention feature, the i-th extracted feature C i =[c1,c2,...,c c ] by S i =[s1,s2,...,s c ] corresponds to the channel and p=[p1,p2,...,p c ] is obtained by multiplying the corresponding elements in .

3. The emotion recognition method based on multi-domain attention fusion network according to claim 2 is characterized in that: The specific implementation process of step 3 is as follows: Step 3-1: The number of convolution kernels is K, and its height is the same as the number of channels. The exponential linear unit ELU is selected as the activation function of the convolution operation. The i-th convolution feature C i ′,i=1,2,...,n is the i-th weighted attention feature C i Obtained through convolution and activation operations; Step 3-2: BD Pooling is used to pool the convolutional features. BD Pooling first performs a bilinear interpolation operation on the input within each step size and window size, obtains n corresponding interpolations, and then represents the features in the pooling window by calculating the interpolation average; each feature encoding Q i It is expressed as the result of interpolation and averaging within the pooling window, and the formula is: Among them, P i,j It is the value obtained after bilinear interpolation within pooling window i. Bilinear interpolation calculation is performed inside each pooling window to obtain the summarized feature representation; Step 3-3: Apply dropout operation to the summarized feature representation to obtain spatial and frequency domain features.

4. The emotion recognition method based on multi-domain attention fusion network according to claim 3 is characterized in that: The specific implementation process of step 4 is as follows: Step 4-1: Bidirectional simple recurrent unit BiSRU receives two inputs, namely the spatial and frequency domain features of the current time step and the input of the previous time step; The bidirectional simple recurrent unit is an extension of the simple recurrent unit. By processing the forward and reverse information of the sequence simultaneously, the i-th output of the bidirectional simple recurrent unit network is {h i ′|h i ′=BiSRU(Q i ),i=1,2,...,s}; Step 4-2: Through the improved multi-dimensional self-attention mechanism, the comprehensive features of spatial and frequency domain features are extracted from the output features of the bidirectional simple recurrent unit.

5. The emotion recognition method based on multi-domain attention fusion network according to claim 4 is characterized in that: The step 4-2 is specifically implemented as follows: Step 4-2-1: Calculate the similarity within each sample from different points; feature score vector z i Considered as the i-th feature h i ′; the improved multi-dimensional self-attention mechanism adds two bias terms inside and outside the activation function, where z i It is expressed as: With i =f(h i ′)=In T σ(W1h i ′+b1)+b Among them, z i represents the intrinsic similarity of the i-th encoded EEG sample; the activation function σ adopts ELU, W and b are the weight term and bias term of the function respectively, W1 is the weight parameter, and b1 is the bias term; Step 4-2-2: Calculate the probability matrix P for each sample ki =[p1,p2,...p i ,...,p s ]: The probability of the i-th EEG sample is expressed as: Step 4-2-3: Comprehensive feature F extracted by improved multi-dimensional self-attention mechanism att ={F1,F2,...,F n }, the i-th attention feature extracted by the improved multi-dimensional self-attention mechanism is expressed as: F i =p i ·h i ′.

Images