A dynamic spatio-temporal CNN-transformer emotion brain-computer interface decoding method

Abstract

The application discloses a dynamic space-time CNN-Transformer emotion brain-computer interface decoding method, relates to the technical field of brain-computer interface, and is characterized in that a dynamic time feature extraction module is designed according to the sensitivity of multi-scale convolution to the time dimension of electroencephalogram sequences, and different size convolution kernels are used to mine time feature information of the electroencephalogram sequences; the asymmetry of the left and right brain regions of the brain is closely related to the emotional state, a local-global space feature extraction module is constructed, and the spatial feature information of the left brain, the right brain and the whole brain of the electroencephalogram sequences is extracted in sequence; a space-time feature fusion module is designed to mine the space-time feature relationship among the left brain, the right brain and the whole brain; the advantage of the long-time sequence processing of the Transformer is used for reference, the long-time dependence relationship in the electroencephalogram sequences is captured through an attention mechanism, and the decoding precision of the emotional electroencephalogram is effectively improved; finally, a multilayer perception machine is used for emotion recognition on the feature sequence coded by the Transformer, and end-to-end emotion electroencephalogram decoding is realized.

Description

TECHNICAL FIELD

[0001] The application relates to the technical field of brain-computer interfaces, in particular to a dynamic spatio-temporal CNN-Transformer emotion brain-computer interface decoding method. BACKGROUND

[0002] An emotion brain-computer interface aims to recognize the emotional state of an individual by analyzing electroencephalogram signals and has wide application prospects in medical rehabilitation, intelligent education and human-computer interaction. In current emotion brain-computer interface decoding methods, traditional feature engineering methods mainly extract time domain, frequency domain, time-frequency domain and nonlinear features of electroencephalogram signals, and use traditional machine learning classification algorithms (Naive Bayes, Linear Discriminant Analysis, Support Vector Machine, Random Forest) for emotion decoding. However, these methods excessively rely on the quality of manually extracted emotion electroencephalogram features and are difficult to quantify uniformly and standardize. Deep learning decoding methods complete end-to-end emotion decoding of electroencephalogram signals by constructing neural networks. For example, deep belief networks, autoencoders, convolutional neural networks (CNN), recurrent neural networks and generative adversarial networks have been applied to emotion brain-computer interface decoding. However, the above deep learning algorithms belong to black box models and have poor model interpretability. They have insufficient cross-subject generalization ability, single electroencephalogram signal time dimension feature extraction and weak spatial dimension lead feature fusion, and still need to be combined with specific neuroscience knowledge of emotion electroencephalogram signals for further analysis.

[0003] In addition, the existing CNN and Transformer-based electroencephalogram emotion decoding methods have the following disadvantages: insufficient spatio-temporal feature extraction, difficulty for CNN algorithms to simultaneously capture short-term transient features and long-term dependencies in electroencephalogram signals; poor dynamic adaptability, large differences in electroencephalogram signal features of different individuals or the same subject at different times, and lack of dynamic adjustment capability of existing models; low computational efficiency, direct use of Transformer to process original high-dimensional electroencephalogram data will result in a dramatic increase in the number of parameters and training difficulties. SUMMARY

[0004] The technical solution of the application to solve the above technical problems is to provide a dynamic spatio-temporal CNN-Transformer emotion brain-computer interface decoding method, comprising the following steps:

[0005] (1) Dynamic time feature extraction: input the original electroencephalogram signal into a dynamic time feature extraction module, extract different time dimension feature sequences through multi-scale one-dimensional time convolution, and splice the time dimension to obtain a time feature sequence;

[0006] (2) Local-global spatial feature extraction: spatial convolution processing is performed on the time feature sequence in the left brain area, the right brain area and the whole brain area respectively to generate a local-global spatial feature sequence, and the spatial feature sequence is obtained by splicing along the spatial dimension;

[0007] (3) Spatio-temporal feature fusion: spatio-temporal convolution operation is performed on the spatial feature sequence to fuse the spatio-temporal correlation features of the left brain area, the right brain area and the whole brain area, and output a spatio-temporal feature fusion sequence;

[0008] (4) Transformer sequence encoding: the long-term dependency relationship of the spatio-temporal feature fusion sequence is modeled by a Transformer encoder, and the context emotional information is extracted by using a multi-head attention mechanism to generate an encoding feature sequence;

[0009] (5) Multi-layer perception classification and recognition: the encoding feature sequence is input into a multi-layer perception classifier, and a sentiment state classification probability is output based on a cross-entropy loss function;

[0010] (6) Model training and testing: the training set and the test set are divided according to the test times, the model is iteratively trained and the optimal parameters are saved, and the test set is input into the trained model to output the emotional recognition result.

[0011] Further, the step (1) specifically comprises:

[0012] The one-dimensional time convolution layer with different convolution kernel sizes is used to process the electroencephalogram signal respectively;

[0013] The time feature sequence is generated by sequentially passing through a nonlinear activation function, an average pooling layer and a batch normalization layer 、 、 ;

[0014] The sequences are spliced along the time dimension, and the specific formula is as follows:

[0015] .

[0016] Further, the step (2) specifically comprises:

[0017] Local spatial processing: the time feature sequence is processed by spatial convolution in the left and right brain area lead channel respectively using a spatial convolution layer; the nonlinear activation function is used to perform nonlinear transformation on the feature data after convolution; then the average pooling layer is used for dimension reduction processing of the feature data; finally, the batch normalization layer is used for normalization operation of the feature data, and the local spatial feature sequence left brain area and right brain area are obtained; the specific formula is as follows:

[0018] ;

[0019] ;

[0020] wherein, , respectively represent the size of the local spatial convolution kernel, respectively represent the number of left brain region lead channels and the number of right brain region lead channels, represents the time feature sequence, represents a one-dimensional spatial convolution layer, represents a nonlinear activation function, represents an average pooling layer, represents a batch normalization layer;

[0021] Global spatial processing: using a spatial convolution layer to perform spatial convolution processing on the time feature sequence in the whole brain region lead channel; then using a nonlinear activation function to perform nonlinear transformation on the feature data after convolution; then using an average pooling layer to perform dimension reduction processing on the feature data; finally using a batch normalization layer to perform normalization operation on the feature data, to obtain a global spatial feature sequence ; the specific formula is as follows:

[0022] ;

[0023] wherein, represents the size of the global spatial convolution kernel, represents the number of whole brain lead channels, represents the time feature sequence, represents a one-dimensional spatial convolution layer, represents a nonlinear activation function, represents an average pooling layer, represents a batch normalization layer;

[0024] The left brain region , the right brain region and the global spatial feature sequence of the whole brain are spliced along the spatial dimension direction to obtain a local-global spatial feature sequence ; the specific formula is as follows:

[0025] .

[0026] Further, the step (3) specifically comprises:

[0027] using a spatio-temporal convolution layer to perform spatio-temporal convolution operation on the local-global spatial feature sequence;

[0028] then using a nonlinear activation function Nonlinear transformation is performed on the feature data after spatio-temporal convolution to enhance the expression ability of the model;

[0029] Then use the average pooling layer Dimensionality reduction processing is performed on the feature data;

[0030] Finally, use the batch normalization layer Normalization operation is performed on the feature data to obtain the spatio-temporal feature fusion sequence The specific formula is as follows:

[0031] Wherein, represents the size of the spatio-temporal convolution kernel, represents the left brain, right brain and whole brain feature sequence dimension, represents the spatial feature sequence, represents a one-dimensional spatio-temporal convolution layer, represents a nonlinear activation function, represents an average pooling layer, represents a batch normalization layer.

[0032] Further, the step (4) specifically comprises:

[0033] a. Construct a Transformer sequence encoding module including multi-head attention mechanism, residual connection, layer normalization and feedforward neural network;

[0034] b. Use multi-head self-attention to capture the complex relationship between spatio-temporal feature fusion sequences: calculate the correlation weight between elements in the sequence to capture the long-term dependence relationship of the emotional state contained in the spatio-temporal feature fusion sequence;

[0035] Calculate the linear transformation of Query (Q), Key (K) and Value (V), and the specific formula is as follows:

[0036] ;

[0037] ;

[0038] ;

[0039] Wherein, , , is a learning parameter matrix;

[0040] Calculate the scaled dot-product attention, and the specific formula is as follows:

[0041] ;

[0042] Wherein, is the dimension of the key vector, and the scaling factor Prevent the dot product result from being too large to cause gradient disappearance;

[0043] Attention calculation is decomposed into heads, each head focuses on the connection of different subspaces, and finally splicing and projection are performed, and the specific formula is as follows:

[0044] Wherein, , , Query, Key and Value in the first Head weight matrix; the output matrix is obtained by splicing each attention head and performing dot product with the weight matrix ;

[0045] c. Use residual connection to add residual operation after each sublayer to alleviate the gradient disappearance problem existing in deep network and ensure effective transmission of feature information; use layer normalization to normalize all feature dimensions of each sample, and the specific formula is as follows:

[0046] ;

[0047] d. Use feedforward neural network to perform nonlinear transformation on the normalized features to output the electroencephalogram feature representation containing global information of context; the specific formula is as follows:

[0048] ;

[0049] e. After the output of the feedforward neural network, residual connection and layer normalization are performed again to obtain the Transformer encoding feature sequence; the specific formula is as follows:

[0050] .

[0051] Further, the step (5) specifically comprises:

[0052] Use multilayer perceptron and cross-entropy loss function to perform emotion classification, and the specific formula is as follows:

[0053] ;

[0054] ;

[0055] Wherein, represents the Transformer encoding feature sequence, represents the multilayer perceptron; represents the number of batchsize, represents the task category corresponding to the emotion electroencephalogram; and These represent the actual label and the predicted label for the corresponding sentiment category, respectively.

[0056] Furthermore, step (6) specifically includes:

[0057] The training set is further divided into a training subset and a validation subset;

[0058] Iterate 150 times with a batch size of 32 and a learning rate of 0.001;

[0059] Save the model parameters that achieve the highest accuracy on the validation set;

[0060] The test set is input into the training model with a batch size of 32 to output the classification results.

[0061] Compared with the prior art, the technical solution of the present invention has outstanding substantive features and significant advantages:

[0062] (1) This invention proposes a dynamic spatiotemporal CNN-Transformer emotion brain-computer interface decoding method. It draws on the sensitivity of multi-scale convolution to EEG sequences along the time dimension, designs a dynamic time feature extraction module, and uses convolution kernels of different sizes to mine the temporal feature information contained in the EEG sequence.

[0063] (2) This invention draws on the fact that the asymmetry between the left and right hemispheres of the brain is closely related to emotional state. It designs a local-global spatial feature extraction module to fully characterize the differences in different brain regions of different emotional states. Combined with a spatiotemporal feature fusion module, it explores the spatiotemporal feature fusion and information between the left brain, right brain and whole brain.

[0064] (3) This invention draws on the advantages of Transformer in processing long time sequences, performs spatiotemporal modeling through attention mechanism, captures global spatiotemporal dependencies in EEG sequences, and effectively improves the accuracy of emotional EEG decoding. Attached Figure Description

[0065] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the structures shown in these drawings without creative effort.

[0066] Figure 1 This is a flowchart illustrating the steps of the dynamic spatiotemporal CNN-Transformer emotion brain-computer interface decoding method described in this invention;

[0067] Figure 2 This is a flowchart of the dynamic time feature extraction process described in this invention;

[0068] Figure 3 A local-global spatial feature extraction flowchart for the present application;

[0069] Figure 4 A spatio-temporal feature fusion flowchart for the present application;

[0070] Figure 5 A Transformer sequence encoding flowchart for the present application;

[0071] Figure 6 A multi-layer perception machine classification and recognition flowchart for the present application;

[0072] Figure 7 A model training and testing flowchart for the present application. DETAILED DESCRIPTION

[0073] The present application proposes a dynamic spatio-temporal CNN-Transformer emotion brain-computer interface decoding method, aiming to effectively extract the spatio-temporal features in the electroencephalogram signal and realize efficient emotion classification.

[0074] The dynamic spatio-temporal CNN-Transformer emotion brain-computer interface decoding method proposed by the present application will be described below in specific embodiments:

[0075] Embodiment 1:

[0076] A dynamic spatio-temporal CNN-Transformer emotion brain-computer interface decoding method, as shown in Figure 1 , includes the following steps:

[0077] (1) Dynamic temporal feature extraction: input the original electroencephalogram signal into the dynamic temporal feature extraction module, mine the multi-scale temporal features of the electroencephalogram signal, and obtain the electroencephalogram temporal feature sequences , , extracted by different size convolution kernels in turn ;

[0078] (2) Local-global spatial feature extraction: the time feature sequence is subjected to spatial convolution processing of the left brain area, the right brain area and the whole brain area respectively, to generate local-global spatial feature sequences , and , and the spatial feature sequence is obtained by splicing along the spatial dimension

[0079] (3) Spatio-temporal feature fusion: the spatial feature sequence The spatio-temporal convolution operation is performed to fuse the spatio-temporal correlation features of the left brain region, the right brain region and the whole brain region, and output a spatio-temporal feature fusion sequence ;

[0080] (4) Transform sequence encoding: modeling the long-term dependency relationship of the spatio-temporal feature fusion sequence by a Transform encoder, extracting context sentiment information by a multi-head attention mechanism, and generating an encoding feature sequence ; ;

[0081] (5) Multi-layer perception classification and recognition: inputting the encoding feature sequence into a multi-layer perception classifier to output a sentiment state classification probability based on a cross-entropy loss function;

[0082] (6) Model training and testing: dividing the training set and the test set according to the test times, iteratively training the model and saving the optimal parameters, and inputting the test set into the trained model to output the sentiment recognition result.

[0083] Specifically, the preprocessed electroencephalogram data is divided into a training set and a test set according to the test times, ensuring that the electroencephalogram data of the same test time only appears in the training set or the test set. The training set electroencephalogram data is input into the dynamic spatio-temporal CNN-Transform model in batches for training operation, and the dynamic spatio-temporal CNN-Transform training model is obtained after several iterations, and the related model parameters are saved. Then, the test set electroencephalogram data is input into the dynamic spatio-temporal CNN-Transform training model in batches, and the test set electroencephalogram data recognition result is output.

[0084] Further, the step (1) specifically comprises:

[0085] The one-dimensional time convolution layer with different convolution kernel sizes is used to process the electroencephalogram signal respectively;

[0086] The time feature sequence is generated by sequentially passing through a nonlinear activation function, an average pooling layer and a batch normalization layer 、 、 ;

[0087] The sequences are spliced along the time dimension, and the specific formula is as follows:

[0088] .

[0089] Specifically, a dynamic time feature extraction module is constructed, which includes a multi-scale one-dimensional time convolution layer , a nonlinear activation function layer , an average pooling layer , and a batch normalization layer .

[0090] One-dimensional temporal convolutional layers using small, medium, and large kernels in sequence The raw EEG data is processed by convolution; then a non-linear activation function is used. A non-linear transformation is performed on the convolutional feature data to enhance the model's expressive power; then an average pooling layer is used. Dimensionality reduction of the feature data is performed to reduce the computational complexity of the model; finally, a batch normalization layer is used. Normalizing the feature data suppresses model overfitting and achieves a regularization effect, thereby obtaining the time feature sequence. , , The specific formula is as follows:

[0091] ;

[0092] ;

[0093] ;

[0094] in, , , These represent convolution kernels of different sizes. This represents the input EEG sample data. Represents a one-dimensional temporal convolutional layer. Represents a non-linear activation function. Indicates the average pooling layer. Representatives approved the normalization layer.

[0095] Temporal feature sequences obtained using convolution kernels of different sizes , , Perform a concatenation operation along the time feature dimension to obtain the concatenated feature sequence. The specific formula is as follows:

[0096] .

[0097] Furthermore, step (2) specifically includes:

[0098] Local spatial processing: Spatial convolutional layers were used to perform spatial convolution processing on the left and right brain region leads of the temporal feature sequences, respectively; non-linear activation functions were used. Perform a non-linear transformation on the convolutional feature data; then use an average pooling layer. Dimensionality reduction of the feature data is performed; finally, a batch normalization layer is used. Normalization of the feature data yields the local spatial feature sequence of the left brain region. and right brain region ; the specific formula is as follows:

[0099] ;

[0100] ;

[0101] wherein, , respectively represent the size of the local spatial convolution kernel, respectively represent the number of left brain region lead channels and the number of right brain region lead channels, represents a time feature sequence, represents a one-dimensional spatial convolution layer, represents a nonlinear activation function, represents an average pooling layer, represents a batch normalization layer;

[0102] Global spatial processing: using a spatial convolution layer to perform whole brain region lead channel spatial convolution processing on the time feature sequence; then using a nonlinear activation function to perform nonlinear transformation on the convolutional feature data; then using an average pooling layer to perform dimension reduction processing on the feature data; finally using a batch normalization layer to perform normalization operation on the feature data, to obtain a global spatial feature sequence ; the specific formula is as follows:

[0103] ;

[0104] wherein, represents the size of the global spatial convolution kernel, represents the number of whole brain lead channels, represents a time feature sequence, represents a one-dimensional spatial convolution layer, represents a nonlinear activation function, represents an average pooling layer, represents a batch normalization layer;

[0105] The local spatial feature sequence left brain region , right brain region and the global spatial feature sequence whole brain are spliced along the spatial dimension direction to obtain a local-global spatial feature sequence ; the specific formula is as follows:

[0106] .

[0107] Specifically, a local-global spatial feature extraction module is constructed, which includes a spatial convolution layer , a nonlinear activation function layer average pooling layer batch normalization layer .

[0108] Further, the step (3) specifically includes:

[0109] spatiotemporal convolution operation is performed on the local-global spatial feature sequence using a spatiotemporal convolution layer;

[0110] Then a nonlinear activation function is used to perform nonlinear transformation on the feature data after spatiotemporal convolution, enhancing the model expression ability;

[0111] Then an average pooling layer is used to perform dimension reduction processing on the feature data;

[0112] Finally, a batch normalization layer is used to perform normalization operation on the feature data to obtain a spatiotemporal feature fusion sequence ; the specific formula is as follows:

[0113] wherein, represents the size of the spatiotemporal convolution kernel, represents the left brain, right brain and whole brain feature sequence dimension, represents the spatial feature sequence, represents a one-dimensional spatiotemporal convolution layer, represents a nonlinear activation function, represents an average pooling layer, represents a batch normalization layer.

[0114] Specifically, a spatiotemporal feature fusion module is constructed, which mainly includes a spatiotemporal convolution layer , a nonlinear activation function layer , an average pooling layer , and a batch normalization layer .

[0115] Further, the step (4) specifically includes:

[0116] a. Construct a Transformer sequence encoding module including multi-head attention mechanism, residual connection, layer normalization and feedforward neural network;

[0117] b. Use multi-head self-attention to capture the complex relationship between the spatiotemporal feature fusion sequence: calculate the correlation weight between the elements in the sequence to capture the long-term dependence relationship of the emotional state contained in the spatiotemporal feature fusion sequence;

[0118] Calculate the linear transformation of Query (Q), Key (K) and Value (V), and the specific formula is as follows:

[0119] ;

[0120] ;

[0121] ;

[0122] where, , , is the learnable parameter matrix;

[0123] The scaled dot-product attention is calculated as follows:

[0124] ;

[0125] where, is the dimension of the key vector, and the scaling factor prevents the dot-product result from being too large, causing gradient vanishing;

[0126] The attention computation is decomposed into heads, each of which focuses on the connection of different subspaces, and finally spliced and projected, as follows:

[0127] where, , , are the weight matrices of Query, Key and Value in the head; the output matrix is obtained by splicing each attention head and performing dot product with the weight matrix ;

[0128] c. A residual operation is added after each sub-layer using a residual connection to alleviate the gradient vanishing problem existing in deep networks and ensure effective transmission of feature information; layer normalization is used to normalize all feature dimensions of each sample, as follows:

[0129] ;

[0130] d. A feedforward neural network is used to perform nonlinear transformation on the normalized features to output the electroencephalogram feature representation containing global information of the context; the specific formula is as follows:

[0131] ;

[0132] e. After the output of the feedforward neural network, residual connection and layer normalization are performed again to obtain the Transformer encoding feature sequence; the specific formula is as follows:

[0133] .

[0134] Further, the step (5) specifically comprises:

[0135] The sentiment classification is performed using a multi-layer perceptron and a cross-entropy loss function, and the specific formula is as follows:

[0136] ;

[0137] ;

[0138] Wherein, represents a Transformer encoding feature sequence, represents a multi-layer perceptron; represents the number of batchsize, represents the task category corresponding to the emotional electroencephalogram; and respectively represent the real label and the predicted label corresponding to the emotional category.

[0139] Specifically, a Transformer sequence encoding module is constructed, which mainly includes a multi-head attention mechanism, a residual connection, a layer normalization, and a feedforward neural network. The multi-head self-attention (MHA) is used to capture the complex relationship between the spatio-temporal feature fusion sequence , the residual connection and the layer normalization LayerNorm are combined to ensure the stability of the training, and finally the feedforward neural network (FFN) is used to output the electroencephalogram feature representation containing the context of the global information.

[0140] Further, the step (6) specifically comprises:

[0141] The training set is subdivided into a training subset and a validation subset;

[0142] Iterative training is performed 150 times with a batch size of 32 and a learning rate of 0.001;

[0143] The model parameters with the highest validation set accuracy are saved;

[0144] The test set is input into the trained model with a batch size of 32 to output the classification result.

[0145] Specifically, the preprocessed electroencephalogram data is divided into a training set and a test set according to the test , wherein the electroencephalogram data of the training set is subdivided into a training part and a validation part , to ensure that the electroencephalogram data of the same test appears only in the training set or the test set.

[0146] The training set of EEG data is input into the dynamic spatiotemporal CNN-Transformer model in batches for model training, with the number of samples in each batch being... Set the learning rate to 32. Set to 0.001. The dynamic spatiotemporal CNN-Transformer model is trained using EEG data from the training portion of the training set, and the loss function value of the model is monitored using EEG data from the validation portion of the training set. and classification accuracy The number of iterations is set to 150.

[0147] When the number of iterations When the value is 1, update the parameters of the dynamic spatiotemporal CNN-Transformer model, calculate and save the loss function value of the validation part. and classification accuracy Similarly, when the number of iterations... When the value is 2, update the parameters of the dynamic spatiotemporal CNN-Transformer model, calculate and save the loss function value of the validation part. and classification accuracy The iteration count is repeated until 150. The classification accuracy is compared across these 150 iterations, and the model parameters with the highest accuracy in the validation phase during training are used as the final dynamic spatiotemporal CNN-Transformer model parameters. Then, the test set EEG data is input into the final dynamic spatiotemporal CNN-Transformer training model in batches, with the number of samples in each batch... Set to 32 to output the emotion recognition results of the test set EEG data.

[0148] Example 2:

[0149] A dynamic spatiotemporal CNN-Transformer emotion brain-computer interface decoding method includes the following steps:

[0150] (1) Dynamic temporal feature extraction:

[0151] Assuming the raw EEG signal can be represented as The corresponding sentiment tags can be represented as ,in The number of samples representing electroencephalogram (EEG) signals. This represents the number of lead channels along the spatial dimension for each EEG sample. This indicates the data length of each EEG sample along the time dimension. , This indicates the sampling frequency of the electroencephalogram (EEG) signal. Indicates time.

[0152] The original electroencephalogram signal The dynamic time feature extraction module is sequentially input, the multi-scale time features of the electroencephalogram signal are mined, and the electroencephalogram time feature sequences extracted by different size convolution kernels are sequentially obtained 、 、 The obtained electroencephalogram feature sequence is spliced along the time dimension to obtain a spliced time feature sequence .

[0153] (2) Local-global spatial feature extraction:

[0154] The time feature sequence is input into the local-global spatial feature extraction module, the asymmetry of the left and right brain regions of the brain is used as reference, and the spatial features of the left brain region, the right brain region and the whole brain of the electroencephalogram signal are sequentially extracted by using the local-global spatial convolution module, to form a local-global spatial feature sequence , and . The obtained local-global spatial feature sequence is spliced along the spatial dimension to obtain a spliced spatial feature sequence .

[0155] (3) Spatio-temporal feature fusion

[0156] The spatial feature sequence is input into the spatio-temporal feature fusion module, the connection among the left brain region, the right brain region and the whole brain of the spatial feature sequence is mined by using spatio-temporal convolution, and a spatio-temporal feature fusion sequence is obtained .

[0157] (4) Transforemr sequence encoding

[0158] The spatio-temporal feature fusion sequence is input into the Transformer sequence encoding module, the long-term dependency relationship among the electroencephalogram spatio-temporal features is modeled by using the Transformer encoder, the context emotional information contained in the spatio-temporal feature fusion sequence is mined by using the multi-head attention mechanism, and a Transformer encoded feature sequence is obtained .

[0159] (5) Multi-layer perceptron classification and recognition

[0160] The Transformer encoded feature sequence is input into the multi-layer perceptron classifier, the cross-entropy function is used as the overall loss function of the model, and the recognition and classification of the emotional state are realized by using the full connection layer.

[0161] (6) Model training and testing

[0162] The preprocessed electroencephalogram data is divided into a training set and a test set according to the trial, ensuring that the electroencephalogram data of the same trial only appears in the training set or the test set. The training set electroencephalogram data is input into the dynamic spatio-temporal CNN-Transformer model in batches for training operation, and is iteratively trained for several times to obtain a dynamic spatio-temporal CNN-Transformer training model, and relevant model parameters are saved. Then the test set electroencephalogram data is input into the dynamic spatio-temporal CNN-Transformer training model in batches, and the test set electroencephalogram data recognition result is output.

[0163] Further, the step of extracting dynamic temporal features comprises:

[0164] A dynamic temporal feature extraction module is constructed, as shown in Figure 2 , which mainly includes a multi-scale one-dimensional time convolution layer , a nonlinear activation function layer , an average pooling layer , and a batch normalization layer .

[0165] The one-dimensional time convolution layer with small, moderate and large convolution kernels is used in sequence to perform convolution processing on the original electroencephalogram data; then the nonlinear activation function is used to perform nonlinear transformation on the convolution feature data to enhance the model expression ability; then the average pooling layer is used to perform dimension reduction processing on the feature data to reduce the computational complexity of the model; finally, the batch normalization layer is used to normalize the feature data to suppress model overfitting and achieve regularization effect, and the time feature sequence , , is obtained in sequence. The specific formula is as follows:

[0166] ;

[0167] ;

[0168] ;

[0169] Among them, , , represent convolution kernels of different sizes, represents the input electroencephalogram sample data, represents the one-dimensional time convolution layer, represents the nonlinear activation function, represents the average pooling layer, This represents the batch normalization layer. In this invention, it will... Set to (1,16), Set to (1,32), Set to (1, 64), number of convolutional kernels to 32, average pooling. Set it to (1,8).

[0170] Temporal feature sequences obtained using convolution kernels of different sizes , , Perform a concatenation operation along the time feature dimension to obtain the concatenated feature sequence. The specific formula is as follows:

[0171]

[0172] Furthermore, the steps of local-global spatial feature extraction include:

[0173] Construct a local-global spatial feature extraction module, such as Figure 3 As shown, this module mainly contains spatial convolutional layers. Nonlinear activation function layer Average pooling layer Normalization layer .

[0174] Spatial convolutional layers in the local spatial feature extraction module Time feature series Spatial convolution of the left and right brain region channels was performed separately; then a non-linear activation function was used. A non-linear transformation is performed on the convolutional feature data to enhance the model's expressive power; then an average pooling layer is used. Dimensionality reduction of the feature data is performed to reduce the computational complexity of the model; finally, a batch normalization layer is used. Normalizing the feature data suppresses model overfitting and has a regularization effect, yielding local spatial feature sequences of the left brain region. and the right brain region The specific formula is as follows:

[0175] ;

[0176] ;

[0177] in, , These represent the size of the local spatial convolution kernel, and respectively represent the number of channels in the left and right brain regions. Represents a time-feature sequence. represents a one-dimensional spatial convolution layer, represents a nonlinear activation function, represents an average pooling layer, represents a batch normalization layer. In the present application, is set to (16, 1), is set to (16, 1), the number of convolution kernels is set to 32, and the average pooling is set to (1, 8).

[0178] Similarly, the spatial convolution layer in the global spatial feature extraction module is used to perform spatial convolution processing on the time feature sequence ; then a nonlinear activation function is used to perform nonlinear transformation on the feature data after convolution, enhance the expression ability of the model, and then an average pooling layer is used to perform dimension reduction processing on the feature data, reduce the calculation complexity of the model, and finally a batch normalization layer is used to perform normalization operation on the feature data, suppress model overfitting, and play a regularization effect to obtain a global spatial feature sequence . The specific formula is as follows:

[0179] ;

[0180] wherein, represents the size of the global spatial convolution kernel, and represents the number of brain lead channels, represents the time feature sequence, represents a one-dimensional spatial convolution layer, represents a nonlinear activation function, represents an average pooling layer, represents a batch normalization layer. In the present application, is set to (32, 1), the number of convolution kernels is set to 32, and the average pooling is set to (1, 8).

[0181] The left brain region , the right brain region , and the global spatial feature sequence of the whole brain are spliced along the spatial dimension direction to obtain a local-global spatial feature sequence . The specific formula is as follows:

[0182] .

[0183] Further, the step of spatiotemporal feature fusion comprises:

[0184] constructing a spatiotemporal feature fusion module, such asFigure 4 As shown, this module mainly contains spatiotemporal convolutional layers. Nonlinear activation function layer Average pooling layer Normalization layer .

[0185] Using the spatiotemporal convolutional layer in the spatiotemporal feature fusion module Local-global spatial feature sequences Perform spatiotemporal convolution operations; then use a nonlinear activation function. Nonlinear transformations are applied to the feature data after spatiotemporal convolution to enhance the model's expressive power; then average pooling layers are used. Dimensionality reduction of the feature data is performed to reduce the computational complexity of the model; finally, a batch normalization layer is used. Normalizing the feature data suppresses model overfitting, achieving a regularization effect and yielding a spatiotemporal feature fusion sequence. The specific formula is as follows:

[0186] in, The size of the spatiotemporal convolution kernel represents the dimension of the feature sequences from the left brain, right brain, and whole brain. Represents spatial feature sequences, Represents a one-dimensional spatiotemporal convolutional layer. Represents a non-linear activation function. Indicates the average pooling layer. This represents the batch normalization layer. In this invention, it will... Set to (3,1), the number of convolutional kernels is set to 32, and average pooling is used. Set it to (1,8).

[0187] Furthermore, the Transformer sequence encoding step includes:

[0188] Construct a Transformer sequence encoding module, such as Figure 5 As shown, this module mainly includes a multi-head attention mechanism, residual connections, layer normalization, and a feedforward neural network. Multi-head attention (MHA) is used to capture spatiotemporal feature fusion sequences. The complex relationships between them are combined with residual connections and layer normalization (LayerNorm) to ensure training stability. Finally, the feedforward network (FFN) outputs a contextual EEG feature representation containing global information.

[0189] Using the Transformer sequence encoding module to fuse spatiotemporal features into sequences Temporal and spatial encoding is performed. The multi-head attention mechanism is the key of the Transformer sequence encoding module, which captures the long-term dependence relationship of the emotional state contained in the temporal and spatial feature sequence by calculating the correlation weight between the elements in the sequence. In the present application, the number of Transformer encoding layers is set to 12, and the number of multi-head attention heads is set to 8. .

[0190] The linear transformation is performed by calculating the query (Query, Q), key (Key, K) and value (Value, V), and the specific formula is as follows:

[0191]

[0192]

[0193]

[0194] wherein, , , is a learning parameter matrix.

[0195] The scaled dot-product attention is calculated, and the specific formula is as follows:

[0196]

[0197] wherein, is the dimension of the key vector, and the scaling factor prevents the gradient from disappearing due to the large dot product result.

[0198] The attention calculation is decomposed into heads, each of which focuses on the connection of different subspaces, and finally splicing and projection are performed, and the specific formula is as follows:

[0199] wherein, , , are the weight matrices of Query, Key and Value in the head respectively. The output matrix is obtained by splicing each attention head and performing dot product with the weight matrix .

[0200] The residual connection is used, and the residual operation is added after each sublayer to alleviate the gradient disappearance problem existing in the deep network and ensure that the feature information can be effectively transmitted. The layer normalization is used to normalize all feature dimensions of each sample, so that it is more suitable for processing temporal and spatial sequences. The specific formula is as follows:

[0201]

[0202] ​​​​​A feedforward neural network is used to perform a nonlinear transformation on the normalized features, increasing the model's expressive power. The specific formula is as follows:

[0203] ;

[0204] After the feedforward neural network outputs, residual connections and layer normalization are performed again to obtain the Transformer-encoded feature sequence. The specific formula is as follows:

[0205] .

[0206] Furthermore, such as Figure 6 As shown, the steps of multilayer perceptron classification and recognition include:

[0207] The feature sequences encoded by Transformer are processed using a Multilayer Perceptron (MLP). This method decodes EEG signals to determine the emotion level and outputs the probability of a specific emotion state. The cross-entropy loss function is used as the model's loss function. The model output and loss function can be expressed as follows:

[0208] ;

[0209] ;

[0210] in, Indicates the number of batch sizes. This indicates the task category corresponding to the emotional EEG. and These represent the true label and the predicted label for the corresponding sentiment category, respectively. In this invention, the number of fully connected layers in the multilayer perceptron is set to 2, and the number of neurons in each layer is set to 32.

[0211] Furthermore, such as Figure 7 As shown, the model training and testing steps include:

[0212] The preprocessed EEG data were divided into training and testing sets according to the number of trials. The training set of EEG data is further subdivided into training components. and verification section This ensures that EEG data from the same trial only appear in the training or test set.

[0213] The training set of EEG data is input into the dynamic spatiotemporal CNN-Transformer model in batches for model training, with the number of samples in each batch being... Set the learning rate to 32. is set to 0.001. The dynamic spatio-temporal CNN-Transformer model is trained using the electroencephalogram data of the training set, and the loss function value of the model is monitored using the electroencephalogram data of the validation set in the training set and the classification recognition accuracy . The number of iterations is set to 150.

[0214] When the number of iterations is 1, the parameters of the dynamic spatio-temporal CNN-Transformer model are updated, the loss function value of the validation set is calculated and saved and the classification recognition accuracy ; similarly, when the number of iterations is 2, the parameters of the dynamic spatio-temporal CNN-Transformer model are updated, the loss function value of the validation set is calculated and saved and the classification recognition accuracy ; until the number of iterations is 150. The classification recognition accuracy during the 150 iterations is compared, and the model parameters with the highest classification recognition accuracy in the validation set during the training process are taken as the final dynamic spatio-temporal CNN-Transformer model parameters. The electroencephalogram data of the test set is then input into the final dynamic spatio-temporal CNN-Transformer training model in batches, and the number of samples in each batch is set to 32, and the emotion recognition results of the test set electroencephalogram data are output.

[0215] The above is only a preferred specific embodiment of the present application, but the protection scope of the present application is not limited thereto, any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A dynamic spatio-temporal CNN-Transformer affective brain-computer interface decoding method, characterized in that, The method comprises the following steps: (1) Dynamic time feature extraction: input the original electroencephalogram signal into the dynamic time feature extraction module, extract different time dimension feature sequences through multi-scale one-dimensional time convolution, and splice to obtain a time feature sequence along the time dimension; (2) Local-global spatial feature extraction: the time feature sequence is subjected to spatial convolution processing in the left brain area, the right brain area and the whole brain area respectively to generate a local-global spatial feature sequence, and the spatial feature sequence is spliced along the spatial dimension; (3) Spatio-temporal feature fusion: the spatial feature sequence is subjected to spatio-temporal convolution operation, the spatio-temporal convolution is used to mine the connection among the left brain area, the right brain area and the whole brain of the spatial feature sequence, and a spatio-temporal feature fusion sequence is output; (4) Transformer sequence encoding: the long-term dependence relationship of the spatio-temporal feature fusion sequence is modeled by a Transformer encoder, the context emotional information is extracted by using a multi-head attention mechanism, and an encoded feature sequence is generated; (5) Multi-layer perception classification and identification: the encoded feature sequence is input into a multi-layer perception classifier, and an emotional state classification probability is output based on a cross-entropy loss function; (6) Model training and testing: the training set and the test set are divided according to the test times, the model is iteratively trained and the optimal parameters are saved, and the test set is input into the trained model to output an emotional recognition result.

2. The method of claim 1, wherein, The step (1) specifically comprises: one-dimensional time convolution layers with different kernel sizes are used to process the electroencephalogram signal; The time feature sequence is generated in sequence through a nonlinear activation function, an average pooling layer, and a batch normalization layer , , ; each sequence is spliced along the time dimension, and the specific formula is as follows: 。 3. The method of claim 1, wherein, The step (2) specifically comprises: Local spatial processing: using spatial convolution layer to respectively perform left and right brain region lead channel spatial convolution processing on the time feature sequence; using a nonlinear activation function Nonlinear transformation is performed on the convolutional feature data; then an average pooling layer is used Dimensionality reduction processing is performed on the feature data; finally, a batch normalization layer is used Normalization operation is performed on the feature data to obtain the left brain region and the right brain region of the local spatial feature sequence; the specific formula is as follows: ； ； wherein, , denote the size of the local spatial convolution kernel, respectively, and represent the number of left and right brain region lead channels, respectively, denotes the time feature sequence, denotes a one-dimensional spatial convolution layer, denotes a nonlinear activation function, denotes an average pooling layer, denotes a batch normalization layer; Global spatial processing: using spatial convolution layer to conduct whole brain area lead channel spatial convolution processing on the time feature sequence; then using nonlinear activation function Nonlinear transformation is performed on the convolutional feature data; then using average pooling layer Dimensionality reduction processing is performed on the feature data; finally using batch normalization layer Normalization operation is performed on the feature data to obtain the global spatial feature sequence The specific formula is as follows: ； wherein, denotes the size of global spatial convolution kernel, represents the number of whole brain lead channels, denotes the time feature sequence, represents a one-dimensional spatial convolution layer, denotes a nonlinear activation function, denotes an average pooling layer, represents a batch normalization layer; The local spatial feature sequence left brain region , right brain region , and global spatial feature sequence whole brain are spliced along the spatial dimension direction to obtain a local-global spatial feature sequence ; the specific formula is as follows: 。 4. The method of claim 1, wherein, The step (3) specifically comprises: a spatio-temporal convolution layer is used to perform spatio-temporal convolution operation on the local-global spatial feature sequence; Then a non-linear activation function is used The feature data after the spatio-temporal convolution is subjected to a non-linear transformation, enhancing the expression capability of the model. Reusing average pooling layers performing dimension reduction processing on the feature data; Finally, a batch normalization layer is used The feature data is normalized to obtain a spatiotemporal feature fusion sequence The specific formula is as follows: wherein, denotes the size of the spatio-temporal convolution kernel, represents the left brain, right brain and whole brain feature sequence dimensions, denotes the spatial feature sequence, represents a one-dimensional spatio-temporal convolution layer, denotes a non-linear activation function, denotes an average pooling layer, represents a batch normalization layer.

5. The method of claim 1, wherein, The step (4) specifically comprises: a. a Transformer sequence encoding module comprising a multi-head attention mechanism, a residual connection, layer normalization and a feedforward neural network is constructed; b. the multi-head self-attention is used to capture the complex connection between the spatio-temporal feature fusion sequences: the correlation weight between the elements in the sequence is calculated to capture the long-term dependence relationship of the emotional state contained in the spatio-temporal feature fusion sequence; the query (Query, Q), the key (Key, K) and the value (Value, V) are calculated for linear transformation, and the specific formula is as follows: ； ； ； wherein , , is a learning parameter matrix; the scaled dot-product attention is calculated, and the specific formula is as follows: ； wherein, is the key vector dimension, scaling factor Prevent the dot product result from being too large to cause gradient vanishing; The attention computation is decomposed into heads, each of which focuses on the connection of a different subspace, and finally stitched and projected, with the specific formula as follows: wherein, , , Query, Key and Value in the first weight matrix of the head; the output matrix is obtained by concatenating each attention head and taking the dot product with the weight matrix ​ c. the residual connection is used to add a residual operation after each sub-layer to relieve the gradient vanishing problem existing in the deep network and ensure that the feature information can be effectively transmitted; the layer normalization is used to normalize all feature dimensions of each sample, and the specific formula is as follows: ； d. the feedforward neural network is used to perform nonlinear transformation on the normalized features to output the electroencephalogram feature representation containing the context of the global information; the specific formula is as follows: ； e. after the feedforward neural network outputs, the residual connection and the layer normalization are performed again to obtain the Transformer encoded feature sequence; the specific formula is as follows: 。 6. The method of claim 1, wherein, The step (5) specifically comprises: a multi-layer perception and a cross-entropy loss function are used for emotional classification, and the specific formula is as follows: ； ； wherein, represents a Transformer encoding feature sequence, represents a multi-layer perceptron; represents the number of batchsize, represents a task category corresponding to the emotional EEG; and respectively represent the real label and the predicted label corresponding to the emotional category.

7. The method of claim 1, wherein, The step (6) specifically comprises: The training set is subdivided into a training subset and a validation subset; the batch size is 32, the learning rate is 0.001, and the iteration is performed 150 times. Save the model parameters with the highest validation set accuracy; The test set is input into the trained model with a batch size of 32 to output the classification results.

Images