A method for EEG emotion recognition based on multi-channel residual convolutional Transformer capsules

Abstract

This invention discloses an EEG emotion recognition method based on multi-channel residual convolutional Transformer capsules, belonging to the field of artificial intelligence technology. The method acquires the user's EEG signal, including baseline and stimulus-evoked signals. After baseline correction of the stimulus-evoked signals, multi-scale spatiotemporal feature maps are extracted using a multi-channel residual convolutional network. These maps are then globally correlated and encoded using a Transformer network to obtain a global contextual feature sequence. Finally, these sequences are converted into primary capsule vectors and aggregated into emotion capsule vectors using a dynamic routing protocol. The emotion recognition result is output based on the magnitude of the emotion capsule vectors. By achieving multi-scale spatiotemporal feature fusion through multi-channel residual convolution, capturing global contextual dependencies through Transformer, and improving classification robustness through the dynamic routing mechanism of the capsule network, the method effectively improves the accuracy and generalization ability of EEG emotion recognition.

Description

Technical Field

[0001] This application relates to the fields of artificial intelligence and brain-computer interface technology, and more specifically, to a brainwave emotion recognition method based on a multi-channel residual convolution Transformer capsule. Background Technology

[0002] Electroencephalography (EEG), as a non-invasive, high-temporal-resolution brain imaging technique, is widely used in emotion recognition research. Existing deep learning-based EEG emotion recognition methods mainly suffer from the following drawbacks: First, the feature extraction scale is singular. Existing models based on convolutional neural networks (CNNs) mostly use convolutional kernels of a single scale, which makes it difficult to simultaneously capture high-frequency (such as gamma wave) details and low-frequency (such as theta wave) rhythmic patterns related to emotion. This leads to the loss of key multi-frequency band information and forms a bottleneck in model performance.

[0003] Second, global contextual information is lacking. CNNs extract features through local receptive fields, making it difficult to effectively model the long-range dependencies that are widespread in EEG signals, such as the dynamic functional connections between electrodes distributed in different brain regions. This results in the model's insufficient ability to capture global emotional contextual information across channels and time.

[0004] Third, the spatial structural information is destroyed and the generalization ability is weak. The pooling operation in traditional CNNs destroys the structural information contained in the spatial distribution of EEG electrodes on the scalp. Moreover, existing methods often rely on predefined, static brain connectivity maps and cannot adaptively learn the dynamic functional connectivity patterns that vary from person to person, resulting in a significant decrease in the generalization performance of the model on cross-sample emotion recognition tasks.

[0005] To address the aforementioned issues, existing technologies urgently need improvement. Summary of the Invention

[0006] The purpose of this application is to provide an EEG emotion recognition method based on multi-channel residual convolution Transformer capsules, which has the advantages of enabling multi-scale feature fusion, global context modeling, and improving the generalization ability of the model.

[0007] In a first aspect, this application provides a brainwave emotion recognition method based on a multi-channel residual convolutional Transformer capsule, including: Acquire the user's electroencephalogram (EEG) signal, which includes a baseline signal and a stimulus-evoked signal; perform baseline correction on the stimulus-evoked signal to obtain a corrected signal sample; Feature extraction is performed on the corrected signal samples using a multi-channel residual convolutional network to obtain a multi-scale spatiotemporal feature map; wherein, the multi-channel residual convolutional network includes an initial convolutional layer, at least one residual block, and parallel multi-scale convolutional layers connected in sequence. The multi-scale spatiotemporal feature map is input into the Transformer network, and global association weights are calculated for each spatiotemporal location in the multi-scale spatiotemporal feature map to obtain a global context feature sequence. The global context feature sequence is converted into multiple primary capsule vectors, and the multiple primary capsule vectors are aggregated into at least one emotion capsule vector through a dynamic routing protocol; the magnitude of each emotion capsule vector is calculated, and the emotion recognition result is determined based on the magnitude.

[0008] Furthermore, the baseline correction of the stimulus-evoked signal includes: The baseline signal and the stimulus-evoked signal are segmented using a non-overlapping sliding window to obtain multiple baseline signal segments and multiple stimulus-evoked signal segments; For each stimulus-evoked signal segment, the average value of all baseline signal segments is calculated, and the average value is subtracted from the stimulus-evoked signal segment to obtain the corrected signal segment; All the corrected signal segments are spliced ​​together to form the corrected signal sample.

[0009] Furthermore, the feature extraction of the corrected signal samples using a multi-channel residual convolutional network includes: The initial convolutional layer performs one-dimensional temporal convolution on the corrected signal samples of each electrode channel to generate an initial temporal feature map. The initial temporal feature map is sequentially input into the at least one residual block, and each residual block outputs an intermediate feature map through two convolution operations and one skip connection; The intermediate feature maps are respectively input into each convolutional layer in the parallel multi-scale convolutional layer, wherein each convolutional layer has a different convolutional kernel size; The outputs of the parallel multi-scale convolutional layers are concatenated along the feature dimension to generate the multi-scale spatiotemporal feature map.

[0010] Furthermore, each residual block includes a first convolutional layer, a first batch normalization layer, a first activation function layer, a second convolutional layer, a second batch normalization layer, a second activation function layer, and a skip connection layer; The skip connection layer is used to add the input of the residual block to the output of the second batch normalization layer element by element, and input the addition result to the second activation function layer.

[0011] Furthermore, before inputting the multi-scale spatiotemporal feature map into the Transformer network, the method further includes performing a one-dimensional max pooling operation on the multi-scale spatiotemporal feature map to reduce the temporal dimension of the multi-scale spatiotemporal feature map.

[0012] Furthermore, the multi-scale spatiotemporal feature map is input into the Transformer network, and global association weights are calculated for each spatiotemporal location in the multi-scale spatiotemporal feature map, including: The multi-scale spatiotemporal feature map is converted into a two-dimensional sequence; Position encoding is added to the two-dimensional sequence to obtain a sequence with position information; The sequence with location information is linearly transformed into a query matrix, a key matrix, and a value matrix; A multi-head attention mechanism is adopted, in which the query matrix, key matrix and value matrix are split into multiple sub-matrix pairs, the attention weight of each sub-matrix pair is calculated in parallel, and the attention calculation results of all sub-matrix pairs are concatenated to obtain the attention weight matrix. Multiply the attention weight matrix by the value matrix to obtain the attention output; The attention output is input into the feedforward network to obtain the global context feature sequence.

[0013] Furthermore, the aggregation of the multiple primary capsule vectors into at least one emotion capsule vector via a dynamic routing protocol includes: For each primary capsule vector, a prediction vector pointing to each emotion capsule is generated by multiplying it with the transformation matrix; Initialize the coupling coefficient between each primary capsule vector and each emotion capsule; The following steps are iteratively executed until a preset number of iterations are reached: the predicted vector is weighted and summed according to the current coupling coefficient to obtain the input vector of each emotion capsule; the input vector is converted into an output vector using a compression function; and the coupling coefficient is updated according to the dot product of the output vector and the predicted vector. The output vector obtained from the last iteration is used as the emotion capsule vector.

[0014] Secondly, this application also proposes an EEG emotion recognition system based on a multi-channel residual convolutional Transformer capsule network, comprising: The acquisition and correction module is used to acquire the user's electroencephalogram (EEG) signal, which includes a baseline signal and a stimulus-evoked signal; and to perform baseline correction on the stimulus-evoked signal to obtain a corrected signal sample. The feature extraction module is used to extract features from the corrected signal samples through a multi-channel residual convolutional network to obtain a multi-scale spatiotemporal feature map; wherein, the multi-channel residual convolutional network includes an initial convolutional layer, at least one residual block, and parallel multi-scale convolutional layers connected in sequence. The global association calculation module is used to input the multi-scale spatiotemporal feature map into the Transformer network, calculate the global association weight for each spatiotemporal location in the multi-scale spatiotemporal feature map, and obtain the global context feature sequence. The recognition module is used to convert the global context feature sequence into multiple primary capsule vectors, and to aggregate the multiple primary capsule vectors into at least one emotion capsule vector through a dynamic routing protocol; calculate the magnitude of each emotion capsule vector, and determine the emotion recognition result based on the magnitude.

[0015] Thirdly, this application also proposes an electronic device comprising: one or more processors, and a memory for storing one or more computer programs; the computer programs are configured to be executed by the one or more processors, the programs including steps for performing the EEG emotion recognition method based on multi-channel residual convolution Transformer capsules as described in the first aspect.

[0016] Fourthly, this application also proposes a storage medium storing a computer program; the program is loaded and executed by a processor to implement the steps of the EEG emotion recognition method based on a multi-channel residual convolution Transformer capsule as described in the first aspect.

[0017] As shown above, the EEG emotion recognition method based on multi-channel residual convolutional Transformer capsules provided in this application solves the problem of insufficient feature decoupling caused by a single convolutional kernel by designing a multi-channel residual convolution module and explicitly extracting full-band spatiotemporal features using parallel multi-scale convolutional kernels. Residual connections promote gradient flow and feature reuse. By constructing a Transformer module to calculate global correlation weights for spatiotemporal sequences, it can simultaneously focus on long-range dependencies between time points and electrode channels, comprehensively modeling the global neural activity patterns of emotions. By fusing Transformer and capsule networks and combining a dynamic routing protocol to adaptively aggregate the most discriminative emotion capsules, overfitting is effectively suppressed, significantly improving the accuracy and robustness of the model on cross-sample emotion recognition tasks. It has the advantages of achieving multi-scale deep fusion, capturing global contextual dependencies, and improving model robustness and generalization ability. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating the steps of the EEG emotion recognition method based on a multi-channel residual convolution Transformer capsule disclosed in an embodiment of the present invention. Figure 2 This is a schematic diagram of the structure of the multi-channel residual convolutional Transformer capsule network disclosed in an embodiment of the present invention; Figure 3 This is a flowchart illustrating the steps of a method for aggregating multiple primary capsule vectors into at least one emotion capsule vector using a dynamic routing protocol, as disclosed in an embodiment of the present invention. Figure 4 This is a schematic diagram of the structure of the EEG emotion recognition system based on a multi-channel residual convolutional Transformer capsule network disclosed in an embodiment of the present invention. Detailed Implementation

[0020] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which these embodiments belong; the terminology used herein and in the specification of the application is for the purpose of describing particular embodiments only and is not intended to limit these embodiments; the terms "comprising" and "having," and any variations thereof, in the specification of these embodiments and the foregoing drawings, are intended to cover non-exclusive inclusion. The terms "first," "second," etc., in the specification of these embodiments and the foregoing drawings are used to distinguish different objects, not to describe a particular order.

[0021] The implementation details of the technical solution in this embodiment are described in detail below: This application proposes an EEG emotion recognition method based on a multi-channel residual convolutional Transformer capsule, such as... Figure 1 As shown, the method includes: S101, acquire the user's electroencephalogram (EEG) signal, the EEG signal including a baseline signal and a stimulus-evoked signal; perform baseline correction on the stimulus-evoked signal to obtain a corrected signal sample; Further, in S101, the baseline correction of the stimulus-evoked signal includes: segmenting the baseline signal and the stimulus-evoked signal using a non-overlapping sliding window to obtain multiple baseline signal segments and multiple stimulus-evoked signal segments; for each stimulus-evoked signal segment, calculating the average value of all baseline signal segments, and subtracting the average value from the stimulus-evoked signal segment to obtain a corrected signal segment; and splicing all corrected signal segments together to form the corrected signal sample.

[0022] Specifically, in this embodiment, the user's raw EEG signal is first acquired. This raw EEG signal consists of two parts: one part is the user's baseline signal in a resting state (also known as the resting-state signal), and the other part is the stimulus-evoked signal when the user is subjected to emotional stimuli (such as watching emotionally evoked videos or pictures). In order to eliminate the interference of individual resting-state EEG background activity on the emotion recognition results, baseline correction processing is required for the stimulus-evoked signal.

[0023] Specifically, the raw EEG signal includes a baseline signal (resting state) and a stimulus-evoked signal (evoked by emotional stimuli). To eliminate the influence of individual resting-state background EEG activity, baseline correction is first performed. The baseline signal... and stimulus-induced signals Each segment was obtained by using a 1-second non-overlapping sliding window. Then, baseline correction is performed on each stimulus-evoked signal segment using the following formula: (1) in, This represents the number of electrode channels. These represent the number of sampling points for the baseline and the experimental signal, respectively. All corrected segments. The samples were pieced together to form the final sample set. Each sample This represents a preprocessed and baseline-corrected EEG trial used as model input.

[0024] In this specific embodiment, the baseline correction operation is implemented through the following sub-steps. First, a non-overlapping sliding window of length 1 second is used to segment the baseline signal and the stimulus-evoked signal in the temporal domain. Each signal segment within the window contains 128 sampling points (corresponding to a time length of 1 second), thus obtaining multiple baseline signal segments and multiple stimulus-evoked signal segments. Then, for each stimulus-evoked signal segment, the arithmetic mean of all baseline signal segments is calculated to obtain an average baseline vector representing the individual's background EEG activity. Next, the average baseline vector is subtracted element-wise from the stimulus-evoked signal segment to obtain the corrected signal segment (see the baseline correction formula above). Finally, all corrected signal segments are concatenated sequentially according to their original temporal order to form a complete corrected signal sample that can be used as input to the model. Each sample in this sample set can be directly input into the subsequent multi-channel residual convolutional network for feature extraction.

[0025] S102, the corrected signal samples are subjected to feature extraction through a multi-channel residual convolutional network to obtain a multi-scale spatiotemporal feature map; wherein, the multi-channel residual convolutional network includes an initial convolutional layer, at least one residual block, and parallel multi-scale convolutional layers connected in sequence.

[0026] Further, in S102, the feature extraction of the corrected signal samples through a multi-channel residual convolutional network includes: performing one-dimensional temporal convolution on the corrected signal samples of each electrode channel through the initial convolutional layer to generate an initial temporal feature map; sequentially inputting the initial temporal feature map into the at least one residual block, with each residual block outputting an intermediate feature map through two convolution operations and one skip connection; inputting the intermediate feature map into each convolutional layer in the parallel multi-scale convolutional layer, wherein each convolutional layer has a different convolutional kernel size; and concatenating the outputs of the parallel multi-scale convolutional layers along the feature dimension to generate the multi-scale spatiotemporal feature map.

[0027] Furthermore, each residual block includes a first convolutional layer, a first batch normalization layer, a first activation function layer, a second convolutional layer, a second batch normalization layer, a second activation function layer, and a skip connection layer; the skip connection layer is used to add the input of the residual block to the output of the second batch normalization layer element by element, and input the addition result to the second activation function layer.

[0028] Specifically, in this embodiment, a multi-channel residual convolutional network is used to deeply extract multi-scale spatiotemporal features from the corrected signal samples.

[0029] like Figure 2 The diagram shown is a schematic representation of the structure of the multi-channel residual convolutional Transformer capsule network provided in this embodiment. Figure 2As shown, the original multi-channel EEG temporal signal is first input into the residual convolution module, and deep features are extracted through three layers of convolution (1×1→3×3→1×1, with 64→64→256 channels) combined with skip connections and ReLU activation. Then, it enters the multi-scale convolution module, and after multi-scale convolution and ReLU activation, information from different receptive fields is fused through multi-scale connections. Next, the multi-scale features are fed into the Transformer attention module, which uses a self-attention mechanism to generate weights and multiplies them element-wise with the input features to optimize feature representation. Finally, it is input into the emotion capsule module, which sequentially performs capsule reshaping (generating a 48-dimensional primary capsule), dynamic routing protocol aggregation, and L2 norm normalization, and outputs capsule vectors for each emotion category to complete classification and recognition.

[0030] like Figure 2 As shown, the network module consists of an initial temporal convolutional layer, multiple stacked residual blocks, and parallel multi-scale convolutional layers. The input is a baseline-corrected signal sample. Where N is the number of electrode channels and T is the number of time sampling points. First, a one-dimensional temporal convolution is performed independently on each electrode channel to capture the local temporal pattern of each channel: (2) in, It is a convolution kernel. This represents the convolution operation. It is the ReLU activation function. It is a bias term. The output feature map is the number of feature dimensions. Next, Input to by In a deep network consisting of stacked residual blocks, each residual block mitigates the vanishing gradient problem through skip connections. Finally, parallel multi-scale convolutional layers are used after the residual network to capture contextual information at different time scales, and the multiple outputs are concatenated to obtain a feature map rich in multi-scale spatiotemporal information.

[0031] For the One residual block: (3) (4) Where BN represents batch normalization. It is an optional 1x1 convolution used for quick connection projection when the input and output dimensions do not match. Each residual block is processed by two convolutions, two batch normalizations, and ReLU activations, plus skip connections, which effectively alleviates the gradient vanishing problem in deep networks and ensures stable training and feature reuse.

[0032] Following the stacked residual blocks, three parallel one-dimensional convolutional layers with different kernel sizes (e.g., 3, 5, and 7) are used to capture contextual information at different time scales (corresponding to oscillation patterns of different frequency rhythms). The outputs of the three parallel paths are then concatenated along the feature dimension. (5) Subsequently, the fused multi-scale features Perform one-dimensional max pooling to reduce the time dimension, enhance the translation invariance of features, and reduce subsequent computational cost: (6) (7) Where P is the pooling window size.

[0033] Ultimately, the module outputs feature maps rich in multi-scale spatiotemporal information. . The final output feature dimension. The time dimension is the result of multi-scale convolution.

[0034] In this embodiment, the internal structure of each residual block is implemented strictly according to the above formula. Specifically, the input... First, it goes through the first convolutional layer. The intermediate features are obtained by passing the first batch normalization layer and then the first activation function layer (ReLU). .then, After the second convolutional layer And the second batch normalization layer. Simultaneously, the skip connection layer. The original input of the residual block Dimension matching is performed (via 1x1 convolutional projection if necessary), and then the result of this skip connection is element-wise added to the output of the second batch normalization layer. Finally, the sum is fed into the second activation function layer (ReLU) to obtain the output of the residual block. This structure ensures that gradients can be directly transferred from deep layers to shallow layers, effectively supporting the training of deep networks.

[0035] S103, the multi-scale spatiotemporal feature map is input into the Transformer network, and global association weights are calculated for each spatiotemporal location in the multi-scale spatiotemporal feature map to obtain a global context feature sequence.

[0036] In this embodiment, the Transformer network is used to model global dependencies in multi-scale spatiotemporal feature maps. For example... Figure 2As shown in the middle section, this module treats each spatiotemporal location in the feature map as an element in the sequence and calculates the association weights between all locations through a self-attention mechanism, thereby capturing global contextual information across channels and across time.

[0037] Furthermore, before inputting the multi-scale spatiotemporal feature map into the Transformer network in step S103, the method further includes performing a one-dimensional max pooling operation on the multi-scale spatiotemporal feature map to reduce the temporal dimension of the multi-scale spatiotemporal feature map.

[0038] Specifically, in S102, the multi-scale residual convolution module outputs feature maps. Next, a one-dimensional max pooling operation is first performed to reduce the time dimension, enhance the translation invariance of the features, and reduce the computational cost of the subsequent Transformer module. The pooling operation is calculated as shown in formulas (6)-(7). The pooled feature map is denoted as... The feature map is then fed into the Transformer network.

[0039] Further, in S103, the multi-scale spatiotemporal feature map is input into the Transformer network, and global association weights are calculated for each spatiotemporal location in the multi-scale spatiotemporal feature map. This includes: converting the multi-scale spatiotemporal feature map into a two-dimensional sequence; adding position encoding to the two-dimensional sequence to obtain a sequence with position information; linearly transforming the sequence with position information into a query matrix, a key matrix, and a value matrix; using a multi-head attention mechanism, splitting the query matrix, key matrix, and value matrix into multiple sub-matrix pairs, calculating the attention weights of each sub-matrix pair in parallel, and then concatenating the attention calculation results of all sub-matrix pairs to obtain an attention weight matrix; multiplying the attention weight matrix with the value matrix to obtain an attention output; and inputting the attention output into a feedforward network to obtain the global context feature sequence.

[0040] Specifically, in this embodiment, the feature map is first... (Here, C is equivalent to N, the number of electrodes) Remodeled into a sequence ,in It is the sequence length. To preserve spatiotemporal location information, the sequence... Add learnable positional encoding: (8) Then, The query (Q), key (K), and value (V) matrices are obtained through linear transformation, and multi-head self-attention (MHSA) is calculated: (9) Multi-head self-attention (MHSA) is calculated. Each attention head is calculated as follows: (10) in, The function representation uses a multi-head self-attention mechanism to model the relationships between all feature units. The activation function is the specific operation; d k The dimension of the key vector; is the projected value matrix; K is the key matrix; L represents the number of elements in the input sequence; The multi-head attention result is obtained by concatenating the outputs of all heads and performing an output transformation: (11) W O To output the projection matrix, the concatenated result is linearly transformed back to the input. The same dimensionality; this process enables the model to simultaneously focus on the relationship between any two locations in the sequence (i.e., any two electrode-time point combinations), thereby capturing the global context across time and space. Next, the output of the multi-head attention is fed into a feedforward network (FFN, typically two fully connected layers and an activation function) to obtain an enhanced global feature sequence. In practice, residual connections and layer normalization operations are added after the multi-head attention and feedforward networks to ensure training stability. The final output... This is the global context feature sequence, which is used by the subsequent capsule network classification module.

[0041] S104, the global context feature sequence is converted into multiple primary capsule vectors, and the multiple primary capsule vectors are aggregated into at least one emotion capsule vector through a dynamic routing protocol; the magnitude of each emotion capsule vector is calculated, and the emotion recognition result is determined based on the magnitude.

[0042] Furthermore, in S104, the aggregation of the multiple primary capsule vectors into at least one emotion capsule vector via a dynamic routing protocol, such as... Figure 3 ,include: S4001, for each primary capsule vector, a prediction vector pointing to each sentiment capsule is generated by multiplying it with the transformation matrix.

[0043] Specifically, in this embodiment, the capsule network classification module first reshapes and segments the global context feature sequence output by the Transformer into a set of primary capsules. Each capsule represents a detected local entity and its attributes. Then, for each primary capsule i and each emotion capsule j, a transformation matrix is ​​used... Map the primary capsule vector to the prediction vector: The prediction vector represents the contribution of the entity detected by primary capsule i to the higher-level emotion category represented by emotion capsule j.

[0044] S4002, initialize the coupling coefficient between each primary capsule vector and each emotion capsule.

[0045] Before the dynamic routing iteration begins, initialize the log-prior probabilities between all primary capsules i and sentiment capsule j. The coupling coefficient is 0. Through the After performing softmax normalization, we get: (12) S4003, iteratively execute the following steps until a preset number of times are reached: perform a weighted summation of the prediction vector based on the current coupling coefficient to obtain the input vector of each emotion capsule; convert the input vector into an output vector using a compression function; update the coupling coefficient based on the dot product of the output vector and the prediction vector.

[0046] Specifically, each iteration includes the following three sub-steps: First, based on the current coupling coefficient The prediction vectors of all primary capsules are weighted and summed to obtain the input vector for each emotion capsule j. : (13) Second, the input vector is compressed using a squash function. Convert to the output vector of the emotion capsule This function compresses the magnitude of a vector to between 0 and 1, making its magnitude represent the probability of an entity's existence. (14) Third, based on the output vector With prediction vector Update the log-prior probability of the dot product : (15) This update rule ensures that: if the predicted vector With the current emotion capsule output If the directions are consistent (the dot product is larger), then increase. This will improve the coupling coefficient in the next iteration. Conversely, the output decreases. Through multiple iterations, the output of the lower-level capsule is adaptively routed to the higher-level sentiment capsule that best matches it.

[0047] S4004, use the output vector obtained from the last iteration as the emotion capsule vector.

[0048] Specifically, in this embodiment, after the iteration reaches a preset number (e.g., 3 times), the update stops, and the output vector obtained from the last iteration is... This serves as the final emotion capsule vector. Each emotion capsule... Length of the module This represents the probability that the input sample belongs to this emotion category (e.g., high arousal, low valence). During training, Margin Loss is used as the loss function for optimization. (16) in If and only if the true label of the sample is , which is a hyperparameter. In testing or practical applications, the category corresponding to the emotion capsule with the largest modulus length is directly taken as the emotion recognition result output.

[0049] Based on the complete description of the above embodiments, those skilled in the art will understand that the EEG emotion recognition method based on multi-channel residual convolution Transformer capsules proposed in this invention achieves end-to-end learning of user EEG signals from local multi-scale spatiotemporal features to global dependencies and then to structured emotion representations by sequentially performing four stages: baseline correction, multi-channel residual convolution feature extraction, Transformer global context encoding, and capsule network dynamic routing classification. This method preserves the spatial distribution information of electrode channels in EEG signals while adaptively capturing long-range dependencies across time and channels. Simultaneously, the dynamic routing mechanism effectively suppresses overfitting, improving the accuracy and generalization ability of emotion recognition.

[0050] Secondly, this embodiment also proposes an EEG emotion recognition system based on a multi-channel residual convolutional Transformer capsule network, such as... Figure 4 As shown, it includes: The acquisition and correction module 401 is used to acquire the user's electroencephalogram (EEG) signal, which includes a baseline signal and a stimulus-evoked signal; and to perform baseline correction on the stimulus-evoked signal to obtain a corrected signal sample. The feature extraction module 402 is used to extract features from the corrected signal samples through a multi-channel residual convolutional network to obtain a multi-scale spatiotemporal feature map; wherein, the multi-channel residual convolutional network includes an initial convolutional layer, at least one residual block, and parallel multi-scale convolutional layers connected in sequence. The global association calculation module 403 is used to input the multi-scale spatiotemporal feature map into the Transformer network, perform global association weight calculation on each spatiotemporal position in the multi-scale spatiotemporal feature map, and obtain a global context feature sequence. The recognition module 404 is used to convert the global context feature sequence into multiple primary capsule vectors, and to aggregate the multiple primary capsule vectors into at least one emotion capsule vector through a dynamic routing protocol; to calculate the magnitude of each emotion capsule vector, and to determine the emotion recognition result based on the magnitude.

[0051] This system can be used to perform the EEG emotion recognition method based on multi-channel residual convolution Transformer capsules as described in the first aspect, which will not be elaborated here.

[0052] Thirdly, this embodiment also proposes an electronic device comprising: one or more processors, and a memory for storing one or more computer programs; the computer programs are configured to be executed by the one or more processors, and the programs include steps for performing the EEG emotion recognition method based on multi-channel residual convolution Transformer capsules as described in the first aspect. This electronic device can be a general-purpose computer, an embedded device, a portable EEG processing terminal, or a cloud server, etc. By running the above program, the electronic device can identify the user's emotional state in real time and accurately, and is suitable for scenarios such as affective computing, human-computer interaction, and psychological monitoring.

[0053] Fourthly, this embodiment also proposes a storage medium storing a computer program; the program is loaded and executed by a processor to implement the steps of the EEG emotion recognition method based on a multi-channel residual convolution Transformer capsule as described in the first aspect.

[0054] The storage medium can be a volatile or non-volatile computer-readable storage medium, such as a read-only memory (ROM), random access memory (RAM), hard disk, solid-state drive (SSD), optical disc (CD-ROM), USB flash drive, SD card, etc. When the computer program in the storage medium is loaded into the processor and executed, it enables the user to obtain efficient and robust EEG emotion recognition capabilities. This storage medium can be sold independently or provided in conjunction with EEG acquisition equipment for installing or updating related processing software. The specific implementation steps during program execution are the same as those described in the above electronic device embodiments, and will not be repeated here.

[0055] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A brainwave emotion recognition method based on a multi-channel residual convolutional Transformer capsule, characterized in that, include: Acquire the user's electroencephalogram (EEG) signal, which includes a baseline signal and a stimulus-evoked signal; perform baseline correction on the stimulus-evoked signal to obtain a corrected signal sample; Feature extraction is performed on the corrected signal samples using a multi-channel residual convolutional network to obtain a multi-scale spatiotemporal feature map; wherein, the multi-channel residual convolutional network includes an initial convolutional layer, at least one residual block, and parallel multi-scale convolutional layers connected in sequence. The multi-scale spatiotemporal feature map is input into the Transformer network, and global association weights are calculated for each spatiotemporal location in the multi-scale spatiotemporal feature map to obtain a global context feature sequence. The global context feature sequence is converted into multiple primary capsule vectors, and the multiple primary capsule vectors are aggregated into at least one emotion capsule vector through a dynamic routing protocol; the magnitude of each emotion capsule vector is calculated, and the emotion recognition result is determined based on the magnitude.

2. The EEG emotion recognition method based on multi-channel residual convolutional Transformer capsules according to claim 1, characterized in that, The baseline correction of the stimulus-evoked signal includes: The baseline signal and the stimulus-evoked signal are segmented using a non-overlapping sliding window to obtain multiple baseline signal segments and multiple stimulus-evoked signal segments; For each stimulus-evoked signal segment, the average value of all baseline signal segments is calculated, and the average value is subtracted from the stimulus-evoked signal segment to obtain the corrected signal segment; All the corrected signal segments are spliced ​​together to form the corrected signal sample.

3. The EEG emotion recognition method based on multi-channel residual convolutional Transformer capsules according to claim 1, characterized in that, The step of extracting features from the corrected signal samples using a multi-channel residual convolutional network includes: The initial convolutional layer performs one-dimensional temporal convolution on the corrected signal samples of each electrode channel to generate an initial temporal feature map. The initial temporal feature map is sequentially input into the at least one residual block, and each residual block outputs an intermediate feature map through two convolution operations and one skip connection; The intermediate feature maps are respectively input into each convolutional layer in the parallel multi-scale convolutional layer, wherein each convolutional layer has a different convolutional kernel size; The outputs of the parallel multi-scale convolutional layers are concatenated along the feature dimension to generate the multi-scale spatiotemporal feature map.

4. The EEG emotion recognition method based on multi-channel residual convolution Transformer capsules according to claim 3, characterized in that, Each residual block includes a first convolutional layer, a first batch normalization layer, a first activation function layer, a second convolutional layer, a second batch normalization layer, a second activation function layer, and a skip connection layer; The skip connection layer is used to add the input of the residual block to the output of the second batch normalization layer element by element, and input the addition result to the second activation function layer.

5. The EEG emotion recognition method based on multi-channel residual convolutional Transformer capsules according to claim 1, characterized in that, Before inputting the multi-scale spatiotemporal feature map into the Transformer network, the method further includes performing a one-dimensional max pooling operation on the multi-scale spatiotemporal feature map to reduce the temporal dimension of the multi-scale spatiotemporal feature map.

6. The EEG emotion recognition method based on multi-channel residual convolutional Transformer capsules according to claim 1, characterized in that, The multi-scale spatiotemporal feature map is input into the Transformer network, and global association weights are calculated for each spatiotemporal location in the multi-scale spatiotemporal feature map, including: The multi-scale spatiotemporal feature map is converted into a two-dimensional sequence; Position encoding is added to the two-dimensional sequence to obtain a sequence with position information; The sequence with location information is linearly transformed into a query matrix, a key matrix, and a value matrix; A multi-head attention mechanism is adopted, in which the query matrix, key matrix and value matrix are split into multiple sub-matrix pairs, the attention weight of each sub-matrix pair is calculated in parallel, and the attention calculation results of all sub-matrix pairs are concatenated to obtain the attention weight matrix. Multiplying the attention weight matrix by the value matrix yields the attention output; The attention output is input into the feedforward network to obtain the global context feature sequence.

7. The EEG emotion recognition method based on multi-channel residual convolutional Transformer capsules according to claim 1, characterized in that, The process of aggregating the multiple primary capsule vectors into at least one sentiment capsule vector using a dynamic routing protocol includes: For each primary capsule vector, a prediction vector pointing to each emotion capsule is generated by multiplying it with the transformation matrix; Initialize the coupling coefficient between each primary capsule vector and each emotion capsule; The following steps are iteratively executed until a preset number of iterations are reached: the predicted vector is weighted and summed according to the current coupling coefficient to obtain the input vector of each emotion capsule; the input vector is converted into an output vector using a compression function; and the coupling coefficient is updated according to the dot product of the output vector and the predicted vector. The output vector obtained from the last iteration is used as the emotion capsule vector.

8. A brainwave emotion recognition system based on a multi-channel residual convolutional Transformer capsule network, characterized in that, include: The acquisition and correction module is used to acquire the user's electroencephalogram (EEG) signal, which includes a baseline signal and a stimulus-evoked signal; and to perform baseline correction on the stimulus-evoked signal to obtain a corrected signal sample. The feature extraction module is used to extract features from the corrected signal samples through a multi-channel residual convolutional network to obtain a multi-scale spatiotemporal feature map; wherein, the multi-channel residual convolutional network includes an initial convolutional layer, at least one residual block, and parallel multi-scale convolutional layers connected in sequence. The global association calculation module is used to input the multi-scale spatiotemporal feature map into the Transformer network, calculate the global association weight for each spatiotemporal location in the multi-scale spatiotemporal feature map, and obtain the global context feature sequence. The recognition module is used to convert the global context feature sequence into multiple primary capsule vectors, and to aggregate the multiple primary capsule vectors into at least one emotion capsule vector through a dynamic routing protocol; calculate the magnitude of each emotion capsule vector, and determine the emotion recognition result based on the magnitude.

9. An electronic device, the electronic device comprising: One or more processors, a memory for storing one or more computer programs; characterized in that the computer programs are configured to be executed by the one or more processors, the programs including steps for performing the EEG emotion recognition method based on a multi-channel residual convolution Transformer capsule as described in any one of claims 1-7.

10. A storage medium storing a computer program; characterized in that, The program is loaded and executed by a processor to implement the steps of the EEG emotion recognition method based on a multi-channel residual convolution Transformer capsule as described in any one of claims 1-7.

Images