A Multi-Subject Motion Imagination Recognition Method Using Shallow Mirror Deformer Networks

Abstract

This invention discloses a multi-agent motion imagery recognition method using a shallow mirror deformer network. The specific implementation steps are as follows: Step 1: Extracting EEG signal data; Step 2: Selecting C required EEG channels and filtering them using a 3rd-order bandpass filter (0-38 Hz); Step 3: Standardizing the EEG signal by performing an exponentially weighted average normalization operation on each channel of the filtered EEG data; Step 4: Expanding the EEG signal by generating mirror EEG signals; Step 5: Inputting the mirror EEG signals generated in Step 4, along with the original EEG signals, into the ST model for training; Step 6: Using ensemble learning in the prediction phase to improve model accuracy. This method can improve the network model's generalization ability among agents, thereby improving the performance of multi-agent brain-computer interfaces.

Description

Technical Field

[0001] This invention belongs to the field of brain-computer interface technology, specifically relating to a multi-subject motion image recognition method using a shallow mirror deformer network. Background Technology

[0002] Brain-computer interfaces (BCIs) establish communication or control channels between the human brain and external devices. BCI technology has been widely applied in fields such as medicine, neurobiology, and psychology, and holds broad commercial potential in virtual reality, educational technology, and smart homes. Motion imagery EEG (MI-EEG) is an endogenous, spontaneous brain signal, characterized by its simplicity, flexibility, non-invasiveness, and low environmental requirements. Therefore, motion imagery-based BCI is an important and widely used type of BCI. MI BCI systems collect EEG signals from subjects performing specific motor imagery (MI), identify the content of the motor imagery based on the EEG signals, and convert the identification results into control commands for peripheral devices.

[0003] Because the location and duration of Event-Related Synchronization / Desynchronization (ERS / ERD) in motor imagery vary, using fixed EEG signal segments for feature extraction leads to poor classification performance. Therefore, a model capable of detecting and utilizing effective signal segments from the entire EEG signal trial may produce good performance in multi-subject motor imagery BCI. However, the receptive field of a network node in a convolutional neural network (CNN) is typically small, preventing CNNs from detecting effective signal segments from the entire EEG signal trial. For example, consider the popular "Shallow ConvNet" network in motor imagery. Assuming an EEG signal input size of 1125×3 (samples × channels), after the first temporal convolutional layer, the receptive field of a network node is 25×1. After the second spatial filtering layer, the receptive field of a network node is 25×3, representing only 2.2% of the entire EEG trial. Therefore, the receptive field of a network node in a convolutional neural network is only a small fraction of the entire EEG signal trial.

[0004] The first method to expand the receptive field is to add more convolutional layers. However, this increases complexity and the number of parameters, and is not suitable for recognizing motor imagery EEG signals due to the limited number of samples. Currently popular convolutional neural network models like Shallow ConvNet, Deep ConvNet, and EEGNet have 1, 4, and 2 convolutional layers respectively, resulting in a relatively small receptive field for the last layer. The second method is to increase the size of the convolutional kernel. However, recent research shows that smaller kernels perform better in feature learning, while larger kernels are rarely used. Therefore, a small receptive field is a difficult problem to solve in multi-agent motor imagery recognition models based on convolutional neural networks. Summary of the Invention

[0005] The purpose of this invention is to provide a multi-subject motion image recognition method using a shallow mirror deformer network, which solves the problem of low recognition accuracy caused by the small temporal receptive field and inability to fully utilize EEG temporal information in existing technologies. This method can improve the generalization ability of the network model among subjects, thereby improving the performance of multi-subject brain-computer interfaces.

[0006] The technical solution adopted in this invention is a multi-subject motion imagination recognition method using shallow mirror deformer networks, and the specific implementation steps are as follows:

[0007] Step 1: Extract the EEG signal data;

[0008] Step 2: Select the required C EEG channels and filter them using a 3rd-order bandpass filter of 0-38Hz;

[0009] Step 3: Perform exponential weighted average standardization on each channel of the filtered EEG data to standardize the EEG signal;

[0010] Step 4: Expand the brainwave signal by creating a mirror brainwave signal;

[0011] Step 5: Input the mirrored EEG signal generated in Step 4 along with the original EEG signal into the ST model for training;

[0012] Step 6: Use ensemble learning to improve model accuracy during the prediction phase.

[0013] The invention is further characterized in that,

[0014] In step 1, the capture range is from 0.5 seconds before the start of the motion imagery task to 4 seconds after the start of the motion imagery task.

[0015] In step 2, three EEG channels, C3, C4, and Cz, are used.

[0016] The specific process of the exponential weighted average standardization operation in step 3 is as follows:

[0017] First, calculate the exponential moving average m. t :

[0018] m t =fmean(x t )+(1-f)m t-1 (1)

[0019] In equation (1), x t It is the value of the t-th sampling point of the original EEG signal, m t The mean(x) is the moving average of the EEG signal at the t-th sampling point, where f is a pre-set update factor. t) is an EEG signal x t Calculate the average across multiple channels;

[0020] Then calculate the exponential moving variance v t :

[0021] v t =f(m t -x t ) 2 +(1-f)v t-1 (2)

[0022] Finally, calculate the EEG signal x after standardization by exponentially weighted moving average. t ':

[0023]

[0024] In equation (3), For the maximum value function, take the maximum value of the two parameters, ε = 10. -6 .

[0025] In step 4, the EEG signals standardized in step 3 are used to construct a mirror EEG by exchanging the EEG data corresponding to channels C3 and C4, and then the EEG signal labels for left and right hand motor imagery are exchanged.

[0026] The ST model in step 5 includes a CNN-based feature extraction layer, a deformer-based encoder layer, and a motor imagery EEG classification layer. The deformer-based encoder layer consists of a location embedding layer, a multi-head self-attention layer, and a feedforward layer. The motor imagery EEG classification layer consists of a fully connected layer and a softmax function, with the number of output nodes equal to the number of motor imagery categories.

[0027] The specific steps for step 5 are as follows:

[0028] Step 5.1: Define the set of mirrored EEG signals and original EEG signals generated in Step 4 as X∈R. K×C Where K is the number of sampling points and C is the number of channels in the EEG; the batch size for model training is B, so the dimension of the input data is B×1×K×C;

[0029] Therefore, the calculation of the feature extraction layer based on CNN is shown in Equation (4):

[0030] F = BN(C s (C t (X))) (4)

[0031] In equation (4), X represents the set of mirrored EEG signals and original EEG signals generated in step 4, and C t This represents a 1×25 time-domain convolution used to simulate the bandpass filter in the FBCSP algorithm; Cs represents a spatial filter of size C×1, used to fuse information from all EEG channels to simulate a CSP spatial filter; BN represents batch normalization of the extracted features; and F represents the features extracted by the CNN-based feature extraction layer.

[0032] Step 5.2: The EEG feature information F extracted by the feature extraction layer will be input into the deformer-based encoder layer to obtain global temporal information, as follows:

[0033] The position embedding layer in the encoder layer of the deformer is represented as shown in Equation (5):

[0034]

[0035] In equation (5), F represents the features extracted by the CNN-based feature extraction layer, and Pos is a set of standard learnable position embedding parameters. Features that incorporate location information;

[0036] After processing by the location embedding layer, the features embedded with location information are... The input is fed into a multi-head self-attention layer with a residual network structure; in the residual network, layer normalization is performed first; then, the multi-head self-attention layer maps the query vector, key vector, and value vector to the output in the form of a weighted sum of value vectors. The calculation method of query vector, key vector, and value vector is shown in Equation (6):

[0037]

[0038]

[0039]

[0040] In equation (6), To embed features with location information, W i q W i k and W i v These are the projection matrices of the query vector, key vector, and value vector of the i-th head, respectively. LN denotes layer normalization, and Q... i K i V i Let $i$ represent the query vector, key vector, and value vector of the $i$-th head, respectively. Then, the similarity between the query vector and the key vector is calculated by dot product, and after scaling, the attention weight of the corresponding value vector is obtained by applying the softmax function. Therefore, the output is the weighted sum of all value vectors, as shown in equation (7).

[0041]

[0042] In equation (7), O i For the output of the i-th head, This represents the transpose of the key vector of the i-th head, Q. i Let V be the query vector for the i-th head. i Let d be the value vector corresponding to the i-th head. k Used to scale the size of the key vector;

[0043] Then, connect all the outputs together, as shown in equation (8):

[0044]

[0045] In equation (8), O1, O2, O N For the output of each head, Concat is the dimension concatenation function. For features that embed location information, O represents the output of N heads;

[0046] Subsequently, the output O of the multi-head self-attention layer is input into the feedforward layer for calculation; the calculation of the feedforward layer is shown in equation (9):

[0047] M = PF(LN(O)) + O (9)

[0048] In equation (9), O is the output of the multi-head self-attention layer, LN represents layer normalization, PF represents the feedforward layer, and M is the output of the entire encoder layer.

[0049] Step 5.3: Use the motor imagery EEG classification layer to predict the motor imagery task category, as shown in Equation (10):

[0050] P = FC(Log(Pool(M⊙M))) (10)

[0051] In Equation (10), ⊙ represents dot product, M represents the output of the encoder layer based on deformer, Pool represents the pooling layer used to reduce the feature size, Log represents the logarithmic activation function, FC represents the fully connected layer, and P represents the prediction probability of the shallow deformer ST model for motor imagery EEG.

[0052] The specific steps for step 6 are as follows:

[0053] The ST model trained in step 5 is copied as a mirror ST model. The original EEG signal is input into the original ST model for model prediction, and the mirror EEG signal is input into the mirror ST model for model prediction. Then, the prediction results of the two models are probabilistically integrated.

[0054] The probability integration method used is shown in equation (11):

[0055]

[0056] In equation (11), P l o This represents the probability of the raw EEG signal predicting the left-hand motor imagery task. P represents the predictive probability of mirror EEG signals for the right-hand motor imagery task. l P r These are the model's final predicted probabilities for the left-hand motor imagery task and the right-hand motor imagery task, respectively; the final predicted label of the raw EEG signal is the category with the highest final predicted probability.

[0057] The beneficial effects of this invention are:

[0058] This invention utilizes a multi-head self-attention layer in a motor imagery network model to adaptively capture EEG signal segments corresponding to different subjects during motor imagery by fully leveraging global temporal information in the shallow layers of the network. This model improves the network's generalization ability among subjects, and the use of a mirror structure further enhances the classification performance of MI-EEG. Attached Figure Description

[0059] Figure 1 This is a schematic diagram of the shallow deformer ST model in the method of the present invention;

[0060] Figure 2 This is a schematic diagram of the construction method of mirrored EEG in the method of the present invention;

[0061] Figure 3 This is a schematic diagram of the multi-subject motion imagination recognition model (SMT) of the shallow mirror deformer network in the method of this invention. Detailed Implementation

[0062] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0063] This invention provides a method for recognizing multi-subject motion imagery using shallow mirror deformer networks, such as... Figure 1-3 As shown, the specific implementation steps are as follows:

[0064] Step 1: Extract the EEG signal data (the extraction range is from 0.5 seconds before the start of the motor imagery task to 4 seconds after the start of the motor imagery task).

[0065] Step 2: Select the required C EEG channels (this patent uses three EEG channels: C3, C4, and Cz), and filter them using a 3rd-order bandpass filter of 0-38Hz;

[0066] Step 3: Perform exponential weighted average standardization on each channel of the filtered EEG data to standardize the EEG signal;

[0067] The specific process of the exponential weighted average standardization operation in step 3 is as follows:

[0068] First, calculate the exponential moving average m. t :

[0069] m t =fmean(x t )+(1-f)m t-1 (1)

[0070] In equation (1), x t It is the value of the t-th sampling point of the original EEG signal, m t is the moving average of the EEG signal at the t-th sampling point, f is a pre-set update factor (an empirical value of 0.001 is used in this patent), mean(x t ) is an EEG signal x t Calculate the average across multiple channels;

[0071] Then calculate the exponential moving variance v t :

[0072] v t =f(m t -x t ) 2 +(1-f)v t-1 (2)

[0073] Finally, calculate the EEG signal x after standardization by exponentially weighted moving average. t ':

[0074]

[0075] In equation (3), For the maximum value function, take the maximum value of the two parameters, ε = 10. -6 .

[0076] Step 4: Amplify the brainwave signal by creating a mirror image of the brainwave signal. The specific construction method is as follows... Figure 2 As shown, for the EEG signals standardized in step 3, the EEG data corresponding to channels C3 and C4 are exchanged to construct a mirror EEG, and then the EEG signal labels for left and right hand motor imagery are exchanged.

[0077] Step 5: Input the mirrored EEG signal generated in Step 4, along with the original EEG signal, into the ST model for training.

[0078] The ST model in step 5 includes a CNN-based feature extraction layer, a deformer-based encoder layer, and a motor imagery EEG classification layer. The deformer-based encoder layer consists of a location embedding layer, a multi-head self-attention layer, and a feedforward layer. The motor imagery EEG classification layer consists of a fully connected layer and a softmax function, with the number of output nodes equal to the number of motor imagery categories. The EEG signal is input to the CNN-based feature extraction layer, and the features extracted by the CNN-based feature extraction layer are input to the deformer-based multi-head self-attention layer. The multi-head self-attention layer captures global temporal information and then inputs the results into the motor imagery EEG classification layer to predict the motor imagery task category.

[0079] Step 5.1: Define the set of mirrored EEG signals and original EEG signals generated in Step 4 as X∈R. K×C Where K is the number of sampling points and C is the number of EEG channels. The batch size for model training is B, therefore the dimension of the input data is B×1×K×C.

[0080] Therefore, the calculation of the feature extraction layer based on CNN is shown in Equation (4):

[0081] F = BN(C s (C t (X))) (4)

[0082] In equation (4), X represents the set of mirrored EEG signals and original EEG signals generated in step 4, and C t This represents a 1×25 time-domain convolution used to simulate the bandpass filter in the FBCSP algorithm. (C) s Let C represent a spatial filter of size C×1, used to fuse information from all EEG channels to simulate a CSP spatial filter. Finally, BN denotes batch normalization of the extracted features, and F represents the features extracted by the CNN-based feature extraction layer.

[0083] Step 5.2: The EEG feature information F extracted by the feature extraction layer will be input into the deformer-based encoder layer to obtain global temporal information, as follows:

[0084] The position embedding layer in the encoder layer of the deformer is represented as shown in Equation (5):

[0085]

[0086] In equation (5), F represents the features extracted by the CNN-based feature extraction layer, and Pos is a set of standard learnable position embedding parameters. Features that incorporate location information.

[0087] After processing by the location embedding layer, the features embedded with location information are... The input is fed into a multi-head self-attention layer with a residual network structure. In the residual network, layer normalization is performed first. Then, the multi-head self-attention layer maps the query vector, key vector, and value vector to the output as a weighted sum of the value vectors. The query vector, key vector, and value vector are calculated as shown in Equation (6):

[0088]

[0089]

[0090]

[0091] In equation (6), To embed features with location information, W i q W i k and W i v These are the projection matrices of the query vector, key vector, and value vector of the i-th head, respectively. LN denotes layer normalization, and Q... i K i V i Let $\mathbf{i}$ represent the query vector, key vector, and value vector of the $i$-th head, respectively. Then, the similarity between the query vector and the key vector is calculated using a dot product, scaled, and then the attention weight of the corresponding value vector is obtained using the softmax function. Therefore, the output is a weighted sum of all value vectors, as shown in equation (7):

[0092]

[0093] In equation (7), O i For the output of the i-th head, K i T This represents the transpose of the key vector of the i-th head, Q. i Let V be the query vector for the i-th head. i Let d be the value vector corresponding to the i-th head. k The size of the key vector is used for scaling. Dividing by the dimension of the key vector is a scaling method that can alleviate the vanishing gradient problem. The attention function is executed in parallel multiple times, and each computational unit of the attention function is called a head. The number of attention heads is denoted by N (this invention uses 8 heads). Then, all the outputs are concatenated, as shown in Equation (8).

[0094]

[0095] In equation (8), O1, O2, O N For the output of each head, Concat is the dimension concatenation function. For features that embed location information, O represents the output of N heads.

[0096] Subsequently, the output O of the multi-head self-attention layer is input into the feedforward layer for calculation. The calculation of the feedforward layer is shown in equation (9):

[0097] M = PF(LN(O)) + O (9)

[0098] In equation (9), O is the output of the multi-head self-attention layer, LN represents layer normalization, PF represents the feedforward layer, and M is the output of the entire encoder layer.

[0099] Step 5.3: Use the motor imagery EEG classification layer to predict the motor imagery task category, as shown in Equation (10):

[0100] P = FC(Log(Pool(M⊙M))) (10)

[0101] In Equation (10), ⊙ represents the dot product operation, M is the output of the deformer-based encoder layer, Pool is the pooling layer used to reduce the feature size, Log is the logarithmic activation function, and FC represents the fully connected layer. P is the prediction probability of the shallow deformer ST model for motor imagery EEG.

[0102] Step 6: Use ensemble learning to improve model accuracy during the prediction phase. Ensemble learning combines multiple weakly supervised models to obtain a better supervised model. Many ensemble learning methods have achieved great success in many different practical tasks. Here, we adopt... Figure 3 The method shown uses mirror EEG signals and mirror network structures to achieve ensemble learning, specifically:

[0103] First, the ST model trained in step 5 is copied as a mirror ST model. The original EEG signal is input into the original ST model for prediction, and the mirror EEG signal is input into the mirror ST model for prediction. Then, the prediction results of the two models are probabilistically integrated.

[0104] The probability integration method used is shown in equation (11):

[0105]

[0106] In equation (11), P l o This represents the probability of the raw EEG signal predicting the left-hand motor imagery task. P represents the predictive probability of mirror EEG signals for the right-hand motor imagery task. l P r These represent the model's final predicted probabilities for the left-hand motor imagery task and the right-hand motor imagery task, respectively. The final predicted label of the raw EEG signal is the category with the highest final predicted probability.

[0107] By mirroring the EEG signal and the network structure, classification performance is improved through ensemble learning without increasing the model and computational burden.

[0108] Dataset introduction:

[0109] The BCI Competition IV dataset 2a contains EEG signals from nine subjects for four categories of motor imagery recognition tasks (left hand, right hand, both feet, and tongue). Each subject underwent two EEG samplings at different times, with 288 EEG signals collected each time (72 signals per category). The EEG sampling frequency was 250 Hz. The amplifier resolution was 100 μV. The trial duration was: a fixed crossover display time of 2 seconds, a cue display time of 1.25 seconds, and a motor imagery time of 3 seconds. The raw signals were preprocessed using a bandpass filter ranging from 0.5 Hz to 100 Hz, and a 50 Hz notch filter was used to suppress AC power frequency noise.

[0110] The BCI Competition IV dataset 2b contains EEG signals from nine subjects for two classes of motor imagery recognition tasks (left hand and right hand). Each subject had five EEG recordings: the first three for training and the last two for testing. The first two sets were based on a no-feedback model, while the last three sets displayed on-screen cues with smiley face feedback. Each trial consisted of 120–160 trials. EEG signals from three electrodes (C3, C2, and C4) were sampled at 250 Hz. The raw signals were preprocessed using bandpass filters from 0.5 Hz to 100 Hz, and AC noise was suppressed using a 50 Hz notch filter.

[0111] To verify the effectiveness of the proposed shallow mirror deformer (ST) model, experiments were conducted according to the framework of the specific implementation scheme. We evaluated the performance of the proposed model on new subjects. Specifically, training data from nine subjects in dataset 2a of BCI Competition IV were included in the training set, test data from all subjects in dataset 2a were included in the validation set, and test data from all subjects in dataset 2b of BCI Competition IV were included in the test set. Therefore, the subjects in the test set were new subjects because they were completely different from the subjects in the training set. The experimental results are shown in Table 1.

[0112] Since we found that training typically converges within 500 iterations, we set the maximum number of training iterations to 500 to ensure model convergence. Due to the differences among subjects in datasets 2a and 2b, overfitting frequently occurs during training; therefore, we employed an early stopping strategy to prevent overfitting. We used the maximum accuracy on the validation set as the target accuracy on the training set, and evaluated the model based on the test accuracy when the training set accuracy first exceeded the target accuracy. Furthermore, we included the maximum test accuracy after 500 iterations and the average test accuracy after 100–200 iterations in the comparison to assess the overall model performance.

[0113] Table 1 compares the experimental results of the proposed model and the latest algorithm on new subjects.

[0114]

[0115] The first column represents the test dataset and bandpass filter, encoded in "dataset-filter" format. For example, "2a-0Hz" indicates that the EEG data from dataset 2b consists of a training set and a validation set, with the test set from 2a used as the test set, and preprocessed using a 0-38Hz bandpass filter. Five state-of-the-art models, including Shallow ConvNet, DeepConvNet, FBCNet, Min2Net, and EEGNet, are used for comparison. The highest accuracy for each experimental setting is indicated in bold. In addition to the Shallow Mirror Deformer (SMT), the Shallow Deformer (ST) without a mirror network structure is also included in the performance comparison.

[0116] Furthermore, a paired-samples one-sided Student's t-test was used to confirm the significance of the performance improvement in the experimental comparison. The null hypothesis of the test was that the accuracy of the SMT model was equal to that of the state-of-the-art algorithm model, and it was compared with the accuracy of the state-of-the-art algorithm model based on different datasets and bandpass filters. Table 2 shows the p-values ​​of SMT and the state-of-the-art algorithm model.

[0117] Table 2. Statistical significance comparison of performance differences between the Shallow Mirror Deformer Model (SMT) and the latest algorithm.

[0118]

[0119] The following conclusions can be drawn from the above experimental results: (1) The SMT method performs better than the latest algorithm on new subjects; (2) The paired-sample one-sided Student's t test results show that the accuracy provided by SMT is significantly improved.

Claims

1. A multi-agent motor imagery recognition method of a shallow mirror transformer network, characterized by, The specific implementation steps are as follows: Step 1, intercept the electroencephalogram signal data; In step 1, the interception range is 0.5 seconds before the motor imagery task starts to 4 seconds after the motor imagery starts; Step 2, select the required C electroencephalogram channels and filter them using a 3rd order band-pass filter of 0-38 Hz; In step 2, the C3, C4, and Cz electroencephalogram channels are used; Step 3, perform exponential weighted average normalization on each channel of the filtered electroencephalogram data to standardize the electroencephalogram signal; The specific process of the exponential weighted average normalization operation in step 3 is as follows: First, the exponential moving average m is calculated t : m t = f mean(x t ) + (1 - f) m t-1 (1) In formula (1), x t is the value of the tth sample point of the original electroencephalogram signal, m t is the moving average of the electroencephalogram signal at the tth sample point, f is a pre-set update factor, mean(x t ) is the electroencephalogram signal x t is averaged on multiple channels; Recalculating the exponential moving variance v t : v t = f - (m t - x t ) 2 + (1 - f) - v t-1 (2) Finally, the exponentially weighted moving average normalized electroencephalogram signal x is obtained t ' In formula (3), is a maximum function that takes the maximum of two parameters, ε = 10 -6 ; Step 4, expand the electroencephalogram signal by creating a mirror image; After standardizing the electroencephalogram signal in step 3, exchange the electroencephalogram data corresponding to the C3 and C4 channels to construct a mirror image electroencephalogram, and then exchange the left and right hand motor imagery electroencephalogram signal labels; Step 5, input the mirror image electroencephalogram signal generated in step 4 together with the original electroencephalogram signal into the ST model for training; The ST model in step 5 includes a CNN-based feature extraction layer, a transformer-based encoder layer, and a motor imagery electroencephalogram classification layer; the transformer-based encoder layer is composed of a position embedding layer, a multi-head self-attention layer, and a feedforward layer; the motor imagery electroencephalogram classification layer is composed of a fully connected layer and a softmax function, and the number of output nodes is equal to the number of motor imagery categories; The specific steps of step 5 are as follows: Step 5.1, define the set of the generated mirror electroencephalogram signals and the original electroencephalogram signals as X ∈ R K×C , wherein K is the number of sampling points, and C is the number of channels of the EEG; the batch size of the model training is B, and thus the dimension size of the input data is B × 1 × K × C; Therefore, the calculation of the CNN-based feature extraction layer is shown in equation (4): F = BN(C s (C t (X)))(4) In formula (4), X represents the set of the mirrored electroencephalogram signal and the original electroencephalogram signal generated in step 4, C t represents a time-domain convolution with a size of 1x25, used to simulate a band-pass filter in the FBCSP algorithm;C s represents a spatial filter with a size of Cx1, used to fuse information from all EEG channels, thereby simulating a CSP spatial filter; BN represents batch normalization on the extracted features, and F represents features extracted by the CNN-based feature extraction layer; Step 5.2, the electroencephalogram feature information F extracted by the feature extraction layer will be input into the transformer-based encoder layer to obtain global time domain information, as follows: The representation of the position embedding layer in the transformer-based encoder layer is shown in equation (5): In formula (5), F is a feature extracted by a CNN-based feature extraction layer, Pos is a set of standard learnable position embedding parameters, is a feature embedded with position information; After being processed by the position embedding layer, the features embedded with position information are input into a multi-head self-attention layer in a residual network structure; in the residual network, layer normalization is first performed; then, the multi-head self-attention layer maps the query vector and the key vector, the value vector to the output in the form of a weighted sum of the value vector, and the calculation manner of the query vector, the key vector and the value vector is as shown in formula (6): In formula (6), W i q W i k W i v are the projection matrix of the query vector, the key vector and the value vector of the i-th head respectively, LN represents layer normalization, Q i , K i , V i represent the query vector, the key vector and the value vector of the i-th head respectively; then, the similarity between the query vector and the key vector is calculated by dot product, and the attention weight of the corresponding value vector is obtained by applying the softmax function after scaling; therefore, the output is the weighted sum of all value vectors, as shown in formula (7): In formula (7), O i is an output of the i-th head, denotes transposing the key vector of the i-th head, Q i is a query vector of the i-th head, V i is a value vector corresponding to the i-th head, d k is used to scale the size of the key vector; Then, all the outputs are connected together, as shown in equation (8): In formula (8), O1, O2, O N is the output of each head, Concat is a dimension concatenation function, is the feature embedded with location information, O is the output of N heads; After that, the output O of the multi-head self-attention layer is input into the feedforward layer for calculation; the calculation of the feedforward layer is shown in equation (9): M = PF(LN(O)) + O (9) In equation (9), O is the output of the multi-head self-attention layer, LN represents layer normalization, PF represents the feedforward layer, and M is the output of the entire encoder layer; Step 5.3, use the motor imagery electroencephalogram classification layer to predict the motor imagery task category, as shown in equation (10): P = FC(Log(Pool(M⊙M))) (10) In equation (10), is the dot product operation, M is the output of the transformer-based encoder layer, Pool is the pooling layer used to reduce the feature size; Log is the logarithmic activation function, FC represents the fully connected layer; P is the prediction probability of the shallow transformer ST model for motor imagery electroencephalogram; Step 6, use ensemble learning to improve the model accuracy in the prediction phase.

2. The multi-agent motion imagination recognition method of the shallow mirror transformer network according to claim 1, characterized in that, The specific steps of step 6 are as follows: Copy the trained ST model in step 5 as the mirror image ST model, input the original electroencephalogram signal into the original ST model for model prediction, and input the mirror image electroencephalogram signal into the mirror image ST model for model prediction; then, perform probability ensemble on the prediction results of the above two models: The probability ensemble method used is shown in equation (11): In formula (11), P l o denotes the prediction probability of the original EEG signal to the left-hand motor imagery task, denotes the prediction probability of the mirror EEG signal to the right-hand motor imagery task, P l , P r are the final prediction probabilities of the model to the left-hand motor imagery task and the right-hand motor imagery task, respectively; the final prediction label of the original EEG signal is the class with the maximum final prediction probability.

Images