Electroencephalogram signal motor imagery decoding method and deep learning neural network model

Abstract

The application provides a kind of electroencephalogram motion imagination decoding method and deep learning neural network model, network structure includes initial feature extraction module, multi-branch transformer module, fusion attention module and classifier module, initial feature extraction module preliminary extracts the space-time feature map of electroencephalogram;Multi-branch transformer module processes the space-time feature map output by initial feature extraction module into different feature depth combinations long and short feature sequence;Fusion attention module fuses feature sequence according to self-attention mechanism, obtains final features for classification;Classifier module uses fusion attention module output, then through linear summation, learns the position relationship between sequence internal elements, completes classification, realizes the effective learning of global space-time frequency characteristics of motion imagination electroencephalogram data, with end-to-end neural network model, can simply and accurately predict motion imagination action label.

Description

Technical Field

[0001] This invention relates to the field of electroencephalogram (EEG) signal analysis technology, and in particular to a method for decoding motor imagery from EEG signals and a deep learning neural network model. Background Technology

[0002] Stroke, commonly known as apoplexy, is the leading cause of death and disability among adults in my country, and the burden of stroke is expected to increase explosively in the future. Most stroke patients suffer from motor dysfunction such as hemiplegia and spasticity, with more than half experiencing functional impairments in the distal upper limbs, such as the hands. This not only limits the patient's ability to live independently but also places a heavy emotional and economic burden on families and society. Because hand movements are delicate and complex, involving a wide range of cerebral cortex regions, rehabilitation after motor function impairment is more difficult than for other parts of the body. Traditional rehabilitation therapy often involves continuous passive exercises led by therapists. This method lacks active brain involvement, and even when successfully applied clinically, the degree of functional recovery gradually decreases once the patient is discharged, failing to address the root cause of the stroke and achieve neurological rehabilitation.

[0003] Brain-computer interface (BCI) technology, as a novel rehabilitation method, not only stimulates patients' intention to move their brains and combines this movement-related brain activity with actual actions, but also helps reduce the burden on hospitals in terms of personnel and space, providing patients with convenient and long-term training, and bringing new hope for solving the problem of hand motor function rehabilitation after stroke.

[0004] With the significant improvement in computer hardware and software performance and the accumulation of big data in various fields, deep learning (DL) technology has developed rapidly. Deep learning-based neural decoding has been widely applied in brain injury rehabilitation and robotics, providing new methods and opportunities for brain-computer interaction in rehabilitation robots. However, traditional convolutional and recurrent network structures have limited learning perspectives and cannot fully extract the multidimensional features of nonlinear and non-stationary EEG signals, making them unsuitable for decoding complex EEG signals related to fine motor skills. Summary of the Invention

[0005] This invention discloses a method for decoding motor imagery signals from electroencephalogram (EEG) signals and a deep learning neural network model. It solves the problem that existing traditional convolutional and recurrent network structures have limited learning perspectives and cannot fully extract the multidimensional features of nonlinear and non-stationary EEG signals, thus failing to decode complex EEG signals related to fine motor movements. Through an end-to-end deep learning neural network model, it achieves the learning of global spatiotemporal frequency features of motor imagery EEG data, and quickly and accurately predicts motor imagery action labels.

[0006] To achieve the above objectives, the technical solution of the present invention is specifically implemented as follows:

[0007] This invention discloses a method for decoding motor imagery from electroencephalogram (EEG) signals, the method comprising:

[0008] Temporal features of EEG signals are extracted by temporal convolution, while spatial features of EEG signals are preserved. Spatial features of EEG signals are then extracted by spatial convolution to obtain a coarse spatiotemporal convolution feature map of EEG signals.

[0009] The spatial weights of the obtained coarse spatiotemporal convolutional feature maps are enhanced to obtain spatiotemporal feature maps;

[0010] The acquired spatiotemporal feature maps are filtered into several time series that focus on different frequency bands and contain spatiotemporal features of EEG signals;

[0011] Trainable location parameters are added to each of the acquired time series.

[0012] Several time series with added positional parameters are encoded using three consecutive transformer layers to output several corresponding feature sequences.

[0013] The output feature sequence is subjected to global average pooling over time to obtain several single-element features corresponding to the feature sequence.

[0014] Upsample the feature dimension of several single-element features in the output to the same dimension as the lowest frequency feature sequence;

[0015] Stack several single-element features of the same dimension, and use MSA to redistribute attention to the dimension of the stacked single-element feature sequence;

[0016] Global flat pooling is performed on the single-element feature sequences after attention redistribution to obtain a global spatiotemporal frequency high-level representation for classification.

[0017] Using a linear fully connected layer as a classifier, the MR category of EEG data is identified based on the acquired global spatiotemporal frequency high-level representation.

[0018] Furthermore, the dimension of the location parameter added to the time series is the same as the dimension of the time series in which it is located.

[0019] Furthermore, the acquired spatiotemporal feature maps are filtered into four time series that focus on different frequency bands and contain spatiotemporal features of EEG signals.

[0020] Furthermore, the spatiotemporal feature map is filtered using a four-branch processing method, generating four time series with different feature depth combinations of varying lengths. The convolution kernel sizes of the two-dimensional convolutions in the four branches are (X,5), (X,10), (X,20), and (X,40), where the value of X is the same as the number of channels in the spatiotemporal feature map. The long sequence obtains global information on the high-frequency spatiotemporal distribution of the EEG signal, while the short sequence obtains low-frequency global information.

[0021] Furthermore, the method also includes:

[0022] The predicted motion imagery labels and the true labels are minimized using the loss function shown in the following formula:

[0023]

[0024] Where c represents the number of label categories, n represents the number of samples, y represents the true label, and a represents the model prediction probability of that category after softmax activation.

[0025] Another aspect of this invention discloses a deep learning neural network model for decoding motor imagery in electroencephalogram (EEG) signals, comprising an initial feature extraction module, a multi-branch transformer module, a fusion attention module, and a classifier module. The initial feature extraction module initially extracts spatiotemporal feature maps of the EEG signals. The multi-branch transformer module processes the spatiotemporal feature maps output by the initial feature extraction module into feature sequences of varying lengths with different feature depths. The fusion attention module fuses the feature sequences based on a self-attention mechanism to obtain the final features used for classification. The classifier module utilizes the output of the fusion attention module and then learns the positional relationships between elements within the sequence through linear summation to complete the classification.

[0026] Furthermore, the initial feature extraction module includes a spatiotemporal convolution unit and a squeezing activation unit. The spatiotemporal convolution unit includes temporal convolution and spatial convolution. The temporal convolution is used to extract the temporal features of the EEG signal, and the spatial convolution is used to extract the spatial features of the EEG signal, outputting a coarse spatiotemporal convolution feature map. The squeezing activation unit is a computational unit with a residual structure. The squeezing activation unit compresses the coarse spatiotemporal convolution feature map along the temporal dimension through global average pooling, strengthens the weight allocation in the spatial dimension, and outputs a spatiotemporal feature map.

[0027] Furthermore, the spatiotemporal convolutional unit also includes batch normalization, activation layers, and loss layers.

[0028] Furthermore, the multi-branch transformer module includes a slice embedding unit, a position embedding unit, and a Transformer encoder unit. The slice embedding unit is used to filter the spatiotemporal feature map to a time series that focuses on different frequency bands and contains spatiotemporal features through convolutional kernels of different lengths and numbers. The position embedding unit is used to add trainable position parameters to the time series output by the slice embedding unit, and the dimension of the position parameters is the same as the dimension of the time series. The Transformer encoder unit is used to perform three-layer Transformer encoding on the time series output by the slice embedding unit to generate a feature sequence.

[0029] Furthermore, the fusion attention module includes a concatenated operation unit for aggregating several feature sequences and an MSA unit for adaptively learning the weighted relationship between the feature sequences. The concatenated operation unit includes a GAP layer, an upsampling layer, and a stacking layer.

[0030] Beneficial technical effects:

[0031] 1. This invention discloses a deep learning neural network model for decoding motor imagery in electroencephalogram (EEG) signals, comprising an initial feature extraction module, a multi-branch transformer module, a fusion attention module, and a classifier module. The initial feature extraction module initially extracts the spatiotemporal feature map of the EEG signal; the multi-branch transformer module processes the spatiotemporal feature map output by the initial feature extraction module into feature sequences of varying lengths with different feature depths; the fusion attention module fuses the feature sequences based on a self-attention mechanism to obtain the final features for classification; the classifier module uses the output of the fusion attention module and then learns the positional relationships between elements within the sequence through linear summation to complete the classification. This end-to-end neural network model learns the global spatiotemporal frequency features of motor imagery EEG data, enabling rapid and accurate prediction of motor imagery action labels.

[0032] 2. In this invention, the spatiotemporal convolutional unit also includes batch normalization, activation layer and loss layer, which can accelerate convergence and prevent overfitting;

[0033] 3. In this invention, a squeeze activation unit is set in the deep learning neural network model, which effectively improves the sensitivity of relevant channels, suppresses irrelevant channels, highlights the interdependence between modeling channels, and adaptively recalibrates the features at the spatial level. Attached Figure Description

[0034] To more clearly illustrate the technical solution of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below.

[0035] Figure 1This is a schematic diagram of the structure of a deep learning neural network model for decoding motor imagery of electroencephalogram signals according to the present invention;

[0036] Figure 2 This is a schematic diagram of the structure of a deep learning neural network for decoding motor imagery of electroencephalogram signals, which integrates an attention module and a classifier, as described in this invention.

[0037] Figure 3 This is a visualization of the scalp topological features using a trained deep learning neural network model on test data. Detailed Implementation

[0038] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0039] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this application. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0040] Unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the invention. It should also be understood that, for ease of description, the dimensions of the various parts shown in the drawings are not drawn to actual scale. Techniques, methods, and devices known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and devices should be considered part of the specification. In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values. It should be noted that similar reference numerals and letters in the following figures denote similar items; therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0041] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0042] To address the limitations of traditional convolutional and recurrent network structures in model learning, which cannot fully extract the multidimensional features of EEG (electroencephalogram) signals, which are nonlinear and non-stationary, this invention discloses a deep learning neural network for decoding motor imagery of EEG signals and a method for decoding EEG signals using this deep learning neural network.

[0043] This invention discloses, in one aspect, a deep learning neural network for decoding motor imagery from electroencephalogram (EEG) signals, see [link to relevant documentation]. Figure 1 The network structure includes an initial feature extraction module, a multi-branch transformer module, a fusion attention module, and a classifier module. The initial feature extraction module initially extracts the spatiotemporal feature map of the EEG signal. The multi-branch transformer module processes the spatiotemporal feature map output by the initial feature extraction module into feature sequences of varying lengths with different feature depths. The fusion attention module fuses the feature sequences based on a self-attention mechanism to obtain the final features used for classification. The classifier module uses the output of the fusion attention module and learns the positional relationships between elements within the sequence through linear summation to complete the classification. With an end-to-end neural network model, it learns the global spatiotemporal frequency features of motor imagery EEG data, enabling fast and accurate prediction of motor imagery action labels.

[0044] As an embodiment of the present invention, the initial feature extraction module includes a spatiotemporal convolution unit and a squeeze activation unit. Preferably, the spatiotemporal convolution unit includes temporal convolution and spatial convolution. Temporal convolution is used to extract the temporal features of the EEG signal, and spatial convolution is used to extract the spatial features of the EEG signal, outputting a coarse spatiotemporal convolution feature map. To achieve fast convergence and prevent overfitting, the spatiotemporal convolution unit is also provided with a batch normalization layer, an activation layer, and a loss layer. The squeeze activation unit is a computational unit with a residual structure. The squeeze activation unit compresses the coarse spatiotemporal convolution feature map along the temporal dimension through global average pooling, strengthens the weight distribution in the spatial dimension, and outputs a spatiotemporal feature map. Preferably, in the activation part, the squeeze activation unit is provided with a bottleneck structure consisting of two fully connected layers and ReLU activation, and then uses Sigmoid activation as a gated activator. The squeeze activation unit effectively improves the sensitivity of relevant channels, suppresses irrelevant channels, highlights the interdependence between modeling channels, and adaptively recalibrates the features at the spatial level.

[0045] As an embodiment of the present invention, the multi-branch transformer module processes the spatiotemporal feature map output by the initial feature extraction module into feature sequences of varying lengths with different feature depth combinations. Preferably, the multi-branch transformer module includes a slice embedding unit, a position embedding unit, and a Transformer encoder unit. The slice embedding unit is used to filter the spatiotemporal feature map to a time series containing spatiotemporal features and focusing on different frequency bands through convolutional kernels of different lengths and numbers. Specifically, the number of slice embedding units in the multi-branch transformer module is the same as the number of Transformer encoder units, that is, a slice embedding unit is set before each Transformer encoder unit. In this embodiment, the multi-branch transformer module sets 4 slice embedding units and 4 parallel Transformer encoder units. The slice embedding unit uses two-dimensional convolution to filter the spatiotemporal feature map output by the initial feature extraction module. The spatiotemporal feature map is filtered to obtain time series containing spatiotemporal features at different frequency bands. In this embodiment, the kernel sizes of the four slice embedding units are (X,5), (X,10), (X,20), and (X,40), respectively, and the number of filters are 10, 20, 40, and 80, respectively. Here, the specific value of X should be the same as the number of channels of the spatiotemporal feature map output by the initial feature extraction module. For example, a spatiotemporal feature map of size (X,1000) can be processed by four slice embedding units to obtain... The time series have dimensions of (200,10), (100,20), (50,40), and (25,80), respectively. The first dimension is the sequence length, and the second dimension is the embedding size. According to the Nyquist theorem, when a sampling frequency of 250Hz is used, these four time series contain frequency information within 25Hz, 12.5Hz, 6.25Hz, and 3.125Hz, respectively. That is, the time series of different lengths output by each chip embedding unit contain different frequency information from the original data.

[0046] As an embodiment of the present invention, the spatiotemporal feature map output by the initial feature extraction module is processed by four slice embedding units to generate four time series of different lengths. The position embedding unit adds trainable position parameters to the four different time series output by the four slice embedding units. The dimension of the position parameters to be added to the time series must be the same as the dimension of the corresponding time series. The positional relationship between elements within the sequence is learned by influencing training through linear summation.

[0047] As an embodiment of the present invention, each Transformer encoder unit includes a three-layer Transformer encoder. Four time series of different lengths with added position parameters are encoded by the three-layer Transformer encoder unit to generate feature sequences of different lengths. That is, the long sequence focuses on the global information of the high-frequency temporal and spatial distribution of the EEG signal, while the short sequence focuses on the low-frequency global information.

[0048] As an embodiment of the present invention, the fusion attention module fuses feature sequences based on a self-attention mechanism to obtain the final features for classification. Preferably, the fusion attention module includes a concatenation operation unit for aggregating several feature sequences and an MSA unit for adaptively learning the weighted relationship between each feature sequence. The concatenation operation unit includes a GAP layer, an upsampling layer, and a stacking layer. The GAP layer can compress the information of each feature sequence along the length of the feature sequence by global average pooling. The upsampling layer can perform linear interpolation upsampling of the feature sequences. The stacking layer stacks the feature sequences output by the upsampling layer to keep the feature dimension of other feature sequences the same as the dimension of the longest branch (i.e., the lowest frequency branch). After concatenation, the MSA unit is used to optimize the attention weights of different feature sequences.

[0049] As an embodiment of the present invention, the classifier module completes the classification using the final features obtained by the fusion attention module. Specifically, the classifier module uses the output of the fusion attention module and then learns the positional relationship between elements within the sequence by linear summation to complete the classification. In the present invention, a fully connected module is used as the classifier, and an LN is placed before the fully connected module to normalize the contextual relationship within the feature elements.

[0050] The steps for decoding EEG signals using the deep learning neural network disclosed in this invention include:

[0051] The multi-channel continuous EEG data in the dataset are all marked with task start points. The continuous signals are segmented into data samples based on specific motor imagery tasks using the marked points, and a deep learning neural network is trained.

[0052] Specifically, 70% of the data in the EEG signal dataset is used as the training set, 20% as the validation set, and 10% as the test set. The training set is used to train the deep learning model, allowing the model to learn the data distribution of the training set; the validation set is used to verify the generalization ability of the model during training and record the model with the best generalization ability; the test set is used to evaluate the performance of the trained model.

[0053] Preferably, a deep learning neural network is trained:

[0054] First, set the model hyperparameters: batch size: 100; learning rate: 10. -3 Number of iterations: 200;

[0055] Then, the deep learning neural network model is trained, and the cross-entropy loss function is used as a metric for the performance of the deep learning neural network model:

[0056]

[0057] Where c represents the number of label categories, n represents the number of samples, u represents the true label, and a represents the model's predicted probability for that category after softmax activation. The physical meaning of this loss function is the difference between the probability distribution predicted by the model and the probability distribution of the true label. The model updates its parameters based on the gradients obtained through backpropagation, making the model's predicted motion and action labels closer to the true labels. During training, there are training losses and validation losses. If both decrease with the same trend, it indicates that the model's predictive ability is gradually improving. When both stop decreasing, it means the model has converged, and training ends.

[0058] The brain signals during the motor imagery task are collected using an EEG signal acquisition device, and the motor imagery EEG signals are input into a pre-trained deep learning neural network to directly decode the labels of the motor imagery actions.

[0059] Specifically, the trained deep learning neural network decodes the input EEG signals as follows:

[0060] The spatiotemporal convolutional unit extracts the temporal features of the EEG signal through temporal convolution and retains the spatial features of the EEG signal. Then, it extracts the spatial features of the EEG signal through spatial convolution to obtain a coarse spatiotemporal convolutional feature map of the EEG signal.

[0061] The squeeze activation unit strengthens the spatial dimension weight allocation of the acquired coarse spatiotemporal convolutional feature map to obtain spatiotemporal feature map;

[0062] The four slice embedding units filter the acquired spatiotemporal feature maps into four time series of varying lengths that focus on different frequency bands and contain spatiotemporal features of EEG signals. Preferably, the spatiotemporal feature maps are filtered using two-dimensional convolution, and the kernel sizes of the four two-dimensional convolutions are (X,5), (X,10), (X,20) and (X,40), where the value of X is the same as the number of channels in the spatiotemporal feature map.

[0063] The location embedding unit adds trainable location parameters to the four time series of varying lengths. It is important to note that the dimension of the location parameters added to the time series is the same as the dimension of the time series in which they are located.

[0064] Four transformer encoder units perform three consecutive transformer encodings on four time series of different lengths with added position parameters, and output four corresponding feature sequences.

[0065] In the fusion attention module, the GAP layer in the concatenated operation unit compresses the information of each feature sequence along the sequence length using global average pooling to obtain four single-element features corresponding to the feature sequences.

[0066] In the fusion attention module, the upsampling layer in the cascaded operation unit upsamples four single-element features corresponding to the feature sequence through linear interpolation, upsampling the feature dimension of the four single-element features to the same dimension as the lowest frequency feature sequence.

[0067] In the fusion attention module, the stacked operation unit stacks four single-element features, and the MSA unit redistributes the attention to the dimension of the stacked single-element feature sequence.

[0068] Global average pooling is performed on the single-element feature sequences after attention redistribution to obtain a high-level global spatiotemporal frequency representation for classification.

[0069] Using a fully connected linear network as a classifier, the method identifies the MR category of EEG data based on the acquired global spatiotemporal frequency high-level representation. It's important to understand that this method minimizes the predicted motor imagery action label and the ground truth label using the loss function shown in the following formula (the ground truth label is set to be visible for each branch to facilitate discriminative feature extraction for each branch):

[0070]

[0071] Where c represents the number of label categories, n represents the number of samples, y represents the true label, and a represents the model prediction probability of that category after softmax activation.

[0072] Generally, the decoding accuracy of a model is evaluated by comparing its predicted labels with the true labels on the test set. Accuracy is the ratio of the number of test samples whose predicted labels match the true labels to the number of test samples, defined as follows:

[0073]

[0074]

[0075] Where S is the test set, S i To predict the test sample set whose labels match the true labels, y i For the actual coordinates, Predict coordinates for the model.

[0076] The model disclosed in this invention was compared with other commonly used motor imagery neural decoding architectures. The test results are shown in Table 1. The results were obtained by decoding on the public dataset BCI Competition IV-2a:

[0077] Table 1:

[0078]

[0079]

[0080] As can be seen from Table 1, the architecture proposed in this invention generally outperforms other model architectures in terms of individual decoding results and average decoding results on public datasets. This indicates that the architecture based on multi-branch and fused transformer modules is indeed superior to other methods in its ability to capture global features of motion imagery signals. This demonstrates that the deep learning neural network and decoding method disclosed in this invention can accurately decode EEG imagery action labels.

[0081] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0082] The above embodiments are merely descriptions of preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made by those skilled in the art to the technical solutions of the present invention without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. A method for decoding motor imagery from electroencephalogram (EEG) signals, characterized in that, The method includes: Temporal features of EEG signals are extracted by temporal convolution, while spatial features of EEG signals are preserved. Spatial features of EEG signals are then extracted by spatial convolution to obtain a coarse spatiotemporal convolution feature map of EEG signals. The spatial weights of the obtained coarse spatiotemporal convolutional feature maps are enhanced to obtain spatiotemporal feature maps; The acquired spatiotemporal feature maps are filtered into several time series that focus on different frequency bands and contain spatiotemporal features of EEG signals; Trainable location parameters are added to each of the acquired time series. Several time series with added positional parameters are encoded using three consecutive transformer layers to output several corresponding feature sequences. The output feature sequence is subjected to global average pooling over time to obtain several single-element features corresponding to the feature sequence. Upsample the feature dimension of several single-element features in the output to the same dimension as the lowest frequency feature sequence; Stack several single-element features of the same dimension, and use MSA to redistribute attention to the dimension of the stacked single-element feature sequence; Global flat pooling is performed on the single-element feature sequences after attention redistribution to obtain a global spatiotemporal frequency high-level representation for classification. Using a linear fully connected layer as a classifier, the MR category of EEG data is identified based on the acquired global spatiotemporal frequency high-level representation.

2. The method for decoding motor imagery from electroencephalogram (EEG) signals according to claim 1, characterized in that, The dimension of the location parameter added to the time series is the same as the dimension of the time series in which it is located.

3. The method for decoding motor imagery from electroencephalogram (EEG) signals according to claim 1, characterized in that, The acquired spatiotemporal feature maps were filtered into four time series that focus on different frequency bands and contain spatiotemporal features of EEG signals.

4. The method for decoding motor imagery from electroencephalogram (EEG) signals according to claim 3, characterized in that, The spatiotemporal feature map is filtered using a four-branch processing method, generating four time series with different feature depth combinations of varying lengths. The convolution kernel sizes of the two-dimensional convolutions in the four branches are (X,5), (X,10), (X,20), and (X,40), where the value of X is the same as the number of channels in the spatiotemporal feature map. The long sequence obtains global information on the high-frequency spatiotemporal distribution of the EEG signal, while the short sequence obtains low-frequency global information.

5. The method for decoding motor imagery of electroencephalogram signals according to claim 1, characterized in that, The method further includes: The predicted motion imagery labels and the true labels are minimized using the loss function shown in the following formula: ， Where C represents the number of label categories, n represents the number of samples, y represents the true label, and a represents the model prediction probability of that category after softmax activation.

6. A deep learning neural network model for decoding the EEG signal motor imagery method of claim 1, characterized in that, include: The initial feature extraction module is used to obtain the spatiotemporal feature map of the EEG signal; The multi-branch transformer module is used to process the spatiotemporal feature map output by the initial feature extraction module into feature sequences of varying lengths with different feature depth combinations. A fusion attention module is used to fuse the feature sequences output by the multi-branch transformer module according to the self-attention mechanism to obtain the final features for classification. The classifier module utilizes the output of the fusion attention module and then learns the positional relationships between elements within the sequence through linear summation to complete the classification.

7. A deep learning neural network model for decoding motor imagery of electroencephalogram (EEG) signals according to claim 6, characterized in that, The initial feature extraction module includes: The spatiotemporal convolution unit includes temporal convolution and spatial convolution. The temporal convolution is used to extract the temporal features of the EEG signal, and the spatial convolution is used to extract the spatial features of the EEG signal, outputting a coarse spatiotemporal convolution feature map. The squeeze activation unit is a computational unit with a residual structure. The squeeze activation unit compresses the coarse spatiotemporal convolutional feature map along the time dimension through global average pooling, strengthens the weight distribution in the spatial dimension, and outputs a spatiotemporal feature map.

8. A deep learning neural network model for decoding motor imagery of electroencephalogram signals according to claim 7, characterized in that, The spatiotemporal convolutional unit also includes batch normalization, activation layers, and loss layers.

9. A deep learning neural network model for decoding motor imagery of electroencephalogram signals according to claim 6, characterized in that, The multi-branch transformer module includes: The slice embedding unit is used to filter the spatiotemporal feature map to a time series that focuses on different frequency bands and contains spatiotemporal features through convolutional kernels of different lengths and numbers; A location embedding unit is used to add trainable location parameters to the time series output by the slice embedding unit, and the dimension of the location parameters is the same as the dimension of the time series. The Transformer encoder unit is used to perform three-layer Transformer encoding on the time series output by the slice embedding unit to generate a feature sequence.

10. A deep learning neural network model for decoding motor imagery of electroencephalogram (EEG) signals according to claim 6, characterized in that, The fusion attention module includes a concatenated operation unit for aggregating several feature sequences and an MSA unit for adaptively learning the weighted relationship between the feature sequences. The concatenated operation unit includes a GAP layer, an upsampling layer, and a stacking layer.

Images