A method and system for decoding complex action motor imagery based on electroencephalogram

Abstract

The present invention discloses a method and system for decoding complex movement imagination based on electroencephalogram. Aiming at the new brain-computer interface task of multi-action movement imagination of unilateral upper limb, a learning model based on wavelet transform-3D convolution-Transformer is designed, which solves the problems of few recognizable actions of unilateral limb, low recognition efficiency and unclear brain patterns in traditional brain-computer interface systems. The differences from traditional movement imagination brain-computer interfaces are mainly as follows: The present invention designs a recognition mode for complex movement imagination of multi-actions of unilateral upper limb; The present invention designs a learning model based on wavelet transform-3D convolution-Transformer, which can obtain more accurate and stable results; The friendly result visualization interface enables the subjects to intuitively and conveniently observe their own test states.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence - pattern recognition - brain - computer interface, and particularly relates to a decoding method and system for complex - task motor imagery actions. Background Art

[0002] Brain - computer interface (BCI) is regarded as one of the important contents in national brain plans. BCI is a discipline that combines hardware, software, and algorithms. By using hardware devices to collect the cerebral cortex signals of a subject when performing an activity, and then analyzing the collected electroencephalogram (EEG) signals to identify the true intention of the subject, it can achieve the purpose of directly obtaining the true thoughts of the subject without relying on peripheral nerves and muscle tissues, thereby realizing the interaction and control between the human brain and the external environment.

[0003] BCI is divided into invasive EEG signals and non - invasive EEG signals. Among them, invasive EEG implants an array for collecting electrical signals in the cerebral cortex, which has the characteristics of stable data collection, less interference signals, and high recognition accuracy. However, due to its harmfulness to the subject, its application range is relatively narrow. Non - invasive EEG acquisition devices directly collect the scalp EEG of the subject. Because of its convenience in use and harmlessness to the subject, it is widely used in various studies.

[0004] Motor imagery refers to the subject generating specific EEG signals by imagining limb movements. During the whole process, there is no need for the patient's limbs to make actual movements, nor for external environmental stimuli, but it is a spontaneous behavior. Therefore, in practical applications, for patients who have lost some or all of their motor abilities due to diseases or accidental injuries, using motor imagery can generate specific EEG signals, thereby identifying the patient's motor intention. Combined with hardware devices, it can not only be used as a rehabilitation treatment method to achieve the patient's actively initiated rehabilitation treatment and help the patient restore their own motor ability to the greatest extent, but also assist the patient in controlling objects, enhancing motor flexibility, and improving the degree of freedom of movement.

[0005] Although rich research results have been achieved in task recognition based on EEG signals, there are still problems in the practical application of BCI. The main problems are as follows: existing motor imagery decoding methods have a single limb movement, and the degree of freedom in actual use is very limited; the motor brain patterns of complex unilateral limb movements have a high degree of overlap, and the movement instructions cannot be effectively analyzed. Summary of the Invention

[0006] The purpose of the present invention is to overcome the deficiencies of the prior art and provide a method and system for decoding complex movement imagination based on electroencephalogram (EEG). The present invention proposes a new learning framework based on a continuous wavelet transform-3D convolution-Transformer architecture, which can effectively solve the problems in traditional methods, such as fewer recognizable actions of unilateral limbs and lower recognition accuracy due to highly overlapping brain patterns.

[0007] To solve the above problems, the technical solutions adopted by the present invention include:

[0008] In the first aspect, the present invention provides a method for decoding complex movement imagination based on EEG, specifically including the following steps:

[0009] Step S1: Data acquisition.

[0010] 1.1 Experimental paradigm design: The experimental paradigm is the movement imagination of four states of the right hand, namely: imagining shoulder flexion, imagining elbow flexion, imagining fist clenching, and the static state. A single task is divided into two parts: a video prompt part and a task imagination part.

[0011] In the video prompt part, the subject is required to be in a calm state without tasks within t1 (t1 > 0 seconds, a manually set parameter), keep breathing smoothly, not blink, and have no other stimuli except the prompt.

[0012] In the task imagination part, the subject is required to keep the body still, have no limb movement, no distracting thoughts, and keep breathing smoothly and not blink within t2 (t2 > 0 seconds, a manually set parameter).

[0013] The appearance order of the four task prompts for the right hand is random, but the total number of experiments for each task is the same, and the imagination time does not exceed the preset time.

[0014] 1.2 Collect the EEG signals of the subject within t2 after the prompt ends, with a sampling frequency of 1000 Hz, and the acquisition device is a 64-channel NeuroScan that conforms to the 10-20 standard.

[0015] Step S2: Preprocess the EEG signals collected in step S1.

[0016] The preprocessing process includes: filtering, common average reference (CAR), ICA decomposition to remove electrooculogram, electromyogram, and electrocardiogram signals. Specifically:

[0017] 2.1 Filter the collected EEG signals to the main frequency band alpha (8-13 Hz) of movement imagination. The band-pass filter is a FIR filter.

[0018] 2.2 Apply common average reference (CAR) to reset the reference baseline.

[0019] 2.3 Apply ICA decomposition to the data obtained in 2.2, and remove artifacts such as electrooculogram, electrocardiogram, and myoelectric potential in the EEG signals according to independent component analysis.

[0020] Step S3: Feature extraction is performed on the preprocessed EEG signals obtained in step S2: Continuous wavelet transform is performed on each channel of the data, and the first-level data after decomposition replaces the original data to become the feature data.

[0021] The mother wavelet used is the Ricker wavelet; among them, the continuous wavelet transform formula is as follows:

[0022]

[0023] Among them, CWT(s,τ) represents the output, s represents the scale, and τ represents the translation factor. represents the complex conjugate function, and ψ(t) represents the mother wavelet. is the complex conjugate function of, <·> represents the inner product, x(t) represents the input data, represents the translation and scaling of the mother wavelet.

[0024] The Ricker wavelet transform formula is as follows:

[0025]

[0026] Among them, t is the time and f0 is the main frequency.

[0027] The feature conversion described above uses all 60 channels to perform CWT continuous wavelet decomposition, and uses the first-level data of the generated time-frequency signal to replace the original signal as the feature data.

[0028] Step S4: Spatial conversion part: According to the spatial distribution characteristics of the electrodes, the two-dimensional data is transformed into three-dimensional data by means of data replication, interpolation, and setting to zero.

[0029] Since the electrodes are irregularly distributed in space, the F area, FC area, CP area, and P area have 9 columns in each row, while the FP area, AF area, PO area, and O area have less than 9 columns of electrodes. In order to better extract the spatial features of the data, the data in the P area, AF area, PO area, and O area are replicated, interpolated, or set to zero according to the spatial distribution, so that the channel feature distribution presents a 9-row and 9-column distribution. As Figure 4 shown. After the conversion, the data format changes from C×T to W×H×T. Among them, C represents the number of channels, T represents the data length, W represents the width in the three-dimensional data, and H represents the height in the three-dimensional data.

[0030] Step S5: Input the feature data into the classification and recognition model to obtain the classification result. The classification and recognition model is divided into three parts, which are responsible for the channel dimension part, the time dimension part, and the classifier part respectively. Specifically:

[0031] 5.1 Channel dimension part: The channel dimension processing includes two parts, namely the multi-head attention mechanism part and the 3D convolution part. The purpose is to extract spatial features and reduce the data volume in the time dimension.

[0032] 5.1.1 Multi-head attention mechanism part: First, transform the data processed in 4.1 into 2D data (C1, T), where C1 represents the total number of expanded channels and T represents the data length. Then use the multi-head attention mechanism to calculate the correlation between each channel, and then perform a dot product with the input data. After that, reshape the data into three-dimensional data.

[0033] 5.1.2 The 3D convolution part contains two convolutional layers. The convolutional kernel size of the first convolutional layer is 2×2×100, and the stride is 1×1×100. After the first layer of convolution, batch normalization and the ReLU non-linear activation function are applied. The second convolutional kernel size is 8x8x4, and the stride size is 1×1×4. In the third dimension, the moving stride and the window size are the same. The number of convolutional kernels in the two layers is 2 and 10 respectively. After that, reshape the data into (n×m), where n is the number of convolutional kernels in the second layer, and m is the output length of the 3D convolution in the third dimension, that is, the number of time dimension slices.

[0034] 5.2 Time dimension part: Input the output of step S5.1 into the Transformer network. The Transformer network includes three identical encoders, and each encoder consists of two parts: multi-head attention and a feed-forward neural network. The multi-head attention part uses a residual connection and internally consists of a normalization layer, a multi-head attention layer, and a Dropout layer. The head data domain is consistent with m in the output of 5.1.. The feed-forward neural network part also uses a residual connection and internally includes a normalization layer, a feed-forward neural network, and a Dropout part.

[0035] 5.3 Classifier part: For the output result of step S5.2, it consists of a Reduce layer, a batch normalization layer, and a fully connected layer. Among them, the Reduce layer is responsible for taking the mean value of the two-dimensional data output by the Transformer in the first dimension, changing from (n×m) to (1×m).

[0036] The present invention provides a method for decoding complex action motor imagery based on electroencephalogram (EEG). By using a classification algorithm including continuous wavelet decomposition, 3D convolutional neural network, and Transformer to process the complex action EEG signals, classification results are obtained, while increasing the number of recognizable actions in motor imagery and improving classification stability.

[0037] In a second aspect, the present invention provides a real-time decoding system for complex action motor imagery based on EEG, which includes:

[0038] A data acquisition module, a preprocessing module, a feature extraction module, a spatial transformation module, a classifier module, and a visualization feedback module.

[0039] The data acquisition module is used to acquire EEG signals and real labels during the motor imagery of the subject.

[0040] The preprocessing module is used to preprocess the original EEG signals acquired by the data acquisition module, including filtering, common reference re-referencing, and ICA operations, for extracting effective frequency bands, removing irrelevant signals, and enhancing the signal-to-noise ratio.

[0041] The feature extraction module is used to perform effective feature transformation on the data processed by the preprocessing module.

[0042] The spatial transformation module, according to the physical positions of the electrodes, transforms the data extracted by the feature extraction module from two-dimensional space to three-dimensional space by means of interpolation, replication, and setting to zero.

[0043] The classifier module is used to predict and classify the motor imagery data of the subject. The classification model includes a channel dimension part, a time dimension part, and a classifier part.

[0044] The visualization feedback module is used to feedback the recognition results of the current task to the subject and present the states of various tasks in the current experiment in real time.

[0045] In a third aspect, the present invention provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed on a computer, the computer is made to execute the method.

[0046] In a fourth aspect, the present invention provides a computing device, including a memory and a processor. An executable code is stored in the memory. When the processor executes the executable code, the method is implemented.

[0047] Compared with the prior art and systems, the present invention mainly has the following advantages:

[0048] 1. For the multi-action decoding of single-limb motor imagery, compared with the single-limb single-action decoding of traditional motor imagery, the user can achieve more degrees of freedom of movement, which is more in line with the actual application scenarios.

[0049] 2. The present invention provides a deep learning model based on wavelet transform - 3D convolutional neural network - Transformer, which can more effectively extract the frequency features, time features and spatial features of electroencephalogram data, and can obtain more accurate and stable recognition ability in the field of single - limb complex action motor imagery with highly overlapping brain patterns.

[0050] 3. The present invention provides a real - time decoding system for complex action motor imagery based on electroencephalogram. Design a data analysis and processing result according to the above - mentioned pre - trained model to realize nervous system feedback. Users can see their own motor imagery analysis results and timely adjust the test state according to the movement results to achieve more accurate classification results.

[0051] 4. The present invention uses the first - level electroencephalogram data after wavelet transform as the input data of the subsequent model, which greatly enhances the distinguishability of multiple actions of single - limb motor imagery and improves the classification accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the drawings required in the embodiments.

[0053] Figure 1 It is the system framework diagram of the embodiment of the present invention.

[0054] Figure 2 It is the classification recognition model architecture diagram in the embodiment of the present invention.

[0055] Figure 3 It is the electroencephalogram channel diagram of the present invention.

[0056] Figure 4 It is the spatial redistribution diagram of the present invention.

[0057] Figure 5 It is the visualization feedback diagram of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0058] The following will combine the drawings in the embodiments of the present invention to describe the technical solutions and system designs in the embodiments of the present invention in detail. The described embodiments are only partial embodiments of the present invention.

[0059] The present invention provides a decoding method and system for complex action motor imagery based on electroencephalogram. The brain - computer interface system includes: a data acquisition module, a pre - processing module, a feature extraction module, a spatial conversion module, a classification model module and a visualization presentation module. The connection relationship of each module is as Figure 1 shown.

[0060] The specific implementation steps of the present invention are as follows:

[0061] Step S1: The data acquisition module consists of two parts: task control and EEG data acquisition.

[0062] 1.1 The task control part is designed using Eprime 2.0. The experimental paradigm is designed as a new brain-computer interface task for unilateral upper limb multi-action motor imagery, specifically the four states of "imagining fist clenching", "imagining elbow bending", "imagining shoulder extension", and "resting state" of the right hand. Each motor imagery experiment lasts for 4 seconds, and there is a one-second video prompt before the experiment starts. After the prompt disappears, the subject starts to perform the motor imagery task. Before the motor imagery starts, tags are added to the experimental data to record the current experimental action. After one motor imagery task ends, there is a three-second rest period, and then the next motor imagery task begins. To ensure the authenticity of the experiment, the occurrence order of the four actions is completely random.

[0063] 1.2 Data acquisition part: The data is acquired using a 64-channel NeuroScan device with a sampling frequency of 1000 Hz, and the electrode cap conforms to the international 10-20 standard. Among them, the 64 electrodes are FP1, FPZ, FP2, AF3, AF4, F7, F5, F3, F1, FZ, F2, F4, F6, F8, FT7, FC5, FC3, FC1, FCZ, FC2, FC4, FC6, FT8, T7, C5, C3, C1, CZ, C2, C4, C6, T8, TP7, CP5, CP3, CP1, CPZ, CP2, CP4, CP6, TP8, P7, P5, P3, P1, PZ, P2, P4, P6, P8, PO7, PO5, PO3, POZ, PO4, PO6, PO8, O1, OZ, O2, HEO, VEO, CB1, CB2. In the actual processing process, the channels with less influence, HEO, VEO, CB1, and CB2, are removed, and the remaining 60 effective channels are left. As Figure 3 shown.

[0064] Step S2: Preprocessing module: Perform data preprocessing on the EEG signals acquired in Step S1.2, and the processing steps include filtering, common average reference, and ICA decomposition.

[0065] The specific steps are as follows:

[0066] Step S2-1: Use a FIR filter for high-pass filtering to filter out the frequency band below 8 Hz; use a FIR filter for low-pass filtering to filter out the frequency band above 13 Hz.

[0067] Step S2-2: Use common average reference (CAR) to calculate the average value of the EEG data of each channel, and subtract the data of each channel from this average value.

[0068] Step S2-3: Removing electrooculogram and myoelectric signals: Using the ICA decomposition provided by the Python toolkit MNE, interfering components such as electrooculogram, electromyogram, and electrocardiogram are removed, and then recombined into effective signals.

[0069] Step S3: Feature extraction module:

[0070] All channel data of the data preprocessed in Step S2 are subjected to time-frequency decomposition using continuous wavelet transform to obtain effective frequency and effective time information. In the present invention, continuous wavelet transform based on Ricker wavelet is used, and the first-level data after decomposition replaces the original data to become feature data.

[0071] Among them, the continuous wavelet transform formula is as follows:

[0072]

[0073] Among them, CWT(s,τ) represents the output, s represents the scale, and τ represents the translation factor. represents the complex conjugate function, and x(t) represents the mother wavelet at time t. is the complex conjugate function of, <·> represents the inner product, x(t) represents the input data. represents the translation and scaling of the mother wavelet.

[0074] The Ricker wavelet formula is as follows:

[0075]

[0076] Among them, t is the time and f0 is the main frequency.

[0077] Step S4: Spatial transformation module:

[0078] In order to effectively retain the spatial features of each channel, in the present invention, the data after feature extraction in Step S3 are channel data with irregular distribution, which are copied, interpolated, and set to 0, so that the data set changes from two-dimensional to three-dimensional. Through this step, the data format is changed from N*L*trail to W*H*L*trail, where N is the number of channels, L is the data length of a single trail, trail is the total number of trials, W is the width of the three-dimensional data, and H is the height of the three-dimensional data. Specifically:

[0079] For channels that do not exist at the edge part, they are directly filled with 0. In the central part of the composition, the channel data of AF3, AF4, O1, and O2 are all copied once, and for the middle part data between AF3 and AF4, linear interpolation is used; for the data between POZ and PO3, PO4 in the middle part, linear interpolation is also used. As Figure 4 shown, Intp represents the linear interpolation result of the data of the left and right two channels.

[0080] Step S5 Classification and Recognition Model:

[0081] After the data is converted from two-dimensional data to three-dimensional data in Step S4, it is input into the classification and recognition model. The classification and recognition model is divided into three parts, which are responsible for channel dimension processing, time dimension processing, and classification and recognition respectively, as Figure 2 shown. Specifically:

[0082] 5.1 Channel Dimension Processing: Channel dimension processing includes two parts, namely the multi-head attention mechanism part and the 3D convolution part. The purpose is to extract spatial features and reduce the amount of data in the time dimension.

[0083] 5.1.1 Multi-Head Attention Mechanism Part: First, the data processed in 4.1 is transformed into two-dimensional data (C1, T), where C1 represents the total number of expanded channels and T represents the data length. Then, the multi-head attention mechanism is used to calculate the correlation between each channel, and then a dot product is performed with the input data. After that, the data is reshaped into three-dimensional data.

[0084] In this step, the attention mechanism is used to evaluate the importance of each channel with other channels. The attention mechanism is as follows:

[0085]

[0086] Q = XW Q

[0087] K = XW k

[0088] V = XW v

[0089] Among them, Attention(Q, K, V) represents the weight. Q, K, and V represent query, key, and value respectively, and X represents the input data. W Q , W k , W v are three trainable parameter matrices. represents the scaling factor, and Softmax is responsible for mapping the output data to the range from 0 to 1. In this step, d k is defined as 100, the size of the trainable parameter matrix is N×N, N represents the number of channels, and is set to 81. The output weight matrix is dot-producted with the input two-dimensional data after passing through the softmax function to obtain the weighted feature data of each channel. 5.1.2 The 3D convolution part contains two convolutional layers.

[0090] The convolutional kernel size of the first convolutional layer is 2×2×100, the stride is 1×1×100, the number of input convolutional kernels is 1, and the number of output convolutional kernels is 2. Batch normalization and the Relu non-linear activation function are applied after the first layer of convolution.

[0091] The convolutional kernel size of the second layer of convolution is 8×8×4, the stride size is 1×1×4, the number of input convolutional kernels is 2, and the number of output convolutional kernels is 10. In the third dimension, i.e., the time dimension, the moving stride and the window size are kept the same, aiming to reduce the amount of data in the time dimension and relieve the computational pressure.

[0092] The first layer of convolution is used to extract local features in the channel, and the second layer of convolution is used to extract global features in the spatial distribution. Through two layers of convolution, the time dimension slices are changed from a length of 4000 to a length of 10. Then the data is reshaped into (n×m), where n is the number of second-layer convolutional kernels, here it is 10, and m is the output length of the 3D convolution in the third dimension, i.e., the number of time dimension slices, here it is 10.

[0093] 5.2 Time dimension processing: The output of step S5.1 is input into the Transformer network for attention learning. The Transformer network consists of 3 identical encoders. Each encoder is composed of two parts: multi-head attention and a feed-forward neural network. The multi-head attention part uses a residual connection and internally consists of a normalization layer with a parameter of 10, a multi-head attention layer, and a 50% Dropout layer. The multi-head data size is consistent with m in the output of 5.1. The feed-forward neural network part also uses a residual connection and internally includes a normalization layer with a parameter of 10, a feed-forward neural network, and a 50% Dropout part.

[0094] 5.2.1 The multi-head attention mechanism, the principle part is similar to 5.1.1. Here d k is defined as 10, and the size of the trainable parameter matrix is head×head, where head represents the number of heads and is set to 10.

[0095] 5.2.2 The first part of the feed-forward neural network is a fully connected layer with parameters (10,5), the second part is the GELU activation function, the third part is a 50% Dropout layer, and the fourth part is a fully connected layer with parameters (10,5).

[0096] 5.3 Classification and Recognition: The output result of step S5.2 is composed of a Reduce layer provided by the einops toolkit, a batch normalization layer, and a fully connected layer. The Reduce layer is responsible for taking the mean of the two-dimensional data output by the Transformer in the first dimension, changing from (n×m) to (1×m). The parameter of the batch normalization layer is 10, and the parameters of the fully connected layer are (head, class). Here, head represents the number of heads in the multi-head attention network, which is 10; class represents the task category, which is 4 here.

[0097] Step S6: Visualization Feedback Module: The visualization interface is as Figure 5 shown. In this module, there are four motor imagery tasks, which are used to identify the model prediction results of the user in the real-time system and flash the corresponding pictures of the currently recognized tasks. There are four output boxes, which are used to display the number of times the four states have appeared / the number of correct identifications and the correct rate; above the interface, there is a current system time and the time that the current experiment has been carried out to display the experiment progress.

Claims

1. A decoding method for complex action motor imagery based on electroencephalogram, characterized by comprising: Step (1), acquisition of electroencephalogram signals of complex action motor imagery tasks 1.1 The experimental paradigm is unilateral upper limb multi-action motor imagery, which includes: imagining shoulder flexion, imagining elbow flexion, imagining fist clenching, and a static state; a single task is divided into two parts: a video prompt part and a task imagery part; The video prompt part requires the subject to be in a calm state without tasks within t1, maintaining a steady breathing, not blinking, and having no other stimuli except the prompt; t1 > 0 seconds; The task imagery part requires the subject to keep the body still within t2, without limb movement, without distracting thoughts, and maintaining a steady breathing and not blinking; t2 > 0 seconds; 1.2 Collect the electroencephalogram (EEG) signals of the subject within t2 after the prompt ends; Step (2): Preprocess the EEG signals collected in step (1) to obtain the preprocessed motor imagery key frequency band alpha (8 - 13Hz) EEG signal data; Step (3): Extract features from the data preprocessed in step (2); Use continuous wavelet transform to decompose the preprocessed signal, and use the first-level decomposed data to replace the original data as feature data; Step (4): Perform spatial transformation on the feature data extracted in step (3). According to the physical positions of the electrodes, use the methods of interpolation, replication, and setting to 0 to transform the two-dimensional data obtained in step (3) into three-dimensional data, so as to retain the spatial position information of each electrode; Step (5): Construct a classification and recognition model, including a channel dimension part, a time dimension part, and a classifier part; the channel dimension part is used to extract spatial features and reduce the data volume in the time dimension; the time dimension part is used to input the output of the channel dimension part into the Transformer network for attention learning; the classifier part classifies the output results of the time dimension part; the channel dimension part includes a multi-head attention mechanism part and a 3D convolution part; The multi-head attention mechanism part is used to identify the importance of each channel; The 3D convolution part contains two convolutional layers: the first convolutional layer has a convolutional kernel size of 2×2×100 and a stride of 1×1×100. After the first layer of convolution, batch normalization and the ReLU non-linear activation function are applied; the second convolutional kernel size is 8×8×4, and the stride size is 1×1×4. The number of convolutional kernels in the two layers is 2 and 10 respectively.

2. The method for decoding complex movement imagery based on electroencephalogram according to claim 1, wherein In step (1.2), the sampling frequency is 1000Hz, and the acquisition device is a 64-channel NeuroScan that conforms to the 10 - 20 standard.

3. A method for decoding complex movement imagery based on electroencephalogram according to claim 1, characterized in that The EEG signal preprocessing in step (2) includes filtering, common average reference, and ICA decomposition to remove electrooculogram, electromyogram, and electrocardiogram signals.

4. A method for decoding complex movement imagery based on electroencephalogram according to claim 1, characterized in that The continuous wavelet transform basis function is the Ricker wavelet; The Ricker wavelet transform formula is as follows: where t is time and f0 is the main frequency.

5. A method for decoding complex action motor imagery based on electroencephalogram according to claim 1, characterized in that For channels where the edge part does not exist, directly fill with 0. In the central part of the composition, the data of the AF3, AF4, O1, and O2 channels are all copied once, and for the data in the middle part between AF3 and AF4, linear interpolation is used; for the data in the middle part between POZ and PO3, PO4, the same linear interpolation is used.

6. The method for decoding complex movement imagery based on electroencephalogram according to claim 1, wherein In the classification and recognition model of the above steps, the Transformer network in the time dimension part includes 3 identical attention layers, and each layer is composed of a 10-head attention layer and a feed-forward neural network layer.

7. A real-time decoding system for complex motor imagery based on electroencephalogram for implementing the method according to any one of claims 1-6, characterized in that The system includes: a data acquisition module, a preprocessing module, a feature extraction module, a spatial transformation module, a classifier module, and a visualization feedback module; a data acquisition module for acquiring electroencephalogram signals and true labels during the motor imagery of the subject; a preprocessing module for preprocessing the original electroencephalogram signals acquired by the data acquisition module; a feature extraction module for performing effective feature transformation on the data processed by the preprocessing module; a spatial transformation module, according to the physical positions of the electrodes, by means of interpolation, replication, and setting to zero, transforms the data extracted by the feature extraction module from a two-dimensional space into a three-dimensional space; a classifier module for predicting and classifying the motor imagery data of the subject, and the classification model includes a channel dimension part, a time dimension part, and a classifier part; a visualization feedback module for feeding back the recognition result of the current task to the subject and presenting the status of each task of the current experiment in real time.

8. A computer-readable storage medium, on which a computer program is stored. When the computer program is executed in a computer, the computer is made to execute the method according to any one of claims 1-6.

9. A computing device, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method according to any one of claims 1-6 is implemented.

Images