Decoding method, device and storage medium of light-weight motor imagination signal

Abstract

The application discloses a kind of decoding methods for lightweight motor imagery signal, first, the MI-EEG signal data to be classified is preprocessed;Then, the preprocessed MI-EEG signal data to be classified is input into decoding model, and decoding model outputs the category of the MI-EEG signal data to be classified;The decoding model is sequentially connected by time domain smoothing module, channel fusion extraction module, multi-scale classification module and classification module;The training process of the classification model is that the MI-EEG signal data in training data set is used as input, and the known category information corresponding to the MI-EEG signal data is used as theoretical output, and the process of constantly adjusting model parameters.The lightweight method can greatly improve the accuracy of MI decoding model, and has good application prospect.

Description

Technical Field

[0001] This invention belongs to the field of brain-computer interfaces, specifically relating to a method, device, and storage medium for decoding lightweight motor imagery signals. Background Technology

[0002] Currently, the application of brain-computer interface (BCI) systems based on the motor imagery (MI) paradigm is mainly focused on the medical field, providing a new control method for patients with movement disorders, helping them achieve self-care in daily life and even complete rehabilitation. These patients with movement disorders include those with brainstem injury, stroke, muscle atrophy, etc. They have intact brains but are unable to control their bodies due to damage to peripheral nerves. MI-based BCI systems (MI-BCI systems) can enable control of external devices, replacing trunk movements. In addition, the combination of MI and physical therapy is an effective method for stroke rehabilitation.

[0003] To date, many studies have attempted to manually extract features and then train machine learning-based classifiers to recognize MI signals. Among these, researchers have proposed a spatial feature extraction method called Common Spatial Pattern (CSP), which significantly improves the classification performance of MI signals (MI-EEG signals) based on electroencephalograms (EEG). The CSP method is suitable for binary classification; its basic idea is to find a spatial filter that maximizes the variation between the two classes by solving a matrix factorization problem. Due to its stable performance, researchers have proposed many improved versions of CSP: for example, the Common Space Spectral Pattern (CSSP), which extends CSP to the state space and improves the classification accuracy of CSP-based MI-EEG signals; and the Regularized Space Spectral Pattern (RCSP), which achieves nearly 10% higher median classification accuracy than CSP and generates more spatial filters relevant to neurophysiology. However, the classification accuracy of these traditional methods largely depends on hand-crafted features, which severely limits the model's generalization ability. Therefore, a new algorithm and framework are needed to address this issue.

[0004] Deep Neural Networks (DNNs), as a subclass of machine learning, have made remarkable progress in computer vision and natural language processing. As an end-to-end model, DNNs are well-suited for MI-EEG signal classification. Classic DNN-based decoding models include DeepConvNet, ShllowConvNet, EEGNet, and MWFFCNN. These models all belong to convolutional neural networks. However, traditional DNNs cannot fully extract features from MI-EEG signals across multiple scales, resulting in poor MI decoding accuracy. Furthermore, MI-EEG signal data is typically limited, and since DNNs usually require more training data than other machine learning methods, they may encounter severe overfitting problems.

[0005] Therefore, developing a method that can achieve high-precision and accurate classification of MI-EEG signals is of great practical significance. Summary of the Invention

[0006] The technical problem to be solved by the present invention is to provide a lightweight method, device and storage medium for decoding motion image signals, which solves the problems of poor accuracy of existing MI decoding methods and severe overfitting of existing DNNs in EEG-based MI classification.

[0007] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0008] A method for decoding lightweight motion imagery signals includes the following steps:

[0009] Step 1: Obtain the laboratory dataset and preprocess it, dividing the laboratory dataset into training dataset and test dataset;

[0010] Step 2: Construct a classification model, which includes a temporal smoothing module, a channel fusion module, a multi-scale feature extraction module, and a classification module. The temporal smoothing module performs low-pass filtering on the MI-EEG signal; the channel fusion module selectively fuses the spatial information of each MI-EEG feature map; the multi-scale feature extraction module obtains the temporal features of the MI-EEG feature map at different scales; and the classification module further extracts the deep features of the MI-EEG feature map.

[0011] Step 3: Build and train the decoding model and the comparison model, and compare their decoding performance;

[0012] Step 4: Conduct ablation experiments on the model to verify the effectiveness of its multi-scale feature extraction module and the impact of the number of multi-scale temporal convolution modules on the model's classification accuracy.

[0013] The specific process of step 1 is as follows:

[0014] First, we acquired a laboratory dataset by conducting two experiments on several subjects, each performing a motor imagery task, and collecting a laboratory dataset of motor imagery signals from the subjects based on electroencephalograms.

[0015] Secondly, bandpass filtering was applied to the laboratory dataset to suppress noise;

[0016] Then, the filtered laboratory dataset is standardized;

[0017] Finally, the laboratory dataset was divided into training and testing datasets according to the proportions of the standard dataset.

[0018] The following formula is used to standardize the EEG-based motor imagery signals:

[0019]

[0020] Where, x k and Let μ and σ represent the filtered and normalized signals at time k, respectively; μ represents the average value of the filtered signal, and σ represents the standard deviation of the filtered signal.

[0021] The temporal smoothing module consists of a cascaded temporal convolution module and a batch normalization module. This module filters the EEG-based motor imagery signal to generate a feature map of the EEG-based motor imagery signal.

[0022] The channel fusion module applies deep convolution to perform channel fusion on the feature maps and extracts spatial information from each feature map.

[0023] The multi-scale feature extraction module includes two large-scale temporal convolution modules, two clock-scale temporal convolution modules, and two small-scale temporal convolution modules. It first extracts temporal information at different scales from the feature map using convolution kernels of different scales along the time dimension, and then compares it with the original feature map. Figure 1 They are spliced ​​together to generate a multi-scale feature map of motion imagery signal.

[0024] The classification module consists of two cascaded temporal convolutional modules with the same structure. The first temporal convolutional module comprises a temporal convolution, a max pooling, a BatchNorm module, an ELU activation function, and a Dropout module. The second temporal convolutional module differs from the first in that it replaces max pooling with average pooling. The output of the second temporal convolutional module is flattened, fully connected, and then classified using the Softmax function.

[0025] The training of the decoding model includes two processes: pre-training and fine-tuning. In the pre-training process, the datasets of all subjects are merged together to train the model and obtain a pre-trained model, and its weight parameters are saved. In the fine-tuning process, the dataset of any one subject is used to fine-tune the pre-trained model to obtain the final model for that subject.

[0026] A lightweight device for decoding motion imagery signals includes at least one processor and a data acquisition unit in communication therewith, wherein the processor applies the method to process a laboratory dataset of subjects acquired by the data acquisition unit to obtain classification results of motion imagery signals.

[0027] A computer-readable storage medium storing computer-readable instructions that, when executed by a processor, invoke all or part of the steps of the method.

[0028] Compared with the prior art, the present invention has the following beneficial effects:

[0029] 1. The motion imagery signal classification method of the present invention utilizes convolutional kernels of different sizes and lengths to fully extract the features of MI-EEG signals at multiple scales when constructing a lightweight decoding model, thereby enriching the information of the MI-EEG feature map;

[0030] 2. In the classification method of motion-imagined signals of the present invention, the temporal smoothing module in the decoding model uses a large-scale convolution kernel to perform low-pass filtering on the MI-EEG signal, making the extracted preliminary features more effective;

[0031] 3. The motion imagery signal classification method of the present invention uses a channel fusion module in the decoding model to selectively fuse the spatial information of each MI-EEG feature map using deep convolution, thereby reducing the risk of overfitting;

[0032] 4. The classification method for motion imagery signals of the present invention further extracts deep features of the MI-EEG feature map in the classification module of the decoding model, which can greatly improve the accuracy of the classification model and has great application prospects. Attached Figure Description

[0033] Figure 1 This is a flowchart illustrating the classification method for motion imagery signals in Example 1.

[0034] Figure 2 A structural diagram of the classification model for motion imagination signals constructed in Example 1;

[0035] Figure 3 A structural diagram of the time-domain smoothing module constructed in Example 1;

[0036] Figure 4A structural diagram of the channel fusion module constructed in Example 1;

[0037] Figure 5 The structure diagram of the multi-scale feature extraction module constructed in Example 1; Detailed Implementation

[0038] The structure and working process of the present invention will be further described below with reference to the accompanying drawings.

[0039] A method for decoding lightweight motion imagery signals includes the following steps:

[0040] Step 1: Obtain the laboratory dataset and preprocess it, dividing the laboratory dataset into training dataset and test dataset;

[0041] Step 2: Construct a classification model, which includes a temporal smoothing module, a channel fusion module, a multi-scale feature extraction module, and a classification module. The temporal smoothing module performs low-pass filtering on the MI-EEG signal; the channel fusion module selectively fuses the spatial information of each MI-EEG feature map; the multi-scale feature extraction module obtains the temporal features of the MI-EEG feature map at different scales; and the classification module further extracts the deep features of the MI-EEG feature map.

[0042] Step 3: Build and train the decoding model and the comparison model, and compare their decoding performance;

[0043] Step 4: Conduct ablation experiments on the model to verify the effectiveness of its multi-scale feature extraction module and the impact of the number of multi-scale temporal convolution modules on the model's classification accuracy.

[0044] Specific embodiments, such as Figures 1 to 5 As shown,

[0045] The decoding method for lightweight motion imagery signals is implemented as follows:

[0046] (1) Obtaining the training dataset:

[0047] Step 1: Collect data from subjects to form a laboratory dataset, and preprocess the laboratory dataset:

[0048] 1.1: Acquisition of the laboratory dataset. The acquisition method for the laboratory dataset was exactly the same as that for the standard dataset BCICompetition IV-2a. The dataset used 22 Ag / AgCl electrodes to record two experiments for each subject, for a total of 9 subjects. For each subject, the two experiments were conducted over two days. During the experiments, subjects sat comfortably in front of the screen. At the start of each experiment, a crosshair with a short warning tone appeared on the black screen. After 2 seconds, an arrow pointing left, right, down, or up (corresponding to left hand, right hand, foot, or tongue) appeared, indicating the execution of the motor imagery task. Subjects were asked to perform the MI task from t=2s to t=6s. Each topic contained 72 trials, for a total of 288 trials obtained from each subject per topic. For the i-th subject, the EEG signal used in each trial can be represented as... Where C is the number of channels and T is the number of data points;

[0049] 1.2: Preprocessing the laboratory dataset:

[0050] 1.2.1: Bandpass filtering is performed on the laboratory dataset. The preprocessing method for the laboratory dataset is exactly the same as that for the standard dataset. The laboratory dataset samples the signal at 250Hz and performs bandpass filtering between 4Hz and 32Hz, and then uses a 50Hz notch filter to suppress noise.

[0051] 1.2.2: Standardization of the laboratory dataset. To eliminate the impact of interference and noise in the laboratory dataset on MI decoding, it is necessary to standardize the filtered MI-EEG signal as follows:

[0052]

[0053] Where x k and Let μ and σ represent the filtered and normalized MI-EEG signals at time k, respectively. μ represents the average value of the filtered MI-EEG signal, and σ represents the standard deviation of the filtered MI-EEG signal.

[0054] 1.2.3: Dataset Splitting. The laboratory dataset is divided into training and testing datasets according to the proportions of the standard dataset.

[0055] (2) Construction of the classification model:

[0056] The overall structure of the framework is as follows Figure 2As shown, the framework consists of four parts: a temporal smoothing module, a channel fusion module, a multi-scale feature extraction module, and a classification module. The temporal smoothing module performs low-pass filtering on the MI-EEG signal; the channel fusion module selectively fuses the spatial information of each MI-EEG feature map; the multi-scale feature extraction module obtains the temporal features of the MI-EEG feature map at different scales; and the classification module further extracts the deep features of the MI-EEG feature map, greatly improving the accuracy of the decoding model.

[0057] Step 2.1, Construct the temporal smoothing module:

[0058] Original MI-EEG signal First, the feature map is filtered using a temporal smoothing module. This module consists of a cascaded temporal convolution and a batch normalization module, generating the MI-EEG feature map. The reason why the time-domain smoothing module does not use the ELU activation function is that this module only filters the original MI-EEG signal and does not need to perform nonlinear processing.

[0059] Step 2.2, Construct the channel fusion module:

[0060] The channel fusion module utilizes depthwise convolution to process the MI feature map generated by the temporal smoothing module. Channel fusion is performed, and spatial information is extracted from each MI feature map. Furthermore, compared to ordinary convolution, depthwise convolution can further reduce the number of parameters and alleviate overfitting. This process generates new MI feature maps.

[0061] Step 2.3, Construct a multi-scale feature extraction module:

[0062] The multi-scale feature extraction module first uses convolutional kernels of different scales in the time dimension to process the MI feature map generated by the channel fusion module. First, extract time information at different scales, then compare it with the original features. Figure 1 The components are then stitched together to generate MI multi-scale feature maps. The specific process of the multi-scale feature extraction module extracting temporal information at different scales is shown below:

[0063]

[0064]

[0065]

[0066]

[0067]

[0068]

[0069]

[0070] This process utilizes two large-scale temporal convolution (LT-Conv(·)) modules, two medium-scale temporal convolution (MT-Conv(·)) modules, and two small-scale temporal convolution (ST-Conv(·)) modules. Taking the large-scale temporal convolution module as an example, the MI feature map generated by the input channel fusion module of the first large-scale temporal convolution module... and output it. This serves as the input to the subsequent large-scale temporal convolutional module. Furthermore, to enrich the obtained temporal information, the spatial distance between the convolutional kernels of the preceding and following large-scale temporal convolutional modules is different. Similarly, the settings for other temporal convolutional blocks are similar. The kernel sizes for the three temporal convolutional modules are 1×25, 1×15, and 1×5, respectively. Each module has a structure consisting of one temporal convolution, one batch normalization module, one ELU activation function, and one Dropout module.

[0071] Step 2.4, construct the classification module:

[0072] The classification module consists of two cascaded temporal convolutional modules with identical structures. The first temporal convolutional module comprises one temporal convolution, one max pooling, one BatchNorm module, one ELU activation function, and one Dropout module. The second temporal convolutional module is similar to the first, except that the max pooling is replaced with average pooling. Finally, the output of the second temporal convolutional block is flattened, fully connected, and then classified using the Softmax function.

[0073] (3) Train the decoding model and comparison model, and compare their decoding performance (DeepConvNet, ShallowConvNet, EEGNet, MWFFCNN):

[0074] Table 1 Comparison of Model Classification Accuracy

[0075]

[0076] To better compare the improvement in MI-EEG signal decoding performance of this patented model (MSCNN refers to this patented model), this study tried to maintain consistency with the training methods of the four models in Chapter 3. The model parameters were set as follows: Epoch set to 300, Batch_size set to 16, learning rate set to a certain value, cross-entropy loss function selected, and Adam optimizer selected. After training, the model was tested, and the results are shown in Table 1 (bold indicates the best classification result). The test results show that the MSCNN model achieved an average classification rate of 72.61%, far exceeding the MI decoding performance of the four models mentioned above. Furthermore, the MSCNN model achieved a classification accuracy exceeding 80% on four subjects (A03, A07, A08, and A09). To demonstrate the effectiveness of the multi-scale feature extraction module, this patented model ablation experiments were conducted.

[0077] To further improve the model's accuracy, this study employed two processes: pre-training and fine-tuning. During pre-training, based on the idea of ​​transfer learning, the training sets of nine subjects were combined to train the model, resulting in a pre-trained model whose weight parameters were saved. The pre-trained model demonstrated strong adaptability and generalization ability. Then, during fine-tuning, the pre-trained model was fine-tuned using only the training set of one subject, thus obtaining the final model for that subject.

[0078] Table 2 Comparison of MSCNN model accuracy with and without pre-training.

[0079]

[0080] For the pre-training phase, the model parameters are set as follows: Epoch is set to 200, Batch_size is set to 64, and learning rate is set to 10. -3 The loss function chosen was the cross-entropy loss function, and the optimizer was the Adam optimizer. The larger batch size during the pre-training phase was chosen because the training data from 9 subjects significantly increases compared to a 1-subject dataset. For the fine-tuning phase, the model parameters were set as follows: Epoch set to 100, Batch size set to 12, learning rate set to a certain value, cross-entropy loss function chosen, and Adam optimizer chosen. This ensures that the sum of the Epochs during the pre-training and fine-tuning phases is the same as the total training Epochs of the model. After training, the model was tested, and the results are shown in Table 4-3 (bold indicates the best classification result). The test results show that the "pre-training + fine-tuning" training strategy can improve the average classification accuracy of the MSCNN model by approximately 2%.

[0081] (4) Ablation test

[0082] To demonstrate the effectiveness of the multi-scale feature extraction module and further investigate the impact of the number of multi-scale temporal convolution modules on the model's classification accuracy, this study designed the following comparative model:

[0083] Model_0: No multi-scale feature extraction module;

[0084] Model_1: Consists only of small-scale temporal convolution (1×5) modules;

[0085] Model_2 consists of a small-scale temporal convolution (1×5) module and a medium-scale temporal convolution (1×15) module;

[0086] Model_3 consists of a small-scale temporal convolution (1×5) module, a medium-scale temporal convolution (1×15) module, and a large-scale temporal convolution (1×25) module.

[0087] Model_4 consists of a small-scale temporal convolution (1×5) module, a medium-scale temporal convolution (1×15) module, a large-scale temporal convolution (1×25) module, and an ultra-large-scale temporal convolution (1×35) module.

[0088] The relevant parameters for each model were set as follows: Epoch was set to 300, Batch_size was set to 16, learning rate was set to a certain value, cross-entropy loss function was selected as the loss function, and Adam optimizer was selected as the optimizer. After training was completed, the model was tested, and the test results are shown in Table 3 (bold indicates the best classification result).

[0089] Table 3. Results of the model ablation experiment

[0090]

[0091] A lightweight device for decoding motion imagery signals includes at least one processor and a data acquisition unit in communication therewith, wherein the processor applies the method to process a laboratory dataset of subjects acquired by the data acquisition unit to obtain classification results of motion imagery signals.

[0092] A computer-readable storage medium storing computer-readable instructions that, when executed by a processor, invoke all or part of the steps of the method.

[0093] Those skilled in the art should understand that variations can be implemented by combining existing technology with the above embodiments, which will not be elaborated here. Such variations do not affect the essence of the present invention, and will not be elaborated here either.

[0094] The preferred embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and the devices and structures not described in detail should be understood as being implemented in a conventional manner in the art. Any person skilled in the art can make many possible variations and modifications to the technical solutions of the present invention using the methods and techniques disclosed above, or modify them into equivalent embodiments with equivalent changes, without departing from the scope of the present invention. This does not affect the essential content of the present invention. Therefore, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the present invention's technical solutions still fall within the protection scope of the present invention.

Claims

1. A method for decoding lightweight motion imagery signals, characterized in that: Includes the following steps: Step 1: Obtain the laboratory dataset and preprocess it, dividing the laboratory dataset into training dataset and test dataset; Step 2: Construct a classification model, which includes a temporal smoothing module, a channel fusion module, a multi-scale feature extraction module, and a classification module. The temporal smoothing module performs low-pass filtering on the MI-EEG signal; the channel fusion module selectively fuses the spatial information of each MI-EEG feature map; the multi-scale feature extraction module obtains temporal features at different scales of the MI-EEG feature map; and the classification module further extracts deep features from the MI-EEG feature map. The multi-scale feature extraction module includes two large-scale temporal convolution modules, two medium-scale temporal convolution modules, and two small-scale temporal convolution modules; utilizing different scales in the time dimension... The convolutional kernel first extracts temporal information at different scales from the feature map, and then concatenates it with the original feature map to generate a multi-scale feature map of motion imagery signal. The classification module consists of two cascaded temporal convolutional modules with the same structure. The first temporal convolutional module consists of one temporal convolution, one max pooling, one BatchNorm module, one ELU activation function, and one Dropout module. The difference between the second and first temporal convolutional modules is that the max pooling is replaced with average pooling. The output of the second temporal convolutional module is flattened, fully connected, and then classified using the Softmax function. Step 3: Build and train the decoding model and the comparison model, and compare their decoding performance; Step 4: Conduct ablation experiments on the model to verify the effectiveness of its multi-scale feature extraction module and the impact of the number of multi-scale temporal convolution modules on the model's classification accuracy.

2. The method for decoding lightweight motion imagery signals according to claim 1, characterized in that: The specific process of step 1 is as follows: First, we acquired a laboratory dataset by conducting two experiments on several subjects, each performing a motor imagery task, and collecting a laboratory dataset of motor imagery signals from the subjects based on electroencephalograms. Secondly, bandpass filtering was applied to the laboratory dataset to suppress noise; Then, the filtered laboratory dataset is standardized; Finally, the laboratory dataset was divided into training and testing datasets according to the proportions of the standard dataset.

3. The method for decoding lightweight motion imagery signals according to claim 2, characterized in that: The following formula is used to standardize the EEG-based motor imagery signals: in, and They represent time. The filtered and standardized signal at the point; This represents the average value of the filtered signal. This represents the standard deviation of the filtered signal.

4. The method for decoding lightweight motion imagery signals according to claim 1, characterized in that: The temporal smoothing module consists of a cascaded temporal convolution module and a batch normalization module. This module filters the EEG-based motor imagery signal to generate a feature map of the EEG-based motor imagery signal.

5. The method for decoding lightweight motion imagery signals according to claim 4, characterized in that: The channel fusion module applies deep convolution to perform channel fusion on the feature maps and extracts spatial information from each feature map.

6. The method for decoding lightweight motion imagery signals according to claim 1, characterized in that: The training of the decoding model includes two processes: pre-training and fine-tuning. In the pre-training process, the datasets of all subjects are merged together to train the model and obtain a pre-trained model, and its weight parameters are saved. In the fine-tuning process, the dataset of any one subject is used to fine-tune the pre-trained model to obtain the final model for that subject.

7. A lightweight decoding device for motion image signals, characterized in that: It includes at least one processor and a data acquisition unit communicating therewith, the processor applying the method of any one of claims 1 to 6 to process a subject laboratory dataset acquired by the data acquisition unit to obtain a classification result of the motor imagery signal.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions that, when executed by a processor, invoke all or part of the steps of the method according to any one of claims 1 to 6.

Images