Cross-period electroencephalogram motor imagery classification method based on transfer learning

Abstract

The invention discloses a transfer learning-based cross-period electroencephalogram motor imagery classification method, and particularly relates to the field of motor brain-computer interfaces, and the method comprises the steps: S1, obtaining source domain and target domain electroencephalogram data through data preprocessing; s2, extracting frequency, space and time information by using an attention-based separable convolutional neural network; s3, utilizing supervised learning to minimize classification loss of source domain sample data; s4, maximizing similarity between a source domain and target and sample features by utilizing transfer learning so as to train a neural network suitable for cross-period electroencephalogram; and S5, classifying a target domain by using the trained model, and comparing motor imagery classification effects on a public data set with several selected common advanced algorithms. And it is verified that the cross-time problem generally exists in the motion brain-computer interface. According to the method provided by the invention, a scientific and reliable motor imagery classification scheme is provided, and a motor imagery classification model with higher accuracy than several selected common advanced algorithms is constructed.

Description

Technical Field

[0001] The present invention relates to the field of motor brain-computer interface, and specifically refers to a cross-time EEG motor imagery classification method based on transfer learning. Background Art

[0002] Motor Brain-Computer Interface (BCI) is an advanced technology that achieves motion control by directly connecting the brain to external devices. It collects the brain's electrical activity, decodes these signals and converts them into control instructions, thereby achieving direct control of external devices (such as robotic arms, wheelchairs, or computer cursors). The core of motor BCI is to bypass the traditional neuromuscular pathways and directly use brain signals (such as electroencephalogram (EEG), electrocorticogram (ECoG), and single neuron recordings) to achieve motion control. Motor brain-computer interface has broad application prospects in many fields. First, in rehabilitation medicine, motor BCI can help patients with paraplegia, hemiplegia, and other movement disorders recover some motor functions. Secondly, in neuroscience research, motor BCI provides a new means to study the brain's motor control mechanism. By analyzing the brain's motor signals, scientists can have a deeper understanding of the brain's function and structure. In addition, in augmented reality (AR) and virtual reality (VR), motor BCI can be used for more natural and intuitive human-computer interaction. Using EEG to classify motor imagery is reasonable and scientific, which is conducive to building a more accurate and reliable motor imagery classification model. It is also conducive to leveraging the advantages of artificial intelligence technology and promoting the common development and prosperity of interdisciplinary disciplines.

[0003] Convolutional neural network (CNN) originated in the 1980s, was proposed by Yann LeCun et al., and was successfully applied to handwritten digit recognition in 1998. The goal of CNN is to extract features in two-dimensional image data, capture local features in the image through convolution operations, and reduce the number of parameters and computational complexity. Its main components include convolutional layers, pooling layers, and fully connected layers. The convolutional layer extracts local features through the convolution kernel sliding window, the pooling layer reduces the size of the feature map, improves the robustness of the model, and the fully connected layer is used for the final classification or regression task. CNN performs well in tasks such as image classification, target detection, and image segmentation. With the development of deep learning, the structure continues to evolve, such as ResNet and DenseNet, which solve the gradient vanishing problem in deep networks by introducing residual connections and dense connections, and significantly improves model performance. In short, convolutional neural networks combine the advantages of convolution operations and neural networks, provide powerful tools for processing image data, and are widely used in various visual recognition tasks. In the present invention, the convolutional neural network fits the spatial topological structure of EEG, and the frequency, space, and time characteristics of EEG can be extracted.

[0004] The attention mechanism is a deep learning technology that was originally introduced in machine translation tasks to enhance the performance of neural networks when processing sequence data. The main idea is to focus on important information by assigning different weights and ignore irrelevant information, thereby improving the efficiency and accuracy of the model. The attention mechanism calculates the importance score of each position in the input sequence and dynamically adjusts the influence of each position, so that the model can more effectively capture global and local dependencies. This mechanism has achieved remarkable success in fields such as natural language processing, computer vision, and speech recognition, especially in improving the processing power of long sequence data and the interpretability of models. In the present invention, the introduction of the attention mechanism focuses on the important information of spatial and temporal features in EEG, significantly improving the recognition accuracy.

[0005] Transfer learning is a deep learning method used to apply knowledge learned from one field to another related but different field, thereby achieving efficient feature extraction and classification. The background technology of transfer learning comes from the analogy of human learning, that is, when people learn new tasks, they often use previous knowledge and experience to speed up the learning process of new tasks. The core idea of ​​transfer learning is to use a model pre-trained on a large dataset and transfer its knowledge to a smaller target dataset for fine-tuning to achieve good performance. This method not only reduces training time, but also improves the generalization ability of the model on small sample datasets. The following is the general workflow of transfer learning:

[0006] 1. Pre-trained models: A deep neural network model is trained on a large-scale source domain dataset, which usually involves a large amount of labeled data and computing resources. These models are trained on large datasets to learn common feature representations.

[0007] 2. Feature extraction: Fix the first few layers of the pre-trained model, which have learned common low-level features, and use them as feature extractors. This step can be regarded as freezing the weights of the model so that they remain unchanged during subsequent training.

[0008] 3. Model fine-tuning: Fine-tune the latter layers of the model on the target dataset so that it can learn new task-related feature representations. The fine-tuning process usually uses a smaller learning rate to prevent the destruction of existing knowledge in the pre-trained model. This step requires unfreezing some frozen weights and retraining the model on new data to adapt to new task requirements.

[0009] 4. Model training: Continue to train the model on the target domain dataset so that it can achieve good performance on new tasks. Generally speaking, the target domain dataset is unlabeled data or small sample data. During the training process, the model gradually learns the high-level features required for the target task and optimizes the model parameters.

[0010] 5. Application to downstream tasks: Once the model is trained, it can be applied to various downstream tasks such as image classification, object detection, semantic segmentation, sentiment analysis, etc. Usually, these feature representations obtained by transfer learning can provide excellent performance in downstream tasks without the need for a large amount of labeled data.

[0011] Transfer learning has made important breakthroughs in computer vision, natural language processing, and recommendation systems, especially in scenarios where data is scarce or labeling is expensive, helping to solve the problems of traditional deep learning, such as long training time, poor generalization ability, and difficulty in cross-domain application. Therefore, the use of transfer learning alleviates the problem of expensive labeled EEG data that is difficult to obtain in small sample scenarios, and only requires fine-tuning of unlabeled data under specific tasks to achieve better recognition results. Summary of the invention

[0012] The technical problem to be solved by the present invention is that, in view of the problem of motor imagery recognition in motor brain-computer interaction, and the problem that EEG data labels are difficult to label and labor-intensive, the present invention provides a cross-time EEG motor imagery classification method based on transfer learning.

[0013] In order to solve the above technical problems, the present invention provides a technical solution: a cross-period EEG motor imagery classification method based on transfer learning, comprising the following specific steps:

[0014] S1. Use data preprocessing to obtain source domain and target domain EEG data;

[0015] S2. Extract frequency, spatial and temporal information using attention-based separable convolutional neural networks;

[0016] S3. Use supervised learning to minimize the classification loss of source domain sample data;

[0017] S4. Use transfer learning to maximize the similarity between the source domain and the target domain and sample features to train a neural network suitable for EEG across time periods;

[0018] S5. Use the pre-trained model to classify the target domain, and compare the motor imagery classification effect on the public dataset with several selected commonly used advanced algorithms. And verify that the cross-time problem is common in motor brain-computer interfaces.

[0019] Furthermore, step S1 specifically includes:

[0020] S11. The original EEG information is segmented and spliced ​​according to preset time points to obtain two sets of data, the source domain and the target domain.

[0021] Furthermore, step S2 specifically includes:

[0022] S21. Use two-dimensional convolution to extract frequency features from the original EEG data, and then perform batch normalization;

[0023] S22. Convolve the feature map output by step S21 using deep convolution to extract spatial features;

[0024] S23. Use the attention mechanism to highlight the most important channel information on the feature map output from step S22. The attention mechanism formula is:

[0025] Attention=tanh[softmax(wM)·H] (1)

[0026] Where H is the input feature, w is a trainable matrix, M = tanh(H);

[0027] S24. The output feature map of step S23 is batch normalized, then passed through the Elu nonlinear activation function, then through the pooling layer for dimensionality reduction, and finally Dropout to prevent overfitting;

[0028] S25. Convolve the feature map outputted from step S24 using separable convolution, first extract the temporal features using two-dimensional convolution, and then optimally combine the feature maps using point convolution;

[0029] S26. The feature map outputted in step S25 is batch normalized, then subjected to the Elu nonlinear activation function, then subjected to the pooling layer for dimensionality reduction, and finally subjected to Dropout to prevent overfitting;

[0030] S27. Use the multi-head self-attention mechanism to highlight the most important feature information of the feature map output in step S26. The multi-head self-attention formula is:

[0031]

[0032] The query vector Q, key vector K, and value vector V are obtained by passing the input vector X through the fully connected layer. is the feature dimension.

[0033] Furthermore, step S3 specifically includes:

[0034] S31. Use the source domain prediction results and source domain data labels for supervised learning, and use the cross-entropy loss function as the classification loss. The loss function formula is:

[0035]

[0036] Where N is the number of classification categories; y ic is a sign function (0 or 1), if the true category of sample i is equal to c, it takes 1, otherwise it takes 0; pic is the predicted probability that the observed sample i belongs to category c.

[0037] Furthermore, step S4 specifically includes:

[0038] S41. An adaptive layer Bottlenck block consisting of a linear layer and a Relu nonlinear activation function is used to perform transfer learning on the source domain and target domain feature maps output by step S2;

[0039] S42. Calculate the loss between the source domain and the target domain features output by the adaptive layer, maximize the similarity between the source domain and the target and sample features, and use the Earth-Mover Distance as the migration loss. The loss function formula is:

[0040] EMD_loss=inf γ＝∏(p,q) E (x,y)～γ [||xy||] (4)

[0041] Where ∏(p,q) represents the set of all possible joint distributions of distributions p and q. For each possible joint distribution γ, we can sample (x,y)~γ to get a sample x and y, and calculate the distance between the two samples ||xy||, E x,y～γ [||xy||] represents the expected value of the distance between samples under the joint distribution γ;

[0042] S43. The migration loss and the classification loss obtained in step S3 together constitute the loss function of the entire network. The total loss function formula is:

[0043] Loss=clf_loss + τ * EMD_loss (5)

[0044] clf_loss is the classification loss, EMD_loss is the migration loss, and τ is the migration rate. Adjust τ to adjust the proportion of migration loss in the total loss to train a neural network suitable for EEG across time periods.

[0045] Furthermore, step S5 specifically includes:

[0046] S51. A classification layer composed of a linear layer and a SoftMax classifier is input into the target domain feature map output in step S2 to obtain a classification result;

[0047] S52. And take the average accuracy rate to evaluate the classification performance: among all samples, the ratio of the number of samples correctly classified by the classifier to the total number of samples, that is, the probability of correct prediction;

[0048] S53. Select several commonly used advanced algorithms and compare their cross-time EEG motor imagery classification effects on public datasets.

[0049] Technical effects and advantages of the present invention:

[0050] The present invention can use labeled EEG data in the source domain and unlabeled EEG data in the target domain to jointly train an EEG model based on a separable convolutional neural network, so as to build a motor imagery recognition model with more accurate and reliable cross-time recognition and stronger generalization ability, and provide a scientific reference for cross-time motor imagery recognition in the real target domain unlabeled scenario.

[0051] Compared with the existing technology, the model of the present invention has stronger cross-domain application, shorter training time, better generalization ability, and transfer learning is more conducive to cross-time motion imagery classification, thereby obtaining more accurate and reliable recognition results.

[0052] In summary, the present invention can provide an accurate and reliable cross-time motor imagery classification model, and provides a favorable method for motor brain-computer interface decoding motor imagery. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 Schematic diagram of training for transfer learning.

[0054] Figure 2 It is a flow chart of the present invention. DETAILED DESCRIPTION

[0055] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0056] Refer to the instruction manual Figure 1-2 The invention provides a cross-period EEG motor imagery classification method based on transfer learning, comprising the following steps:

[0057] S1. According to the self-reading file, the EEG data is segmented according to the preset time points, and the source domain and target domain EEG data are obtained by data preprocessing;

[0058] S11. Cutting the original EEG information according to the preset time points, retaining the time period from the stimulation time point to the end of motor imagery;

[0059] S12. Remove the electrooculogram data and evenly distribute the data to obtain a source domain data set (source data) and a target domain data set (target data).

[0060] S2. Extract frequency, spatial and temporal information using attention-based separable convolutional neural networks;

[0061] S21. Use a two-dimensional convolution with a convolution kernel size of (1, kernelLength) to convolve the original EEG data to extract frequency features, and then use BatchNorm2d to perform batch normalization. The kernelLength is set to half the sampling frequency of the dataset, so that frequency information of 2 Hz and above can be captured;

[0062] S22. Convolve the feature map output from step S21 using a depthwise convolution with a kernel size of (channels, 1) to extract spatial features, where channels is set equal to the number of channels so that all channel information can be captured;

[0063] S23. Use the attention mechanism to highlight the most important channel information on the feature map output in step S22. The attention mechanism formula is:

[0064] Attention=tanh[softmax(wM)·H] (1)

[0065] Where H is the input feature, w is a trainable matrix, M = tanh(H);

[0066] S24. The feature map output in step S23 is batch normalized using BatchNorm2d, then passed through the Elu nonlinear activation function, and then through the pooling layer to reduce the sampling rate to 1 / 4 of the original, and finally Dropout is used to prevent overfitting;

[0067] S25. Convolve the feature map output by step S24 using separable convolution, where the separable convolution consists of a two-dimensional convolution with a convolution kernel size of (1, kernelLength2) and a point convolution with a convolution kernel size of (1,1), where kernelLength2 is set to half the input feature sampling frequency, the two-dimensional convolution learns the temporal characteristics of each feature map, and the point convolution learns a kernel to summarize each feature map separately, and then optimally merges the outputs to explicitly decouple the relationship between the feature maps and the internal features of the feature maps;

[0068] S26. The feature map output in step S25 is batch normalized, then passed through the Elu nonlinear activation function, and then through the pooling layer to reduce the sampling rate to 1 / 8 of the original, and finally Dropout is used to prevent overfitting.

[0069] S27. Use the multi-head self-attention mechanism to highlight the most important feature information of the feature map output in step S26. The multi-head self-attention formula is:

[0070]

[0071] The query vector Q, key vector K, and value vector V are obtained by inputting the input vector X into the fully connected layer. is the feature dimension.

[0072] S3. Use supervised learning to minimize the classification loss of source domain sample data;

[0073] S31. Use the source domain prediction results and source domain data labels for supervised learning, and use the cross-entropy loss function as the classification loss. The loss function formula is:

[0074]

[0075] Where N is the number of classification categories; y ic is a sign function (0 or 1), if the true category of sample i is equal to c, it takes 1, otherwise it takes 0; p ic is the predicted probability that the observed sample i belongs to category c.

[0076] S4. Use transfer learning to maximize the similarity between source domain and target and sample features to train a neural network based on cross-time EEG;

[0077] S41. An adaptive layer Bottlenck block consisting of a linear layer and a Relu nonlinear activation function is used to perform transfer learning on the source domain and target domain feature maps output by step S2;

[0078] S42. Calculate the loss between the adaptively layered source domain and target domain features, maximize the similarity between the source domain and the target and sample features, and use the Earth-Mover Distance as the migration loss. The loss function formula is:

[0079] EMD_loss=inf γ＝∏(p,q) E (x,y)～γ [||xy||] (4)

[0080] Where ∏(p,q) represents the set of all possible joint distributions of distributions p and q. For each possible joint distribution γ, we can sample (x,y)~γ to get a sample x and y, and calculate the distance between the two samples ||xy||, E x,y～γ [||xy||] represents the expected distance of the sample under the joint distribution γ;

[0081] S43. The migration loss and the classification loss obtained in step S3 together constitute the loss function of the entire network. The total loss function formula is:

[0082] Loss=clf_loss + τ * EMD_loss (5)

[0083] clf_loss is the classification loss, EMD_loss is the migration loss, and τ is the migration rate. Adjust τ to adjust the proportion of migration loss in the total loss to train a neural network suitable for EEG across time periods.

[0084] S5. Use the pre-trained model to classify the target domain, and compare the motor imagery classification effect on the public dataset with several selected commonly used advanced algorithms. And verify that the cross-time problem is common in motor brain-computer interfaces;

[0085] S51. A classification layer composed of a linear layer and a SoftMax classifier is input into the target domain feature map obtained in step S2 to obtain a classification result;

[0086] S52. And take the average accuracy rate to evaluate the classification performance: among all samples, the ratio of the number of samples correctly classified by the classifier to the total number of samples, that is, the probability of correct prediction;

[0087] S53. The motor imagery classification effect on the BCI-2a public dataset was compared with several selected commonly used advanced algorithms, and the results are shown in Table 1. Table 1 also experimentally verifies that the cross-period problem is common in motor brain-computer interfaces.

[0088] Table 1: Comparison of the effects of this method and several advanced algorithms on the BCI-2a dataset:

[0089]

[0090] Finally: The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

[0091] The present invention and its embodiments are described above, and such description is not restrictive. The drawings show only one embodiment of the present invention, and the actual structure is not limited thereto. In short, if ordinary technicians in the field are inspired by it, without departing from the purpose of the invention, they can design a structure and embodiment similar to the technical solution without creativity, which should belong to the protection scope of the present invention.

Claims

1. A cross-period EEG motor imagery classification method based on transfer learning, Features: The following steps are involved: S1. Preprocess the original EEG data to obtain source domain and target domain EEG data; S2. Extracting frequency, spatial and temporal information using attention-based separable convolutional neural networks; S3. Supervised learning minimizes the classification loss of source domain sample data; S4. Transfer learning maximizes the similarity between the source domain and the target domain and the sample features, which is used to train a neural network suitable for EEG across time periods; S5. Use the trained model to classify the target domain and compare the motor imagery classification results on the public dataset with the selected algorithm. It is found that the cross-period problem is common in motor brain-computer interfaces.

2. According to the method for classifying cross-period EEG motor imagery based on transfer learning according to claim 1, Features: The S1 specifically includes: segmenting and splicing the original EEG information according to preset time points to obtain two sets of data, the source domain and the target domain.

3. According to the method for classifying cross-period EEG motor imagery based on transfer learning according to claim 2, Features: The S2 specifically includes: S21. Use two-dimensional convolution to extract frequency features from the original EEG data and then perform batch normalization; S22. Convolve the feature map after step S21 using deep convolution to extract spatial features; S23. Use the attention mechanism to highlight the channel information on the feature map of step S22, where: The attention mechanism formula is: Attention=tanh[softmax(wM)·H](1) Where H is the input feature, w is the trainable matrix, and M = tanh(H); S24. The feature map outputted in step S23 is batch normalized, and the dimension is reduced by using the Elu nonlinear activation function and the pooling layer, and finally Dropout is used to prevent overfitting; S25. Convolve the feature map outputted from step S24 using separable convolution, first extract the temporal features using two-dimensional convolution, and then optimally combine the feature maps using point convolution; S26. Perform batch normalization on the feature map output in step S25, perform dimensionality reduction through the Elu nonlinear activation function and pooling layer, and finally perform Dropout to prevent overfitting; S27. Use a multi-head self-attention mechanism to highlight the feature information of the feature map output in step S26. The multi-head self-attention formula is: The query vector Q, key vector K, and value vector V are obtained by passing the input vector X through the fully connected layer. is the feature dimension.

4. According to the method for classifying cross-period EEG motor imagery based on transfer learning according to claim 1, Features: The S3 specifically includes: S31. Use the source domain prediction results and source domain data labels for supervised learning, and use the cross entropy loss function as the classification loss. The loss function formula is: Where N is the number of classification categories; y ic is a sign function (0 or 1), if the true category of sample i is equal to c, it takes 1, otherwise it takes 0; p ic is the predicted probability that the observed sample i belongs to category c.

5. According to claim 3, a cross-period EEG motor imagery classification method based on transfer learning, Features: The S4 specifically includes: S41. A linear layer and a Relu nonlinear activation function are combined into an adaptive layer, and the source domain and target domain feature maps outputted in step S2 are subjected to transfer learning; S42. Calculate the loss between the source domain and the target domain features output by the adaptive layer, maximize the similarity between the source domain and the target and sample features, and use the bulldozer distance as the migration loss. The loss function formula is: EMD_loss=inf γ＝∏(p,q) AND (x,y)～γ [||xy||](4) Where ∏(p,q) represents the set of all possible joint distributions of distributions p and q. For each possible joint distribution γ, we can sample (x,y)~γ to get a sample x and y, and calculate the distance between the two samples ||xy||, E x,y～γ [||xy||] represents the expected value of the distance between samples under the joint distribution γ; S43. The migration loss and the classification loss obtained in step S3 together constitute the loss function of the entire network. The total loss function formula is: Loss=clf_loss+τ*EMD_loss(5) clf_loss is the classification loss, EMD_loss is the migration loss, τ is the migration rate, and τ is adjusted to adjust the proportion of migration loss in the total loss to train a neural network suitable for EEG across time periods.

6. According to the method of claim 3 for classifying cross-period EEG motor imagery based on transfer learning, Features: The S5 specifically includes: S51. A linear layer and a SoftMax activation function are combined into a classification layer, and the target domain feature map output in step S2 is input to obtain a classification result; S52. The classification performance is evaluated by taking the average accuracy rate: among all samples, the ratio of the number of samples correctly classified by the classifier to the total number of samples is the probability of correct prediction; S53. Compare the cross-time EEG motor imagery classification effect on public datasets with selected algorithms.

Images