A cross-period electroencephalogram motor imagery classification method based on transfer learning

Abstract

The application discloses a cross-period electroencephalogram motor imagery classification method based on transfer learning and concretely relates to the field of motor brain-computer interface and comprises the following steps: S1, obtaining source domain and target domain electroencephalogram data by data preprocessing; S2, extracting frequency, space and time information by using a separable convolutional neural network based on attention; S3, minimizing the classification loss of source domain sample data by using supervised learning; S4, maximizing the similarity between source domain and target sample features by using transfer learning, so as to train a neural network suitable for cross-period electroencephalogram; S5, classifying the target domain by using the trained model, and comparing the motor imagery classification effect on a public data set with that of several selected common advanced algorithms. It is verified that the cross-period problem exists universally in the motor brain-computer interface. The method provided by the application provides a scientific and reliable motor imagery classification scheme, and constructs a motor imagery classification model with higher accuracy than the several selected common advanced algorithms.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of motor brain-computer interface, and particularly to a cross-period electroencephalogram motor imagination classification method based on transfer learning. BACKGROUND

[0002] Motor brain-computer interface (BCI) is an advanced technology that directly connects the brain to external devices to achieve motor control. It collects the electrical activity of the brain, decodes these signals and converts them into control commands, thereby achieving direct control of external devices such as robotic arms, wheelchairs or computer cursors, etc. The core of motor BCI is to bypass the traditional neuromuscular pathway and directly use brain signals such as electroencephalogram (EEG), cortical electroencephalogram (ECoG) and single neuron recording to achieve motor control. Motor BCI has broad application prospects in many fields. First, in the field of rehabilitation medicine, motor BCI can help patients with paraplegia, hemiplegia and other motor disorders to restore some motor functions. Second, in the field of neuroscience research, motor BCI provides a new means for studying brain motor control mechanisms. By analyzing the brain's motor signals, scientists can gain a deeper understanding of the brain's function and structure. In addition, in augmented reality (AR) and virtual reality (VR), motor BCI can be used for more natural and intuitive human-computer interaction. Using electroencephalogram to classify motor imagination is reasonable and scientific, which is conducive to building more accurate and reliable motor imagination classification models, and also helps to leverage the advantages of artificial intelligence technology, promoting the common development and prosperity of cross-disciplines.

[0003] Convolutional Neural Network (CNN) originated in the 1980s and was proposed by Yann LeCun et al. in 1998 and successfully applied to handwritten digit recognition. The goal of CNN is to extract features in two-dimensional image data through convolution operations to capture local features in images while reducing the number of parameters and computational complexity. Its main components include convolution layers, pooling layers and fully connected layers. Convolution layers extract local features through convolution kernel sliding windows, pooling layers reduce feature map size to improve model robustness, and fully connected layers are used for final classification or regression tasks. CNN performs well in image classification, object detection and image segmentation tasks, and with the development of deep learning, the structure has evolved, such as ResNet and DenseNet, which solve the gradient vanishing problem in deep networks by introducing residual connections and dense connections, significantly improving model performance. In summary, convolutional neural networks combine the advantages of convolution operations and neural networks to provide a powerful tool for processing image data, and are widely used in various visual recognition tasks. In the present application, convolutional neural networks fit the spatial topology of electroencephalogram, and can extract the frequency, spatial and temporal features of electroencephalogram.

[0004] Attention mechanism is a deep learning technique originally introduced in the task of machine translation to enhance the performance of neural networks when dealing with sequential data. The main idea is to focus on important information by assigning different weights while ignoring irrelevant information, thus improving the efficiency and accuracy of the model. Attention mechanism works by calculating importance scores for each position in the input sequence, dynamically adjusting the influence of each position, allowing the model to capture global and local dependencies more effectively. This mechanism has achieved significant success in natural language processing, computer vision, and speech recognition, especially in improving the processing of long sequential data and model interpretability. In this invention, the introduction of attention mechanism focuses on the important information of spatial and temporal features in electroencephalogram, significantly improving the accuracy of recognition.

[0005] Transfer learning is a deep learning method that applies knowledge learned from one domain to another related but different domain, achieving efficient feature extraction and classification. The background technology of transfer learning is analogous to human learning, where people often use previous knowledge and experience to speed up the learning process of new tasks. The core idea of transfer learning is to use a pre-trained model on a large dataset to transfer its knowledge to a smaller target dataset for fine-tuning to achieve good performance. This method not only reduces training time but also improves the model's generalization ability on small sample datasets. The following is the general workflow of transfer learning:

[0006] 1. Pre-trained model: Train a deep neural network model on a large-scale source domain dataset, which usually involves a large amount of labeled data and computational resources. These models are trained on large datasets, learning general feature representations.

[0007] 2. Feature extraction: Fix the first few layers of the pre-trained model, which have learned general low-level features, and use them as a feature extractor. This step can be seen as freezing the model's weights, keeping them unchanged during subsequent training.

[0008] 3. Model fine-tuning: Fine-tune the last few layers of the model on the target dataset, allowing it to learn new task-related feature representations. Fine-tuning usually uses a smaller learning rate to prevent damaging the existing knowledge in the pre-trained model. This step requires unfreezing the frozen weights and retraining the model on new data to adapt to new task requirements.

[0009] 4. Model training: Continue training the model on the target domain dataset to achieve good performance on the new task. Generally, the target domain dataset is unlabeled data or small sample data. During training, the model gradually learns the high-level features required for the target task and optimizes the model's parameters.

[0010] 5. Application to downstream tasks: Once the model is trained, it can be applied to various downstream tasks such as image classification, object detection, semantic segmentation, sentiment analysis, etc. Generally, the feature representation obtained through transfer learning can provide excellent performance in downstream tasks without the need for a large amount of labeled data.

[0011] Transfer learning has made important breakthroughs in computer vision, natural language processing, and recommendation systems, especially in scenarios where data is scarce or annotation costs are high, helping to solve the problems of long training time, poor generalization ability, and difficulty in cross-domain application of traditional deep learning. Therefore, transfer learning alleviates the problem of expensive labeled EEG data that is difficult to obtain in small sample scenarios, and only needs to fine-tune with unlabeled data in a specific task to achieve good recognition effect. SUMMARY

[0012] The technical problem to be solved by the present application is that, for the problem of motor imagination recognition in motor brain-computer, and the problem of EEG data labels being difficult to annotate and consuming manpower, the present application provides a cross-period EEG motor imagination classification method based on transfer learning.

[0013] To solve the above technical problems, the technical solution provided by the present application is: a cross-period EEG motor imagination classification method based on transfer learning, comprising the following specific steps:

[0014] S1. Obtain source domain and target domain EEG data by data preprocessing;

[0015] S2. Extract frequency, spatial and temporal information using attention-based separable convolutional neural network;

[0016] S3. Minimize the classification loss of source domain sample data using supervised learning;

[0017] S4. Maximize the similarity between source domain and target domain sample features using transfer learning to train a neural network suitable for cross-period EEG;

[0018] S5. Classify the target domain using the pre-trained model, and compare the motor imagination classification effect on the public dataset with several commonly used advanced algorithms. And it is verified that the cross-period problem exists universally in motor brain-computer interface.

[0019] Further, step S1 specifically comprises:

[0020] S11. Split and splice the original EEG information according to the preset time points to obtain two groups of source domain and target domain data.

[0021] Further, step S2 specifically comprises:

[0022] S21. Adopting two-dimensional convolution to extract frequency features from the original electroencephalogram data, and then performing batch standardization;

[0023] S22. Adopting deep convolution to extract spatial features from the feature map output by step S21;

[0024] S23. Using attention mechanism to highlight the most important channel information of the feature map output by step S22, and the attention mechanism formula is:

[0025] Attention=tanh[softmax(wM)·H] (1)

[0026] Wherein, H is the input feature, w is a trainable matrix, and M=tanh(H);

[0027] S24. Batch standardization is performed on the feature map output by step S23, and then Elu nonlinear activation function is performed, and then dimension reduction is performed through the pooling layer, and finally Dropout is used to prevent overfitting;

[0028] S25. Adopting separable convolution to convolve the feature map output by step S24, first using two-dimensional convolution to extract time features, and then using point convolution to optimally combine feature maps together;

[0029] S26. Batch standardization is performed on the feature map output by step S25, and then Elu nonlinear activation function is performed, and then dimension reduction is performed through the pooling layer, and finally Dropout is used to prevent overfitting;

[0030] S27. Using multi-head self-attention mechanism to highlight the most important feature information of the feature map output by step S26, and the multi-head self-attention formula is:

[0031]

[0032] Wherein, the query vector Q, the key vector K and the value vector V are obtained by full connection layer from the input vector X, is the feature dimension.

[0033] Further, step S3 specifically comprises:

[0034] S31. Supervised learning is performed by using the source domain prediction result and the source domain data label, and cross-entropy loss function is used as the classification loss, and the loss function formula is:

[0035]

[0036] Wherein, N is the number of classification categories; y ic is a symbol function (0 or 1), which takes 1 if the real category of sample i is equal to c, and 0 otherwise; pic to observe the predicted probability that the sample i belongs to the class c.

[0037] Further, step S4 specifically comprises:

[0038] S41. An adaptive layer Bottlenck block composed of a linear layer and a Relu nonlinear activation function is used to transfer the source domain and target domain feature maps output by step S2;

[0039] S42. The loss between the source domain and target domain features output by the adaptive layer is calculated, the similarity between the source domain and target domain features is maximized, and the Earth-Mover Distance is used as the transfer loss, and the loss function formula is:

[0040] EMD_loss = inf γ＝ ∏(p,q)E (x,y)～γ [||x-y||] (4)

[0041] Where ∏(p,q) represents the set of all possible joint distributions of the distribution p and q, for each possible joint distribution γ, a sample x and y can be obtained from it, and the distance ||x-y|| of the sample pair is calculated, E x,y～γ [||x-y||] represents the expected value of the sample pair distance under the joint distribution γ.

[0042] S43. The transfer loss and the classification loss obtained in step S3 jointly constitute the loss function of the entire network, and the total loss function formula is:

[0043] Loss = clf_loss + τ * EMD_loss (5) clf_loss is the classification loss, EMD_loss is the transfer loss, and τ is the transfer rate. Adjust τ to adjust the proportion of the transfer loss in the total loss to train the neural network suitable for cross-time EEG.

[0044] Further, step S5 specifically comprises:

[0045] S51. A classification layer composed of a linear layer and a SoftMax classifier is used to input the target domain feature map output by step S2 to obtain the classification result;

[0046] S52. The average accuracy is used to evaluate the classification performance: in all samples, the ratio of the number of samples correctly classified by the classifier to the total number of samples, i.e. the probability of correct prediction;

[0047] S53. Several commonly used advanced algorithms are selected, and the cross-time EEG motor imagination classification effect on the public data set is compared.

[0048] Technical effects and advantages of the present application:

[0049] The present application can use source domain labeled and target domain unlabeled electroencephalogram data to jointly train electroencephalogram model based on separable convolutional neural network, so as to make the constructed cross-period recognition more accurate and reliable, and the motion imagination recognition model has stronger generalization ability, and provides scientific reference for cross-period motion imagination recognition in real target domain unlabeled scene.

[0050] Compared with the prior art, the model of the present application has stronger cross-domain application, shorter training time, better generalization ability, and migration learning is more conducive to cross-period motion imagination classification, so as to obtain more accurate and reliable recognition results.

[0051] In summary, the present application can provide an accurate and reliable cross-period motion imagination classification model, and provides an advantageous method for decoding motion imagination of motor brain-computer interface. BRIEF DESCRIPTION OF DRAWINGS

[0052] Fig. 1 The figure is a schematic diagram for migration learning training.

[0053] Fig. 2 The figure is a flowchart of the present application. DETAILED DESCRIPTION

[0054] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0055] Referring to the drawings in the specification Figs. 1-2 The present application provides a cross-period electroencephalogram motion imagination classification method based on migration learning, comprising the following steps:

[0056] S1. According to the self-description file, the electroencephalogram data is cut according to the preset time point, and the source domain and target domain electroencephalogram data are obtained by data preprocessing;

[0057] S11. The original electroencephalogram information is cut according to the preset time point, and the time period from the stimulation time point to the end of motion imagination is retained;

[0058] S12. Remove the electrooculogram data, and evenly distribute the data to obtain the source domain data set (source data) and the target domain data set (target data).

[0059] S2. Extract frequency, space and time information by using attention-based separable convolutional neural network;

[0060] S21. Adopting a two-dimensional convolution with a kernel size of (1, kernelLength) to extract frequency features from the original electroencephalogram data, and then performing batch normalization BatchNorm2d, and setting kernelLength to half the sampling frequency of the data set, so as to capture frequency information of 2Hz and above;

[0061] S22. Adopting a deep convolution with a kernel size of (channels, 1) to convolve the feature map output by step S21, to extract spatial features, wherein channels is set to equal the number of channels, so as to capture all channel information;

[0062] S23. Using an attention mechanism to highlight the most important channel information for the feature map output by step S22, and the attention mechanism formula is:

[0063] Attention=tanh[softmax(wM)·H] (1)

[0064] wherein H is the input feature, w is a trainable matrix, and M=tanh(H);

[0065] S24. Performing batch normalization BatchNorm2d on the feature map output by step S23, and then passing it through an Elu nonlinear activation function, then reducing the dimension through a pooling layer to reduce the sampling rate to 1 / 4 of the original, and finally Dropout to prevent overfitting;

[0066] S25. Adopting separable convolution to convolve the feature map output by step S24, which is composed of a two-dimensional convolution with a kernel size of (1, kernelLength2) and a point convolution with a kernel size of (1, 1), wherein kernelLength2 is set to half the sampling frequency of the input feature, the two-dimensional convolution learns the time characteristics of each feature map, and the point convolution learns to summarize each feature map respectively, and then optimally combines the output to explicitly decouple the relationship between the feature map and the feature map;

[0067] S26. Performing batch normalization on the feature map output by step S25, and then passing it through an Elu nonlinear activation function, then reducing the dimension through a pooling layer to reduce the sampling rate to 1 / 8 of the original, and finally Dropout to prevent overfitting.

[0068] S27. Using a multi-head self-attention mechanism to highlight the most important feature information for the feature map output by step S26, and the multi-head self-attention formula is

[0069]

[0070] Wherein the query vector Q, key vector K, value vector V are obtained by inputting the input vector X into the full connection layer, is the feature dimension.

[0071] S3. Minimize the classification loss of the source domain sample data by supervised learning;

[0072] S31. Supervised learning is performed using the source domain prediction result and the source domain data label, and the cross-entropy loss function is used as the classification loss, and the loss function formula is:

[0073]

[0074] Wherein, N is the number of classification categories; y ic is the symbol function (0 or 1), which takes 1 if the real category of sample i is equal to c, otherwise 0; p ic is the predicted probability that the observation sample i belongs to category c.

[0075] S4. Maximize the similarity between the source domain and the target and sample features by using transfer learning to train the neural network based on cross-period electroencephalogram;

[0076] S41. An adaptive layer Bottlenck block composed of a linear layer and a Relu nonlinear activation function is used to transfer the source domain and target domain feature maps output by step S2;

[0077] S42. The loss between the source domain and target domain features after passing through the adaptive layer is calculated, the similarity between the source domain and the target and sample features is maximized, and the Earth-Mover Distance is used as the transfer loss, and the loss function formula is:

[0078] EMD_loss = inf γ＝∏(p,q) E (x,y)～γ [||x-y||] (4)

[0079] Wherein ∏ (p, q) represents the set of all possible joint distributions of the combination of distributions p and q, for each possible joint distribution γ, a sample x and y can be obtained from it by sampling (x, y) ~ γ, and the distance ||x-y|| of the sample pair is calculated. x,y～γ [||x-y||] represents the expected distance of the sample pair under the joint distribution γ.

[0080] S43. The transfer loss and the classification loss obtained in step S3 jointly constitute the loss function of the entire network, and the total loss function formula is:

[0081] Loss = clf_loss + τ * EMD_loss (5) clf_loss is the classification loss, EMD_loss is the transfer loss, and τ is the transfer rate. τ is adjusted to adjust the proportion of the transfer loss in the total loss to train the neural network suitable for cross-period electroencephalogram.

[0082] S5. Classify the target domain using the pre-trained model, and compare the motor imagination classification effect on the public dataset with several selected common advanced algorithms. It is verified that the cross-period problem exists universally in motor brain-computer interface.

[0083] S51. A classification layer composed of a linear layer and a SoftMax classifier is used to input the target domain feature map obtained in step S2 to obtain a classification result.

[0084] S52. The classification performance is evaluated by using the average accuracy: the ratio of the number of samples correctly classified by the classifier to the total number of samples, i.e. the probability of correct prediction.

[0085] S53. The motor imagination classification effect on the BCI-2a public dataset is compared with several selected common advanced algorithms, and the results are shown in Table 1. It is verified from Table 1 that the cross-period problem exists universally in motor brain-computer interface.

[0086] Table 1: Comparison of the effects of the method and several advanced algorithms on the BCI-2a dataset

[0087]

[0088] Finally: The above only describes the preferred embodiments of the present application and is not used to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

[0089] The present application and its embodiments are described above, which is not restrictive, and the embodiments shown in the drawings are only one of the embodiments of the present application, and the actual structure is not limited thereto. In general, if a person skilled in the art is inspired by it, without departing from the purpose of the present application, without creative design, similar structure and embodiments of the technical solutions can be designed, which should belong to the protection scope of the present application.

Claims

1. A cross-session electroencephalogram motor imagery classification method based on transfer learning, characterized in that: The method comprises the following steps: S1. Preprocessing the original electroencephalogram data to obtain source domain and target domain electroencephalogram data; S2. Extracting frequency, space and time information using an attention-based separable convolutional neural network; S3. Supervised learning to minimize the classification loss of the source domain sample data; S4. Transfer learning to maximize the similarity between the source domain and target domain sample features, for training a neural network suitable for cross-period electroencephalogram; S5. Classifying the target domain using the trained model, and comparing the motor imagery classification effect on the public data set with selected algorithms to obtain the universality of the cross-period problem in motor brain-computer interface; The S3 specifically comprises: S31. Supervised learning using the source domain prediction results and the source domain data labels, and using a cross-entropy loss function as the classification loss, with the loss function formula being: where N is the number of classification classes; y ic is the indicator function (0 or 1) that takes the value 1 if the true class of sample i is equal to c and 0 otherwise; p ic is the predicted probability that observation i belongs to class c. The S4 specifically comprises: S41. An adaptive layer composed of a linear layer and a Relu nonlinear activation function is used to perform transfer learning on the source domain and target domain feature maps output by step S2; S42. The loss between the source domain and target domain features output by the adaptive layer is calculated, the similarity between the source domain and target domain sample features is maximized, and the bulldozer distance is used as the transfer loss, with the loss function formula being: EMD_loss = inf γ＝∏(p,q) E (x,y)～γ [||x-y||] (4) where Π(p, q) denotes the set of all possible joint distributions that p and q can combine into, for each possible joint distribution γ one can sample a pair (x, y) ~ γ and compute the distance ||x - y||, E x,y～γ [||x - y||] denotes the expected value of the distance of a pair of samples under the joint distribution γ. S43. The transfer loss and the classification loss obtained in step S3 jointly constitute the loss function of the entire network, with the total loss function formula being: Loss = clf_loss + τ * EMD_loss (5) clf_loss is the classification loss, EMD_loss is the transfer loss, τ is the transfer rate, and τ is adjusted to adjust the proportion of the transfer loss in the total loss to train the neural network suitable for cross-period electroencephalogram.

2. The cross-period electroencephalogram motor imagery classification method based on transfer learning according to claim 1, characterized in that: The S1 specifically comprises: The original electroencephalogram information is segmented and spliced according to the preset time points to obtain two groups of data of the source domain and the target domain.

3. The cross-session electroencephalogram motor imagery classification method based on transfer learning according to claim 2, characterized in that: The S2 specifically comprises: S21. Batch standardization is performed after extracting frequency features by two-dimensional convolution on the original electroencephalogram data; S22. Spatial features are extracted by performing convolution on the feature maps after step S21 using a deep convolution; S23. The attention mechanism is used to highlight the channel information of the feature maps of step S22, wherein, The attention mechanism formula is: Attention = tanh [softmax (wM) · H] (1) Wherein, H is the input feature, w is a trainable matrix, and M = tanh (H); S24. Batch standardization is performed on the feature maps output by step S23, followed by Elu nonlinear activation function clipping and pooling layer dimension reduction, and finally Dropout is used to prevent overfitting; S25. Separable convolution is used on the feature maps output by step S24, two-dimensional convolution is used to extract time features first, and then point convolution is used to optimally combine the features together; S26. Batch standardization is performed on the feature maps output by step S25, followed by Elu nonlinear activation function and pooling layer dimension reduction, and finally Dropout is used to prevent overfitting; S27. The multi-head self-attention mechanism is used to highlight the feature information of the feature maps output by step S26, with the multi-head self-attention formula being: where the query vector Q, the key vector K, and the value vector V are obtained from the input vector X through a fully connected layer, is the feature dimension.

4. The cross-period electroencephalogram motor imagery classification method based on transfer learning according to claim 3, characterized in that: The S5 specifically comprises: S51. A linear layer and SoftMax activation function are combined to form a classification layer, and the target domain feature map output by step S2 is input to obtain a classification result; S52. The average accuracy is used to evaluate the classification performance: the ratio of the number of samples correctly classified by the classifier to the total number of samples in all samples is the probability of correct prediction; S53. The cross-time brain electrical motion imagination classification effect of the selected algorithm on the public data set is compared.

Images