Motion imagination classification algorithm based on transfer learning

Abstract

The invention discloses a motor imagery classification algorithm based on transfer learning, and relates to the field of electroencephalogram signal processing, comprising the following steps: step 1, acquiring a source domain data set and a target domain data set of motor imagery electroencephalogram signals; 2, constructing a motion imagination classification model based on transfer learning by adopting a multi-source domain selective depth transfer learning framework based on an attention mechanism; the motor imagery classification model based on transfer learning comprises a domain recognition network, a selective domain adaptive module, a backbone neural network and a full connection layer. Step 3, obtaining source domain classification loss through a domain identification network, and calculating a source domain weight factor of each source domain for the target domain; 4, calculating a global loss function of the motor imagery classification model based on transfer learning; 5, optimizing parameters in the motor imagery classification model based on transfer learning; and step 6, carrying out selective transfer learning on the updated source domain instance features.

Description

Technical Field

[0001] The present invention relates to the field of electroencephalogram signal processing, and in particular to a motor imagery classification algorithm based on transfer learning. Background Art

[0002] The "Brain-computer Interface" (BCI) system can convert user intentions into machine instructions, thereby establishing a direct communication method between the human brain and external devices. EEG, as a non-invasive neuroimaging technology with high temporal resolution, has been widely used to record the electrical activity of the cerebral cortex. According to different neural patterns, EEG signals are mainly divided into motor imagery (MI), P300 and SSVEP. Due to its great potential in neurorehabilitation, the BCI paradigm based on motor imagery has received increasing attention. It can support the diagnosis and treatment of various diseases, such as chronic stroke, Parkinson's disease, spinal cord injury, etc. It plays an important role in assisting scientific researchers to study the working mechanism of the brain and helping patients with behavioral disorders, and has promoted the development of brain science and rehabilitation engineering.

[0003] Deep learning methods can significantly improve the decoding effect of MI-based EEG signals. However, training deep learning models requires collecting a large amount of motor imagery EEG, and collecting data requires a long calibration time, which is very unfriendly to disabled users. In the application scenario, we hope to reduce the dependence on new subject data and save the time cost of collecting data by mining data from other subjects to make up for the lack of current target user data. EEG signals are very weak, with a low signal-to-noise ratio, and there are differences between sessions of the same subject, and there are huge distribution differences between subjects, so it is very challenging to build a machine learning model that performs well for different subjects and different sessions.

[0004] In recent years, in the existing research in the field of BCI: transfer learning mainly involves instance selection and two-stage fine-tuning transfer. The instance selection method combines active learning with transfer learning, by selecting source domain data similar to a small number of new subjects' EEG trials, and adding them to the training set to train the BCI model together. The fine-tuning method exploits the similarity between the source domain and the target domain by fine-tuning the pre-trained model. However, these methods have almost no related work that considers the source domain and all the instances therein at the same time. In weighted transfer learning based on motor imagery EEG, the difference in classification parameters between new subjects and historical subjects is measured, and different weights are given to the source domain and the target domain by measuring feature similarity through KL divergence, which has achieved good results in binary classification tasks on different data sets, which encourages us to conduct more complex research.

[0005] In the actual application of BCI systems based on motor imagery, making full use of source domain EEG data is a challenging and promising approach. However, there are a lot of differences between the EEG data of different subjects (including source-source domain gaps and source-target domain gaps), and each subject has a different impact on transfer learning. Therefore, it is easy to cause serious "negative transfer" effects. Summary of the invention

[0006] Purpose of the invention: To address the problem that it takes a long time to calibrate the deep learning model collected from EEG data and the problem of negative transfer caused by large data differences between different source domains, the present invention provides a motor imagery classification algorithm based on transfer learning. The present invention uses the attention-based multi-source domain selective deep transfer learning framework (ASMSTL) to construct a motor imagery classification model based on transfer learning to improve the transfer ability of the target subject's EEG data.

[0007] The present invention provides a motor imagery classification algorithm based on transfer learning, which specifically includes the following steps:

[0008] Step 1: Obtain a source domain dataset and a target domain dataset of motor imagery EEG signals, and perform artifact removal and filtering on the source domain dataset and the target domain dataset;

[0009] Step 2: Using a multi-source domain selective deep transfer framework based on an attention mechanism to construct a motor imagery classification model based on transfer learning; the motor imagery classification model based on transfer learning includes:

[0010] 1) Domain identification network: The domain identification network is used to calculate the source domain weight factor w of each source domain for the target domain, so that the backbone neural network can learn domain knowledge with high similarity in a targeted manner;

[0011] 2) Selective Domain Adaptation Module:

[0012] The selective domain adaptation module assigns a weight coefficient β to each source domain instance feature extracted by the backbone neural network according to the similarity weight coefficient calculated by the domain recognition network;

[0013] 3) Backbone Neural Network:

[0014] The backbone neural network uses a shallow convolutional neural network φ(x; θ) as the backbone neural network to extract EEG features of the source domain and the target domain respectively, θ represents the parameters of the shallow convolutional neural network; x represents the motor imagery EEG signal;

[0015] 4) Fully connected layer:

[0016] The fully connected layer uses a softmax function for classification, updates weights through an Adam optimization algorithm, and outputs a motor imagery category;

[0017] Step 3: Obtain the source domain classification loss L through the domain identification network dr , and calculate the source domain weight factor w of each source domain for the target domain;

[0018] Step 4: Calculate the global loss function L of the motor imagery classification model based on transfer learning total ;

[0019] Step 5: Optimize the parameters θ and weight coefficient β in the motor imagery classification model based on transfer learning;

[0020] Step 6: Use the updated β to selectively transfer the source domain instance features.

[0021] Furthermore, in step one,

[0022] Source domain dataset: EEG signals usually come from different subjects, and the data distribution is different. The conversations recorded by N historical subjects are composed into the source domain dataset D S ={D1,...,D N}, N means that the source domain dataset contains data from N subjects, and its distribution is x represents all EEG data of the Kth subject in the source domain, and y represents the label corresponding to the data;

[0023] The target domain dataset refers to the dataset of the target subject. The dataset of a target subject is D T , whose distribution is P t (x,y);

[0024] In the source domain dataset, the EEG data of the Kth subject can be expressed as in m represents the number of source domain experiments, represents the EEG data in the K-th source domain subject dataset, is its corresponding label, so the data of the target subject can be expressed as Where n is the number of labeled trials in the target domain.

[0025] Furthermore, the specific operations of step three are as follows:

[0026] Step 3.1: Labels are redefined;

[0027]

[0028] Among them, C is the number of motor imagery categories, and K represents the serial number of the Kth source domain;

[0029] Step 3.2: Given the source domain EEG data and new labels After that, the domain recognition network is trained to obtain the source domain classification loss L dr :

[0030]

[0031] Among them, L dr is the source domain classification loss, represents the classification loss function;

[0032] Step 3.3, assuming that a domain recognition network is used to calculate the domain-related weights of an EEG batch size BS:

[0033]

[0034] Among them, w K represents the weight factor of the kth source domain, sgn() is the symbolic function, Indicates rounding up. hour,

[0035] Furthermore, the specific operations of step 4 are as follows:

[0036] Step 4.1. Calculate the loss L of the selective deep transfer learning algorithm based on the allocation domain related weight factor w sel ;

[0037]

[0038] Among them, L sel Represents the loss function of the selective transfer learning algorithm; is the unique weight obtained by the i-th instance in the K-th source domain dataset, its initial value is set to 1, and the minimum threshold is set to 0; represents the feature vector of the i-th EEG trial in the K-th source domain dataset, where Indicates mapping EEG data features into the regenerated Hilbert space for measurement; refers to the feature vector of the jth EEG trial in the target domain data;

[0039] Step 4.2: Calculate the classification loss L based on the attention mechanism att :

[0040]

[0041] in, represents the classification loss function of the Kth source domain;

[0042] Step 4.3: Global loss function L of the motor imagery classification model based on transfer learningtotal for:

[0043]

[0044] Among them, the hyperparameter λ1 represents the penalty coefficient for source domain data classification, the hyperparameter λ2 represents the degree of migration of source domain knowledge, and L dr is the source domain classification loss; It refers to the classification loss function of the target domain.

[0045] Furthermore, the specific operations of step five are as follows:

[0046] Under the premise of given parameters θ, the gradient descent method is used for alternating optimization:

[0047]

[0048] Among them, η1 and η2 are the learning rates of parameter θ and weight coefficient β, respectively. t represents the model parameters that are not currently updated, θ t+1 represents the optimized model parameters; β t Represents the current weight coefficient, β t+1 Represents the optimized weight coefficient, L in (θ t ,β t ) represents the loss of inner layer optimization calculated under the current model parameters and weight coefficients; L out (θ t+1 ) represents the calculated loss after parameter optimization; represents the gradient of the model parameters, represents the gradient of the weight parameter;

[0049] Formula θ t+1 With β t+1 Time synchronization solution Substitute into the formula get:

[0050]

[0051] Among them, L in () is the loss function of the inner layer optimization, which refers to the selective global loss L total ; L out () is the loss function of the outer optimization, which refers to the loss function when the model parameter is θ t+1 The classification loss when there is a small amount of target domain data

[0052] Beneficial effects:

[0053] (1) The present invention adopts a multi-source domain selective deep transfer learning framework based on the attention mechanism. The framework selectively transfers knowledge from multiple source domains to improve the generalization ability of the neural network in the target domain, which not only reduces the dependence on the amount of EEG data of new subjects, but also effectively alleviates the negative transfer effect. Specifically, the present invention introduces a domain recognition model to focus on source domains with higher similarity. In addition, the framework uses the classification performance of labeled data in the target domain as an indicator to adaptively select and utilize reliable source domain data.

[0054] (2) The method of the present invention was applied to the BCI competition dataset for testing experiments. The experiments showed that the method achieved satisfactory classification results and surpassed other transfer learning methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 A framework diagram of a motor imagery classification algorithm based on transfer learning of the present invention;

[0056] Figure 2 It is a block diagram of the direct transfer learning algorithm in the embodiment;

[0057] Figure 3 It is a block diagram of the selective transfer learning algorithm in the embodiment;

[0058] Figure 4 A block diagram of a closed-loop brain-computer interface system based on MIEEG in an embodiment;

[0059] Figure 5 It is the overall design diagram of the hardware in the embodiment;

[0060] Figure 6 Schematic diagram of the international 10-20 system in the embodiment;

[0061] Figure 7 This is a software and hardware joint debugging test diagram in the embodiment;

[0062] Figure 8 Topographic map of the correlation between motor imagery classification and EEG activity during implementation. DETAILED DESCRIPTION

[0063] The technical solution of the present invention is described in detail below with reference to the accompanying drawings, but the protection scope of the present invention is not limited to the embodiments.

[0064] Because the EEG data distribution of the target domain and the source domain is inconsistent Simply merging multiple source domain data for learning will not achieve satisfactory results. For this reason, multi-source domain transfer learning is a challenging and complex task, in which there are three main problems: 1) The data distribution of the source domain and the target domain is inconsistent. 2) There are differences in the quality and distribution of each EEG acquisition experiment. 3) The EEG of each historical subject has a different impact on transfer learning. In response to these three problems, the present invention proposes a multi-source domain selective deep transfer learning framework (ASMSTL) based on the attention mechanism. First, the maximum mean difference (MMD) algorithm is integrated into the framework and optimized to obtain a selective deep transfer learning (SDTL) algorithm. On this basis, a domain recognition network is introduced to conduct subject-specific and focused learning on historical EEG data.

[0065] The framework diagram of the motor imagery classification algorithm based on transfer learning of the present invention is as follows: Figure 1 As shown in the figure, the model uses multiple source domain data and combines a small amount of target domain data for transfer learning to improve its classification accuracy in the EEG of the target subject. It uses a domain recognition network to calculate the similarity weight coefficient (source domain weight factor) of each source domain to the target domain, and selectively uses the EEG instance features in the source domain to continuously track the transfer learning performance, effectively avoiding the negative transfer effect and improving the generalization ability of the model in the target domain.

[0066] The conception process of the motor imagery classification algorithm based on transfer learning of the present invention is as follows:

[0067] (I) Construct a multi-source domain direct transfer learning algorithm (MDTL), such as Figure 2 shown.

[0068] To address the problem that it takes a long time to calibrate a deep learning model after collecting EEG data, this paper proposes a method for direct migration from multiple source domains. In order to reduce the domain drift between multiple source domains and the target domain, the multi-source direct transfer learning algorithm uses the maximum mean difference to calculate the distribution difference between the source domain and target domain feature sets, and uses deep learning technology to adaptively minimize this distribution difference.

[0069] In this algorithm, a shallow convolutional neural network φ() is first used as the backbone to extract the EEG features of the source domain and the target domain respectively. The maximum mean difference (MMD) is used to measure the feature moment distance between each source domain and the target domain, and it is minimized. Finally, D is aligned in the feature space. S With D T Center of:

[0070]

[0071] in, represents the feature vector of the i-th EEG trial in the K-th source domain dataset, where Indicates that the EEG data features are mapped into the Regenerated Hilbert Space (RKHS) for measurement; MMD is a distribution measure based on the kernel method, which is used to compare the differences between two distributions. S Refers to the data set of N source domains, that is, D T Refers to the target domain data set.

[0072] The basic idea of ​​MMD is to map the data into a high-dimensional feature space, in which the distance between the means of two distributions is calculated. In transfer learning, MMD is often used for domain adaptation, the goal of which is to minimize the distribution difference between the source domain and the target domain.

[0073] The multi-source domain direct transfer learning algorithm not only needs to minimize the distance between domains, but also needs to effectively classify EEG in multiple domains, that is, to train a powerful classifier so that it can smoothly transfer learning across domains and accurately classify. Therefore, the global loss function is set:

[0074]

[0075] in Indicates the source domain and a small number of target domains Classification loss on labeled data, and is the corresponding true label. The hyperparameter λ1 represents the penalty coefficient for source domain data classification, and the hyperparameter λ2 determines the degree to which the source domain knowledge is desired to be transferred.

[0076] The multi-source direct transfer learning (MDTL) algorithm aims to reduce the domain difference between historical data and new data, so as to improve the classification performance on the EEG data of new subjects. However, it performs transfer learning on all historical data without screening. However, the distribution of EEG signals collected by different devices and in different sessions is not the same, which leads to deviations in the features extracted by the network.

[0077] The maximum mean difference (MMD) algorithm is used to continuously reduce the feature distribution differences between different domains. Although it can reduce the gap between domains, this method does not take into account the sample contribution within the domain, which will lead to a decrease in model performance.

[0078] (ii) Construct a selective deep transfer learning algorithm (SDTL), such as Figure 3 shown.

[0079] In order to solve the problem that direct migration from multiple source domains leads to decreased model performance, the present invention is inspired by deep secure semi-supervised learning and constructs a selective deep transfer learning algorithm. Different from weighting the instance loss of unlabeled samples, the selective deep transfer learning algorithm integrates the idea of ​​semi-supervised learning into the transfer learning method by learning the features of labeled source domain EEG instances to different degrees, so as to better mine the value of source domain EEG data.

[0080] The goal of deep transfer learning is to learn the model The data from all source domains and a small amount of target domains are parameterized as θ∈Θ, where Θ refers to the set of model parameters of each source domain, which is used to minimize the empirical risk R(h)=E (X,Y) [l(φ(X;θ),Y)], where l() represents the loss function, such as cross entropy loss. If the historical data is learned indiscriminately, the network will not be able to accurately learn the appropriate parameters θ. In fact, the EEG features of the new subject may be similar to some of the EEG features of the historical subjects. Strengthening the transfer learning of some data will greatly help improve the performance of the model. However, the EEG features of other parts are significantly different from them. If the learning of them is not suppressed, it will still cause negative transfer.

[0081] To address this problem, we selectively perform deep transfer learning to continuously optimize N sets of feature weight vectors based on the classification performance of the target subject’s labeled data. This means that during the training process, domain prior knowledge has different contributions to the classification of target domain data. The weight of source domain data that is beneficial to classification will gradually increase, so the contribution of this part of data to the target classification will gradually be strengthened. On the contrary, the other part of data that is destructive to the model will gradually weaken its influence on classification. We make improvements to formula (1), such as Figure 3 A selective transfer learning algorithm SDTL is proposed, which can be expressed as:

[0082]

[0083] in, is the unique weight obtained by the i-th instance in the K-th source domain. Its initial value is set to 1, and the minimum threshold is set to 0. It is continuously updated as the transfer learning proceeds. Therefore, the global loss is:

[0084]

[0085] (3) A motion imagery classification algorithm based on transfer learning is constructed by using the attention mechanism-based multi-source domain selective deep transfer learning framework (ASMSTL).

[0086] In the field of image processing, considering the correlation between domains, transfer learning is performed through domain recognition models and weighted moment distance, and gratifying results are obtained in computer vision tasks. Therefore, in order to better complete multi-source migration across subjects, in order to solve the problem of negative migration caused by large data differences between different source domains, this paper introduces a domain recognition network to calculate the correlation between each source domain and the target domain on the basis of the selective deep transfer learning algorithm (SDTL), and regularizes the migration loss and classification loss.

[0087] like Figure 1 As shown, the present invention provides a motor imagery classification algorithm based on transfer learning, which adopts an attention-based multi-source domain selective deep transfer learning framework (ASMSTL) to construct a motor imagery classification model based on transfer learning to improve the migration ability of the target subject's EEG data. The motor imagery classification model based on transfer learning includes:

[0088] Domain recognition network: The domain recognition network is used to calculate the source domain weight factor w of each source domain for the target domain, so that the backbone neural network can learn domain knowledge with high similarity in a targeted manner;

[0089] Selective domain adaptation module: The selective domain adaptation module assigns a weight coefficient β to each source domain instance feature extracted by the backbone neural network according to the similarity weight coefficient calculated by the domain recognition network;

[0090] Backbone neural network: The backbone neural network uses a shallow convolutional neural network φ(x; θ) as the backbone neural network to extract EEG features of the source domain and the target domain respectively, where θ represents the parameters of the shallow convolutional neural network; x represents the motor imagery EEG signal;

[0091] Fully connected layer; the fully connected layer uses the softmax function for classification, updates the weights through the Adam optimization algorithm, and outputs the motor imagery category.

[0092] After the source domain data and the target domain data are input into the motor imagery classification model based on transfer learning, they enter the backbone neural network. The backbone neural network adopts a selective deep transfer learning algorithm. According to the source domain weight factor calculated by the domain recognition network, the classification performance of the labeled data in the target domain is used as an indicator. It adaptively uses reliable source domain data and selectively uses instance features in the source domain for transfer learning, that is, learning domain knowledge with high similarity and selectively aligning the feature distributions of each domain, so as to perform effective transfer learning and avoid negative transfer effects.

[0093] (IV) Training strategy of motor imagery classification model based on transfer learning.

[0094] First, the framework calculates the domain relevance factor through a domain recognition network, which is trained through supervised learning of source domain data, where the labels of the source domain data are redefined as Therefore, after inputting a batch of target domain data, we will get the proportion of target domain data falling into each source domain, so as to determine the weight factor w assigned to each domain. In the following training process, L att and L sel The domain weight factor w is used in both.

[0095] Next, the backbone neural network will perform selective deep transfer learning. This method is mainly divided into two parts. The first part is to obtain the most appropriate instance feature selection coefficient β by solving a two-level optimization problem. The second part is to use the updated β to selectively transfer the source domain instance features. Specifically, the first part minimizes the selective global loss given the source domain data and a small amount of target domain data, and uses automatic differentiation technology to calculate θ t The gradient of the model is then manually updated to obtain θ t+1 The advantage of this is that after updating the parameters, the calculation graph is retained for the next step, and the gradient does not affect the subsequent steps through back propagation. The framework uses the model φ(θ t+1） Minimize the target domain EEG classification loss and get β after back propagation t The gradient of the selectivity coefficient β can be updated t+1 In the second part, ASMSTL selectively uses source domain instances and trains them together with target domain data, and updates the network. At this point, the framework completes a complete learning. In the next iteration, the network parameters θ in the first part t Provided by the model in the second part. Through such learning, the generalization ability of the framework in the target domain is significantly enhanced, effectively alleviating the negative transfer effect.

[0096] 1) Calculation of source domain weight factor:

[0097] The SDTL algorithm selects the EEG data of historical subjects and performs different degrees of migration. This critical migration is conducive to the model's accurate classification of new data of target subjects. However, it does not take into account the differences between historical subjects. Due to the existence of such individual differences, if the similarities between multiple source domains and target domain EEG features are not considered and they are directly mixed and trained to obtain a model, the expected classification effect will not be achieved on the new data of the target subjects. In order to solve this problem, the present invention introduces a domain recognition network in the ASMSTL framework, and calculates the source domain weight factor w={w1,...,w N}, the calculation method is as follows:

[0098] Although each source domain data contains the same motor imagery labels, such as left hand, right hand, tongue, feet, etc., there are still differences in data distribution between them, so we redefine the labels:

[0099]

[0100] Where C is the number of motor imagery categories. For example, in the dataset BCI Competition IV2a, C = 4. There are nine subjects in total. We set eight source domains and the data of the remaining subject as the target domain, so the label is redefined as K represents the sequence number of the Kth source domain.

[0101] Given source domain EEG data and new labels After that, the domain recognition network is trained to obtain the source domain classification loss L dr :

[0102]

[0103] Among them, L dr is the source domain classification loss; represents the classification loss function.

[0104] The source domain weight factor is calculated by the domain recognition network, which is trained by supervised learning of the source domain data, where the label of the source domain data is redefined as Therefore, after inputting a batch of target domain data, we will obtain the proportion of target domain data falling into each source domain, so as to determine the source domain weight factor w assigned to each domain.

[0105] Because the domain-aware network is composed of new labels Therefore, the present invention uses the domain recognition network to calculate the relevance of a batch of new EEG data to each historical subject. Suppose we use it to calculate the domain relevance factor of an EEG with a batch size of BS:

[0106]

[0107] Among them, w K represents the weight factor of the kth source domain, sgn() is the symbolic function, Indicates rounding up. hour, The domain recognition network obtains the proportion of each source domain from this batch of new EEG data, and thus obtains the weight factor corresponding to each source domain, where

[0108] 2) Global loss function L of the motor imagery classification model based on transfer learning total The calculation method is as follows:

[0109] By assigning domain-dependent weight factors w, the selective deep transfer learning algorithm SDTL can automatically select similar historical subjects’ motor imagery EEG for transfer learning:

[0110]

[0111] Among them, L sel Represents the loss function of the selective transfer learning algorithm SDTL; is the unique weight of the i-th instance in the K-th source domain dataset. Its initial value is set to 1 and the minimum threshold is set to 0. It is continuously updated as transfer learning proceeds. represents the feature vector of the i-th EEG trial in the K-th source domain dataset, where Indicates mapping EEG data features into the Regenerated Hilbert Space (RKHS) for measurement; refers to the feature vector of the jth EEG trial in the target domain data.

[0112] In addition to aligning the feature distribution of the target with that of the similar source domain, the discriminant loss used to train the network to learn the knowledge of each source domain also plays a vital role. If only selective alignment distribution is performed without selectively reducing the classification loss of the source domain, the negative transfer phenomenon cannot be eliminated. Therefore, the framework also assigns the weight factor calculated by the domain recognition network to the backbone neural network for supervised learning of the source domain, and defines the following classification loss based on the attention mechanism:

[0113]

[0114] in, represents the classification loss function of the Kth source domain;

[0115] In summary, the global loss function of ASMSTL is:

[0116]

[0117] Among them, the hyperparameter λ1 represents the penalty coefficient for source domain data classification, the hyperparameter λ2 represents the degree of migration of source domain knowledge, and L dr is the source domain classification loss, refers to the target domain classification loss.

[0118] This multi-source selective transfer learning scheme can perform high-quality EEG cross-subject classification tasks. When training the model, it will automatically learn the inter-domain correlation and calculate the weight factor to guide the backbone neural network to learn more similar source domain knowledge. When faced with multiple EEG instances in the domain, ASMSTL also adopts critical learning, so it can learn more representative instance features with transfer contributions, while suppressing instance features that do not contribute or are even destructive, thereby weakening their impact on the target domain EEG classification.

[0119] 3) Optimize the parameters θ and weight system β in the motor imagery classification model based on transfer learning:

[0120] On the one hand, the multi-source domain selective deep transfer learning framework (ASMSTL) based on the attention mechanism needs to selectively use the source domain data. The main method is to continuously optimize N groups of feature weights β, and all source domain instances are assigned unique weights. On the other hand, the multi-source domain selective deep transfer learning framework (ASMSTL) based on the attention mechanism attempts to minimize the global loss of selective transfer learning. Specifically, this is a two-layer optimization problem. The first inner optimization is to find a model that minimizes the selective global loss under given source domain data and a small amount of target domain data; the next outer optimization is to minimize the target domain EEG classification loss under a given model to find the most appropriate instance feature selection coefficient.

[0121] Under the premise of given parameters θ, the gradient descent method is used for alternating optimization:

[0122]

[0123] Among them, η1 and η2 are the learning rates of parameter θ and weight coefficient β respectively, among which, θ t represents the model parameters that are not currently updated, θ t+1 represents the optimized model parameters; β t Represents the current weight coefficient, β t+1 Represents the optimized weight coefficient, L in (θ t ,β t ) represents the loss of inner layer optimization calculated under the current model parameters and weight coefficients. out (θ t+1 ) represents the loss calculated after parameter optimization. represents the gradient of the model parameters, represents the gradient of the weight parameter.

[0124] But the formula θ t+1 With β t+1 Time synchronization needs to be solved, so Substitute into the formula get:

[0125]

[0126] Among them, L in ( ) is the loss function of the inner layer optimization, which refers to the selective global loss L total , that is, the training classification loss L att , L dr and migration loss L sel The sum of L out () is the loss function of the outer optimization, which means that when the model parameter is θ t+1 The classification loss when there is a small amount of target domain data

[0127] Given source domain data and a small amount of target domain data, minimize the selective global loss and use automatic differentiation techniques to calculate θ t The gradient of the model is then manually updated to obtain θ t+1 ; The advantage of this is that after updating the parameters, the calculation graph is retained for the next step, and the gradient does not affect the subsequent steps through back propagation. The framework uses the model φ(θ t+1 ) minimizes the target domain EEG classification loss, and after back propagation, β t The gradient of the selectivity coefficient β can be updated t+1 .

[0128] 4) Use the updated β to selectively transfer the source domain instance features.

[0129] The method for verifying the effect of the algorithm proposed in the present invention is as follows:

[0130] 1. Build the implementation platform: including system hardware circuit construction and system software and hardware joint debugging.

[0131] The hardware construction includes steps such as PCB soldering and EEG cap component installation. The system software is supported by the native system of Raspberry Pi, using multi-threaded programming, VNC Viewer to provide remote desktop services, and transfer learning model training and testing under the Pytorch deep learning framework. Figure 4 As shown, the overall system solution is a closed-loop workflow design based on the motor imagery paradigm.

[0132] The overall hardware design is based on the Raspberry Pi's EEG signal acquisition expansion board, which realizes high-quality EEG signal acquisition and visual analysis. EEG signals refer to an electrical signal recorded on the scalp that can reflect the activity of brain neurons. This electrical signal is very weak and often requires an amplifier to amplify it thousands of times before analysis. This design uses a low-noise amplifier and a multi-channel ADC (analog-to-digital converter) chip ADS1299, which is connected to the Raspberry Pi through a self-designed expansion board, and the transmission of EEG data is realized through the SPI communication protocol, achieving high-quality EEG signal acquisition. The overall hardware design plan is as follows: Figure 5 shown.

[0133] After completing the hardware construction and software code debugging, the EEG cap is worn. The EEG cap is worn using the international 10-20 system electrode position matching, and the acquisition signal is 8 channels, namely: F7, Fz, F8, C3, C4, T5, Pz, T6. The international 10-20 system is as follows: Figure 6 shown.

[0134] The system can run smoothly after the software and hardware joint debugging test. The results of the system software and hardware joint debugging test are as follows Figure 7 As shown, the lower left corner is a diagram of the hardware assembly, the interface is the Raspberry Pi remote desktop, and the display window is the data visualization interface.

[0135] The topographic map of the correlation between motor imagery and EEG activity is as follows Figure 8 As shown, the four-category motor imagery EEG classification is displayed, where the red part is the brain area that is strongly correlated with the specific motor imagery action, and the blue part is the brain area that is weakly correlated. The depth of the color reflects the strength of the correlation, indicating the extent of the influence of the EEG activity of this brain area on the specified motor imagery action.

[0136] 2. To evaluate the proposed algorithm, this embodiment uses dataset 2a from BCI Competition IV 2008 and dataset IV a from BCI Competition III.

[0137] Dataset 1: Dataset 2a of BCI Competition IV includes EEG signals of 9 subjects (S01-S09). It uses 22 electrodes with a sampling frequency of 250Hz to record four-category motor imagery EEG data from the left hand, right hand, feet, and tongue. Each subject conducted two experiments on different days, and each experiment contained 288 trials. In this paper, we use the data recorded on the first day as the training set, all the data of the first day of 8 subjects as the source domain data, and only 48 trials (12 trials per category) of the first day of the remaining subject are selected as the training data of the target domain. The 288 data of the target subject on the second day are used as the test set to evaluate the classification performance of the algorithm.

[0138] Dataset 2: Dataset Iva of BCI Competition III consists of EEG data recorded by 5 subjects (S01-S05) using 118 electrodes. The sampling frequency is 100Hz. The binary motor imagery data from the right hand and both feet are recorded. Each subject recorded 280 trials, all completed on the same day.

[0139] In the preprocessing stage, the dataset was de-artifacted and the EEG data was filtered using a 4-28 Hz bandpass filter. Dataset 1 selected 3.5 s of EEG data, with a total of 875 sample points, and dataset 2 selected 3 s of EEG data, with a total of 300 sample points.

[0140] The classification accuracy is used as the evaluation index of each comparison algorithm, and the calculation formula is:

[0141]

[0142] Among them, X represents EEG data, y represents the true label of x, and o(x i ) is x i The predicted label of .

[0143] The framework is implemented by programming in Python based on the PyTorch platform. The present invention uses a shallow convolutional neural network for feature extraction and trains the entire network through the Adam optimizer. The backbone network learning rate is 5e-4 and 1500 epochs of training are performed.

[0144] First, the domain recognition network is trained once. After each training, the correlation factor of each source domain is obtained. Then, we perform selective deep transfer learning on the shallow convolutional network, use the bottleneck layer to output relatively suitable 256-dimensional features, center-align the feature distribution of the source domain and the target domain, and perform classification training simultaneously. In order to analyze the effectiveness of the transfer learning of this algorithm, this embodiment sets two baseline methods for comparison on the data set as shown in Table 1. In Table 1, baseline one refers to the use of only source domain data for training of specific subjects. Baseline two refers to not using the transfer learning algorithm, and only using 22.2% of the target subject's training data and all the source subject's training data for training.

[0145] Table 1. Classification accuracy of two baseline methods and ASMSTL on the target subject test set

[0146]

[0147] In addition, in order to better illustrate that the transfer learning of the framework designed in this paper has stronger generalization ability, we compare ASMSTL (the algorithm of the present invention) with the fine-tuning algorithm, multi-source direct transfer (MDTL), and SDTL as shown in Tables 2 and 3. The fine-tuning algorithm uses the source domain data for pre-training, and after the training is completed, it performs 20 epochs of fine-tuning training on the target domain training data. DDC directly uses the MMD algorithm for multi-source transfer learning. SDTL is the selective deep transfer learning algorithm mentioned in the method part. In comparison, it can show that the domain recognition network plays a powerful role in this framework. Dataset 1 uses only 22.2% of the training data, with 16 experiments per class, and Dataset 2 uses 6% of the training data, with 16 experiments per class.

[0148] Table 2. Comparison of algorithm effects on the 2a dataset

[0149]

[0150] Table 3. Comparison of algorithm effects on the 4a dataset

[0151]

[0152] In the above embodiment, the framework proposed in the present invention is evaluated on a four-category and two-category public motor imagery EEG dataset, two baseline algorithms are set to test its effectiveness, and compared with three transfer learning algorithms: direct transfer, selective deep transfer, and fine-tuning. The data results show that the motor imagery classification algorithm based on transfer learning proposed in the present invention can improve the classification performance when the data is scarce or sufficient, and the task type is binary or multi-classification. The experimental results show that the proposed ASMSTL framework can significantly improve the performance of cross-domain EEG signal classification, and has good robustness and generalization ability.

[0153] As described above, although the present invention has been shown and described with reference to specific preferred embodiments, it should not be construed as limiting the present invention itself. Various changes in form and details may be made without departing from the spirit and scope of the present invention.

Claims

1. A motor imagery classification algorithm based on transfer learning, characterized in that: The steps include: Step 1: Obtain source domain dataset and target domain dataset of motor imagery EEG signals; Step 2: Use the multi-source domain selective deep transfer framework based on attention mechanism to build a transfer learning-based motor imagery classification model; The motor imagery classification model based on transfer learning includes: 1) Domain identification network: The domain identification network is used to calculate the source domain weight factor w of each source domain for the target domain, so that the backbone neural network can learn domain knowledge with high similarity in a targeted manner; 2) Selective Domain Adaptation Module: The selective domain adaptation module assigns a weight coefficient β to each source domain instance feature extracted by the backbone neural network according to the similarity weight coefficient calculated by the domain recognition network; 3) Backbone Neural Network: The backbone neural network uses a shallow convolutional neural network φ(x; θ) as the backbone neural network to extract EEG features of the source domain and the target domain respectively, θ represents the parameters of the shallow convolutional neural network; x represents the motor imagery EEG signal; 4) Fully connected layer: The fully connected layer uses a softmax function for classification, updates weights through an Adam optimization algorithm, and outputs a motor imagery category; Step 3: Obtain the source domain classification loss L through the domain identification network dr , and calculate the source domain weight factor w of each source domain for the target domain; Step 4: Calculate the global loss function L of the motor imagery classification model based on transfer learning total ; Step 5: Optimize the parameters θ and weight coefficient β in the motor imagery classification model based on transfer learning; Step 6: Use the updated β to selectively transfer the source domain instance features.

2. The motor imagery classification algorithm according to claim 1, characterized in that: In step one, Source domain dataset: The conversations recorded by N historical subjects form the source domain dataset D S ={D1,...,D N }, N means that the source domain dataset contains data from N subjects, and its distribution is x represents all EEG data of the Kth subject in the source domain, and y represents the label corresponding to the data; The target domain dataset refers to the dataset of the target subject. The dataset of a target subject is D T , whose distribution is P t (x,y); In the source domain dataset, the EEG data of the Kth subject is represented as in m represents the number of source domain experiments, represents the EEG data in the K-th source domain subject dataset, The data of the target subject is represented as its corresponding label. Where n is the number of labeled trials in the target domain.

3. The motor imagery classification algorithm according to claim 2, characterized in that: The specific operations of step three are as follows: Step 3.1: Labels are redefined; Among them, C is the number of motor imagery categories, and K represents the serial number of the Kth source domain; Step 3.2: Given the source domain EEG data and new labels After that, the domain recognition network is trained to obtain the source domain classification loss L dr : Among them, L dr is the source domain classification loss, represents the classification loss function; Step 3.3, assuming that a domain recognition network is used to calculate the domain-related weights of an EEG batch size BS: Among them, w K represents the weight factor of the kth source domain, sgn() is the symbolic function, Indicates rounding up. hour, 4. The motor imagery classification algorithm according to claim 3, characterized in that: The specific operations of step 4 are as follows: Step 4.

1. Calculate the loss L of the selective deep transfer learning algorithm based on the allocation domain related weight factor w sel ; Among them, L sel Represents the loss function of the selective transfer learning algorithm; is the unique weight obtained by the i-th instance in the K-th source domain dataset, its initial value is set to 1, and the minimum threshold is set to 0; represents the feature vector of the i-th EEG trial in the K-th source domain dataset, where Indicates mapping EEG data features into the regenerated Hilbert space for measurement; refers to the feature vector of the jth EEG trial in the target domain data; Step 4.2: Calculate the classification loss L based on the attention mechanism att : in, represents the classification loss function of the Kth source domain; Step 4.3: Global loss function L of the motor imagery classification model based on transfer learning total for: Among them, the hyperparameter λ1 represents the penalty coefficient for source domain data classification, the hyperparameter λ2 represents the degree of migration of source domain knowledge, and L dr is the source domain classification loss; It refers to the classification loss function of the target domain.

5. The motor imagery classification algorithm according to claim 4, characterized in that: The specific operations of step five are as follows: Under the premise of given parameters θ, the gradient descent method is used for alternating optimization: Among them, η1 and η2 are the learning rates of parameter θ and weight coefficient β, respectively. t represents the model parameters that are not currently updated, θ t+1 represents the optimized model parameters; β t Represents the current weight coefficient, β t+1 Represents the optimized weight coefficient, L in (θ t ,β t ) represents the loss of inner layer optimization calculated under the current model parameters and weight coefficients; L out (θ t+1 ) represents the calculated loss after parameter optimization; represents the gradient of the model parameters, represents the gradient of the weight parameter; Formula θ t+1 With β t+1 Time synchronization solution Substitute into the formula get: Among them, L in () is the loss function of the inner layer optimization, which refers to the selective global loss L total ; L out () is the loss function of the outer optimization, which refers to the loss function when the model parameter is θ t+1 The classification loss when there is a small amount of target domain data

Images