Motor imagery electroencephalogram recognition method based on domain adaptation network

Abstract

The invention discloses a motor imagery electroencephalogram recognition method based on a domain adaptation network, and the method comprises the steps: collecting multi-channel motor imagery electroencephalogram signals EEG of a subject, carrying out the band-pass filtering, obtaining a plurality of EEG signals of different frequency bands, taking the EEG signals of the target subject as target domain data, and taking the rest as source domain data. Secondly, on the basis of the source domain data and the target domain data, source domain features and target domain features are obtained through a multi-view convolution Transform feature extractor composed of a parallel space-time convolution module and a Transform module in sequence; and finally, performing adversarial domain adaptation by using the source domain features and the target domain features to obtain a model capable of adapting to target domain data, and identifying a motor imagery category corresponding to the electroencephalogram signal. According to the method, source domain data distribution can be guided to be transferred to a target domain, and the classification accuracy of cross-subject motor imagery electroencephalogram signals is improved.

Description

Technical Field

[0001] The invention belongs to the field of pattern recognition and proposes a motor imagery EEG recognition method based on a domain adaptation network. Background Art

[0002] EEG signals are spontaneous electrophysiological signals that are not affected by subjective factors and can reflect the subject's true motor imagery state. With the development of non-invasive electrode technology, effective EEG signals can be collected by wearing a simple EEG cap. Therefore, the motor imagery recognition method based on EEG signals has higher accuracy and reliability, and can overcome the limitations of traditional motor imagery recognition methods.

[0003] As deep neural networks have made more and more significant progress in computer vision tasks, more and more researchers have applied deep learning methods to the field of motor imagery EEG recognition. Although many deep learning methods can extract effective features of the subjects and achieve high classification accuracy, their models are all trained for a single subject and cannot be applied to other subjects. Due to individual differences in EEG signals, motor imagery recognition models require stronger generalization capabilities in cross-subject problems. Therefore, while extracting subject features, it is necessary to extract cross-domain invariant features, so domain adaptation is required. In general, there are two types of methods in the field of domain adaptation, one is the moment matching method, and the other is the adversarial learning method.

[0004] Moment matching methods usually quantify domain differences through statistical moments between source and target domains, and introduce the differences into deep networks in the form of loss functions, thereby directly reducing domain shifts and aligning the distribution of source and target domain features. Adversarial learning methods introduce an adversarial strategy between feature generators and discriminators, which makes it impossible to distinguish whether the features extracted by the feature generator are from source domain data or target domain data, thereby achieving cross-domain invariance of features and reducing the difference in feature distribution between source and target domains. Inspired by generative adversarial nets (GANs), researchers proposed domain-adversarial neural networks (DANNs), which introduced an additional domain-level discriminator to determine whether the features are from source domain samples or target domain samples, and induced the feature generator to extract features that are indistinguishable at the domain level in back propagation through a gradient reversal layer. Summary of the invention

[0005] In response to the above problems, the present invention proposes a method for motor imagery EEG recognition based on a domain adaptation network, which adds a multi-view convolution module and a Transformer module to the adversarial domain adaptation network, extracts the long-term dependency of the EEG after extracting the multi-band view information of the EEG, fully mines the time domain, frequency domain, and spatial domain features of the EEG signal, and then uses the adversarial domain adaptation method to generate domain invariant features, thereby realizing cross-subject motor imagery EEG signal recognition. Since the present invention fully mines the feature information of the EEG signal on the basis of the deep adversarial domain adaptation method, it helps to transfer the source domain features to the target domain more easily, thereby improving the classification performance of the model. The present invention first divides the original EEG signal into multi-frequency views according to different frequencies, uses the spatiotemporal convolution module and the Transformer module to extract the spatiotemporal features of the EEG signal of each view, and then introduces the discriminator and the classifier to construct a domain adaptation network for adversarial training. Dividing EEG signals into multiple frequency views helps to mine their frequency feature information; using parallel spatiotemporal convolution modules to extract spatiotemporal features helps to retain the complete temporal and spatial feature structure; using Transformer modules to further extract long-term dependencies in features, and using the attention mechanism to guide the migration of source domain features to the target domain; finally, through domain adaptation, the feature distribution of the source domain and the target domain is gradually reduced to improve the recognition performance of the model in the target domain.

[0006] The present invention comprises the following steps:

[0007] Step (1): Multi-channel EEG signal acquisition and preprocessing. First, the multi-channel motor imagery EEG signal of the subject is acquired, and then the EEG signal of each channel is band-pass filtered using a Butterworth filter bank to obtain multiple EEG signals of different frequency bands. The EEG signal of the target subject is used as the target domain data, and the rest is used as the source domain data.

[0008] Step (2): Based on the source domain data and the target domain data, the source domain features and the target domain features are obtained respectively through a multi-view convolutional Transformer feature extractor composed of a parallel spatiotemporal convolution module and a Transformer module.

[0009] For the source domain data and target domain data, parallel spatiotemporal convolution modules are used to extract the temporal and spatial features of EEG signals in different frequency bands respectively, and then add them together to obtain the spatiotemporal features of EEG. After the spatiotemporal features are extracted, the Transformer module is used to extract the long-term dependencies contained therein. The long-term dependencies extracted from the spatiotemporal features of multiple frequency bands are added together to obtain the final source domain features and target domain features.

[0010] Step (3): Use source domain features and target domain features for adversarial domain adaptation. Use the inverted gradient layer to guide the feature extractor and discriminator to form an adversarial relationship, maximize the domain prediction loss of the source domain data and the target domain data, align the marginal distribution of the source domain data and the target domain data, and align the conditional distribution between different categories through the supervision of the classifier on the category prediction loss. Finally, a model that can adapt to the target domain data is obtained. Input the target domain test data into the model to identify the motor imagery category corresponding to the EEG signal, such as left hand movement and right hand movement.

[0011] The beneficial effects of the present invention are as follows: the domain adaptation network based on multi-view convolutional Transformer can fully explore the inherent time domain, frequency domain, and spatial domain information of EEG signals, and extract the long-term dependencies of EEG signals on this basis, and learn domain-invariant features through adversarial domain adaptation training, which helps to guide the transfer of source domain data distribution to the target domain, thereby improving the classification accuracy of motor imagery EEG signals across subjects. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 This is a flowchart of the domain adaptation network based on multi-view convolutional Transformer;

[0013] Figure 2 This is the structure diagram of the parallel spatiotemporal convolution module;

[0014] Figure 3 This is the Transformer module structure diagram. DETAILED DESCRIPTION

[0015] The motor imagery EEG recognition method based on domain adaptation network is used in this paper to evaluate the effectiveness of the proposed method. The EEG dataset is provided by Graz University of Technology and contains 9 subjects (S01-S09), using 25 Ag / AgCl electrodes and a sampling rate of 250Hz. Four different motor imagery tasks were performed in two different sessions, including imagining moving the left hand, right hand, feet and tongue.

[0016] The present invention is described in detail below with reference to the accompanying drawings, which is a method for motor imagery EEG recognition based on a domain adaptation network of a multi-view convolutional Transformer.

[0017] Step (1): EEG signal acquisition and preprocessing.

[0018] The original EEG signal was filtered with a 50 Hz notch filter, and then an overlapping third-order Butterworth filter was used to decompose the original EEG signal into four sub-band signals: 0-4 Hz, 2-10 Hz, 6-22 Hz, and 16-38 Hz.

[0019] Finally, the filtered data is z-score normalized to reduce non-stationarity:

[0020]

[0021] where x i and x o They represent the bandpass filtered data and the normalized data respectively. μ and σ 2 represents the mean and variance, which are calculated from the filtered data.

[0022] Step (2): Use multi-view convolutional Transformer to extract features.

[0023] The overall flow chart of the present invention is as follows: Figure 1 There are two main steps in the feature extraction process. The first step is to use the parallel spatiotemporal convolution module to extract spatiotemporal features from the preprocessed multi-band data.

[0024] First, the preprocessed data obtained in step (1) is sent to the parallel spatiotemporal convolution module, such as Figure 2 As shown, the spatiotemporal features are initially extracted, and the module consists of two parallel one-dimensional convolutions. For temporal convolution, the kernel size is (k×1×n), where k represents the number of feature maps and n represents the length of the temporal kernel. In spatial convolution, the kernel size is (k×m×1), where m is usually equal to the number of channels C of the EEG signal. The temporal and spatial convolution results are then passed to the reshaped convolution layer to obtain two one-dimensional feature vectors, and the two vectors are connected to obtain the final spatiotemporal feature vector. Each convolution layer is followed by a batch normalization layer and an ELU activation function to enhance the nonlinear representation ability of the network. Compared with the disadvantage that the traditional feature extractor may destroy the original feature structure information after applying spatial convolution to the temporal feature output, the parallel strategy used in the present invention can retain the integrity of the spatiotemporal feature structure to the greatest extent. Therefore, the use of parallel spatiotemporal convolution for each frequency branch can simultaneously retain the temporal and spatial structure information, which is conducive to feature transfer between different domains.

[0025] Secondly, the spatiotemporal feature vector obtained by the parallel spatiotemporal convolution module is input into the Transformer module, such as Figure 3As shown, long-term dependencies in EEG signals are further extracted. These features are guided from the source domain to the target domain through the scaled dot product attention mechanism between the encoder and decoder in the Transformer. Specifically, the spatiotemporal features obtained in the previous step are first flattened in the channel direction, and then they are split into several slices of equal length along the time dimension, with a length of d, each slice representing a feature representation segment. The encoder and decoder of the Transformer are used to linearly transform the slices from the two domains into a query vector (Q) of the same size, a key vector (K) of the target domain, and a value vector (V) of the source domain. Then, the long-term dependencies of the target domain are evaluated by calculating the attention between the target domain and the target domain. Attention can be calculated by the following formula:

[0026]

[0027] On this basis, the multi-head attention mechanism (MHA) is used to improve the performance of Transformer. The multi-head attention mechanism divides the generated Q, K, V vectors into h subspaces and generates h different groups of Q, K, V. Each head performs attention calculations independently, so that the model pays attention to different parts of the input sequence in different subspaces. The diversity of the multi-head attention mechanism can capture contextual information at different levels, thereby improving the model's understanding of the input information and providing more comprehensive contextual features for subsequent layers. The multi-head attention mechanism can be calculated using the following formula:

[0028] MHA(Q,K,V)=[head0;…;head h-1 ], (3)

[0029] Among them, head l =Attention(xW l Q ,xW l K ,xW l V ), x is the input data, W l Q ,W l K ,W l V They are the weight matrices corresponding to the query vector Q, the key vector K of the target domain, and the value vector V of the source domain.

[0030] Step (3): Adversarial training using domain discriminator and classifier

[0031] In order to conduct adversarial training, a domain discriminator and a classifier are constructed. They have similar structures and are both composed of two fully connected layers. Among them, the domain discriminator is used to determine whether the features of the input data come from the source domain or the target domain. Through the feedback of the classifier, the difference in the feature distribution of the two domains can be measured. In adversarial training, the feature extractor and the domain discriminator exist in opposition. The goal of the feature extractor is to extract domain-invariant features so that the domain discriminator cannot distinguish the source of the features (source domain or target domain). Through this adversarial training, the feature extractor will gradually learn features that are applicable to both the source domain and the target domain. The classifier is used to predict the classification results. The training of the classifier will guide the feature extractor to learn features that can correctly distinguish categories, so that the domain-invariant features achieved by the discriminator have category separability and can better complete the classification task. Specifically, during the training process, there are mainly the following goals.

[0032] The first goal is to minimize the prediction loss of the classifier, and the loss function is as follows:

[0033]

[0034] Among them, N t 、N s are the number of target domains and source domains respectively, M is the number of categories, y ic is the actual value, is the predicted value.

[0035] The second goal is to guide the feature extractor to form an adversarial relationship with the discriminator by inverting the gradient layer, maximizing the domain prediction loss of the source domain data and the target domain data, and preliminarily aligning the distribution of the source domain data and the target domain data at the domain level. The loss function is as follows:

[0036]

[0037] in, is the expectation, x s is the source domain data; D(·) is the discriminator; B is the number of frequency bands; MBTCN (b) (·) is the multi-view convolutional Transformer feature extractor.

[0038] For the feature output of each frequency band branch, a classifier is also used to predict the classification result, and the loss function is as follows:

[0039]

[0040] Among them, N is the number of data, M is the number of categories, and y ic is the true value, is the tag value.

[0041] The final overall objective function is:

[0042]

[0043] In order to verify the rationality of the motor imagery EEG recognition method based on the domain adaptation network of the multi-view convolutional Transformer proposed in the present invention, a variety of methods were compared, as shown in Table 1.

[0044] Table 1 Classification results of similar methods on the BCI competition IV 2a dataset

[0045] method S01 S02 S03 S04 S05 S06 S07 S08 S09 average FBCSP 76 56.5 81.25 61 55 45.25 82.75 81.25 70.75 67.75 ssCSP 76.74 58.68 81.25 57.64 38.54 48.26 76.39 79.17 78.82 66.17 SSMM 82.64 60.76 85.76 67.01 58.68 54.51 90.97 81.25 79.51 73.45 ConvNet 76.39 55.21 89.24 74.65 56.94 54.17 92.71 77.08 76.39 72.53 MI-CNN 73.26 28.82 89.58 68.06 26.39 28.82 75.35 78.82 77.08 60.69 C2CM 87.5 65.28 90.28 66.67 62.5 45.49 89.58 83.33 79.51 74.46 SHNN 82.76 68.97 79.31 65.52 58.62 48.28 86.21 89.66 89.87 74.26 DRDA 83.19 55.14 87.43 75.28 62.29 57.15 86.18 83.61 82 74.74 GAT 88.89 61.11 93.4 71.86 50.35 60.07 89.58 87.5 86.46 76.58 The present invention 93.75 65.63 90.63 71.88 56.25 59.38 90.28 87.5 84.38 77.74

[0046] In order to comprehensively evaluate the performance of the framework, the proposed method is compared with the state-of-the-art methods on the same dataset, including manual feature extraction methods FBCSP and ssCSP, SSMM, end-to-end convolutional neural networks ConvNet, MI-CNN, C2M and SHNN, and adversarial learning-based domain adaptation models DRDA and GAT. The classification performance of each subject and the average results are shown in Table 1. The proposed method shows excellent performance on this dataset, achieving the highest average accuracy of 77.74%. First, this shows that the proposed method is significantly better than manual feature extraction methods such as FBCSP and ssCSP, and CNN-based end-to-end models such as ConvNet and MI-CNN. Despite combining the advantages of traditional feature extractors and CNNs and fine-tuning models for each subject, C2CM and SHNN are still insufficient compared to the proposed method. Compared with the DRDA and GAT methods, the proposed method not only focuses on the long-term dependencies of the EEG data of the target subject using the Transformer block, but also utilizes the frequency characteristics of the EEG, and the average accuracy is improved by 3.00% and 1.16%, respectively, which is sufficient to verify the effectiveness of the proposed method.

[0047] Overall, in order to address the challenges brought by individual differences and instability of EEG signals, the present invention proposes a new adversarial domain adaptation network. This method uses multi-view parallel spatiotemporal convolution to extract the spatiotemporal structural information of EEG signals. In addition, the method also combines the Transformer module to extract long-term dependencies from EEG signals and aligns the feature distributions between the source domain and the target domain. The experimental results of the present invention clearly support the effectiveness of this method. Through the multi-view parallel spatiotemporal convolution module and the Transformer module, the present invention can successfully capture the cross-domain invariant features of EEG signals of different subjects, thereby realizing domain adaptation across subject tasks, and providing new ideas for subsequent adversarial domain adaptation network methods.

Claims

1. A motor imagery EEG recognition method based on domain adaptation network, characterized in that: The following steps are involved: Step 1: Collect the multi-channel motor imagery EEG signals of the subjects, and use the Butterworth filter bank to bandpass filter the EEG signals of each channel to obtain multiple EEG signals of different frequency bands. The EEG signals of the target subjects are used as the target domain data, and the rest are used as the source domain data; Step 2: Based on the source domain data and the target domain data, the source domain features and the target domain features are obtained respectively through the multi-view convolutional Transformer feature extractor composed of a parallel spatiotemporal convolution module and a Transformer module; Step 3: Use source domain features and target domain features to perform adversarial domain adaptation to obtain a model that can adapt to target domain data; input the target domain test data into the model to identify the motor imagery category corresponding to the EEG signal.

2. The method for motor imagery EEG recognition based on domain adaptation network according to claim 1, characterized in that: The specific implementation process of step 2 is: use parallel spatiotemporal convolution modules to extract the time domain features and spatial domain features of EEG signals in different frequency bands for the source domain data and the target domain data respectively, and add them up to obtain the spatiotemporal features of the EEG; use the Transformer module to extract the long-term dependencies contained therein after the extracted spatiotemporal features; add the long-term dependencies extracted from the spatiotemporal features of multiple frequency bands to obtain the final source domain features and target domain features.

3. The method for motor imagery EEG recognition based on domain adaptation network according to claim 2 is characterized in that: The parallel spatiotemporal convolution module consists of two parallel one-dimensional convolutions; for temporal convolution, the kernel size is (k×1×n), where k represents the number of feature maps and n represents the temporal kernel length; in spatial convolution, the kernel size is (k×m×1), where m is equal to the number of channels of the EEG signal; The temporal and spatial convolution results are then passed to the reshape convolution layer to obtain two one-dimensional feature vectors, and the two vectors are concatenated to obtain the final spatiotemporal feature vector; each convolution layer is followed by a batch normalization layer and an ELU activation function.

4. The method for motor imagery EEG recognition based on domain adaptation network according to claim 3 is characterized in that: The Transformer module is specifically implemented as follows: first, the obtained spatiotemporal features are flattened in the channel direction, and then they are divided into several slices of equal length along the time dimension, with a length of d, and each slice represents a feature representation segment; The encoder and decoder of the Transformer linearly transform slices from the two domains into a query vector Q, a key vector K of the target domain, and a value vector V of the source domain with the same size; then, the long-term dependency of the target domain is evaluated by calculating the attention between the target domain and the target domain; the attention is calculated by the following formula: On this basis, a multi-head attention mechanism is used to improve the performance of Transformer. The multi-head attention mechanism divides the generated Q, K, and V vectors into h subspaces and generates h different groups of Q, K, and V. Each head performs attention calculation independently. The multi-head attention mechanism is calculated by the following formula: MHA(Q,K,V)=[head0;…;head h-1 ] Among them, head l =Attention(xW l Q ,xW l K ,xW l V ), x is the input data, W l Q ,W l K ,W l V They are the weight matrices corresponding to the query vector Q, the key vector K of the target domain, and the value vector V of the source domain.

5. The method for motor imagery EEG recognition based on domain adaptation network according to claim 3 is characterized in that: The specific implementation process of step 3 is as follows: Construct a domain discriminator and a classifier. Both have the same structure, consisting of two fully connected layers. The domain discriminator is used to determine whether the features of the input data come from the source domain or the target domain. The feedback from the classifier is used to measure the difference in feature distribution between the two domains. In adversarial training, the feature extractor and the domain discriminator exist in opposition. The goal of the feature extractor is to extract domain-invariant features so that the domain discriminator cannot distinguish the source of the features. The classifier is used to predict the classification results. The training of the classifier guides the feature extractor to learn features that can correctly distinguish categories, so that the domain-invariant features achieved by the discriminator have category separability. During the training process, there are several goals: The first goal is to minimize the prediction loss of the classifier, and the loss function is as follows: Among them, N t 、N s are the number of target domains and source domains respectively, M is the number of categories, y ic is the actual value, is the predicted value; The second goal is to guide the feature extractor to form an adversarial relationship with the discriminator by inverting the gradient layer, maximizing the domain prediction loss of the source domain data and the target domain data, and preliminarily aligning the distribution of the source domain data and the target domain data at the domain level. The loss function is as follows: in, is the expectation, x s is the source domain data; D(·) is the discriminator; B is the number of frequency bands; MBTCN (b) (·) is the multi-view convolutional Transformer feature extractor; For the feature output of each frequency band branch, a classifier is also used to predict the classification result, and the loss function is as follows: Among them, N is the number of data, M is the number of categories, and y ic is the true value, is the label value; The final overall objective function is: Among them, λ and ω G are the corresponding weight coefficients.

Images