Motor imagery eeg recognition method based on domain adaptation network

Abstract

The application discloses a motor imagery electroencephalogram recognition method based on a domain adaptation network, which collects multi-channel motor imagery electroencephalogram (EEG) signals of a subject, and carries out band-pass filtering to obtain EEG signals in multiple different frequency bands; the EEG signals of a target subject are taken as target domain data, and the rest are taken as source domain data. Secondly, based on the source domain data and the target domain data, the source domain features and the target domain features are obtained in sequence through a multi-view convolution Transformer feature extractor composed of a parallel space-time convolution module and a Transformer module. Finally, the source domain features and the target domain features are used for adversarial domain adaptation to obtain a model capable of adapting to the target domain data, and the motor imagery category corresponding to the electroencephalogram signals is recognized. The application is helpful to guide the source domain data distribution to shift to the target domain, and improves the classification accuracy of cross-subject motor imagery electroencephalogram signals.

Description

Technical Field

[0001] The invention belongs to the field of pattern recognition and proposes a motor imagery EEG recognition method based on a domain adaptation network. Background Art

[0002] EEG signals are spontaneous electrophysiological signals that are unaffected by subjective factors and can reflect the subject's true motor imagery state. With the development of non-invasive electrode technology, effective EEG signals can be collected simply by wearing a simple EEG cap. Therefore, EEG-based motor imagery recognition methods offer greater accuracy and reliability, overcoming the limitations of traditional motor imagery recognition methods.

[0003] As deep neural networks have made significant progress in computer vision tasks, more and more researchers are applying deep learning methods to motor imagery EEG recognition. While many deep learning methods can extract effective features from subjects, achieving high classification accuracy, these models are typically trained or targeted at a single subject and cannot be applied to other subjects. Due to individual variability in EEG signals, motor imagery recognition models require stronger generalization capabilities for cross-subject problems. Therefore, while extracting subject features, it is also necessary to extract features that are invariant across domains, necessitating domain adaptation. Generally speaking, there are two main approaches to domain adaptation: moment matching and adversarial learning.

[0004] Moment matching methods typically quantify domain differences through statistical moments between the source and target domains, introducing these differences into deep networks in the form of a loss function. This directly reduces domain shift and aligns the distributions of source and target domain features. Adversarial learning methods employ an adversarial strategy between a feature generator and a discriminator. This strategy makes it impossible to distinguish whether the features extracted by the feature generator originate from source or target domain data, thereby achieving cross-domain invariance and reducing the difference in feature distributions between the source and target domains. Inspired by generative adversarial networks (GANs), researchers proposed domain-adversarial neural networks (DANNs). These networks introduce an additional domain-level discriminator to determine whether features originate from source or target domain samples. Using a gradient reversal layer during backpropagation, they induce the feature generator to ultimately extract features that are indistinguishable at the domain level. Summary of the Invention

[0005] In response to the above problems, the present invention proposes a method for motor imagery EEG recognition based on a domain adaptation network, which adds a multi-view convolution module and a Transformer module to the adversarial domain adaptation network, extracts the long-term dependency of the EEG after extracting the multi-band view information of the EEG, fully mines the time domain, frequency domain, and spatial domain features of the EEG signal, and then uses the adversarial domain adaptation method to generate domain-invariant features, thereby realizing cross-subject motor imagery EEG signal recognition. Since the present invention fully mines the feature information of the EEG signal on the basis of the deep adversarial domain adaptation method, it helps to transfer the source domain features to the target domain more easily, thereby improving the classification performance of the model. The present invention first divides the original EEG signal into multi-frequency views according to different frequencies, uses the spatiotemporal convolution module and the Transformer module to extract the spatiotemporal features of the EEG signal of each view, and then introduces the discriminator and classifier to construct a domain adaptation network for adversarial training. Dividing EEG signals into multiple frequency views helps to mine their frequency feature information; using a parallel spatiotemporal convolution module to extract spatiotemporal features helps to retain the complete temporal and spatial feature structure; using the Transformer module to further extract long-term dependencies in the features, while using the attention mechanism to guide the migration of source domain features to the target domain; finally, through domain adaptation, the feature distribution of the source and target domains is gradually reduced, thereby improving the recognition performance of the model in the target domain.

[0006] The present invention comprises the following steps:

[0007] Step (1): Multi-channel EEG signal acquisition and preprocessing. First, the multi-channel motor imagery EEG signal of the subject is acquired. Then, the EEG signal of each channel is bandpass filtered using a Butterworth filter bank to obtain multiple EEG signals of different frequency bands. The EEG signal of the target subject is used as the target domain data, and the rest is used as the source domain data.

[0008] Step (2): Based on the source domain data and the target domain data, the source domain features and the target domain features are obtained respectively through the multi-view convolutional Transformer feature extractor composed of a parallel spatiotemporal convolution module and a Transformer module.

[0009] Parallel spatiotemporal convolution modules are used to extract the temporal and spatial features of EEG signals in different frequency bands from the source domain data and target domain data respectively, and then add them together to obtain the spatiotemporal features of the EEG. After the spatiotemporal features are extracted, the Transformer module is used to extract the long-term dependencies contained therein. The long-term dependencies extracted from the spatiotemporal features of multiple frequency bands are added together to obtain the final source domain features and target domain features.

[0010] Step (3): Using source domain features and target domain features for adversarial domain adaptation, the feature extractor and discriminator are guided to form an adversarial relationship through the inverted gradient layer, maximizing the domain prediction loss of the source domain data and the target domain data, aligning the marginal distribution of the source domain data and the target domain data, and aligning the conditional distribution between different categories through the supervision of the classifier on the category prediction loss. Finally, a model that can adapt to the target domain data is obtained. The target domain test data is input into the model to identify the motor imagery category corresponding to the EEG signal, such as left hand movement and right hand movement.

[0011] The beneficial effects of the present invention are as follows: the domain adaptation network based on multi-view convolutional Transformer can fully explore the inherent time domain, frequency domain, and spatial domain information of EEG signals, and extract the long-term dependencies of EEG signals on this basis. By learning domain-invariant features through adversarial domain adaptation training, it helps to guide the transfer of source domain data distribution to the target domain, thereby improving the classification accuracy of motor imagery EEG signals across subjects. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 Flowchart of the domain adaptation network based on multi-view convolutional Transformer;

[0013] Figure 2 This is the structure diagram of the parallel spatiotemporal convolution module;

[0014] Figure 3 This is the Transformer module structure diagram. DETAILED DESCRIPTION

[0015] This paper presents a motor imagery EEG recognition method based on a domain adaptation network. The proposed method is evaluated using the BCI competition IV2a public dataset. This EEG dataset, provided by the Graz University of Technology, contains nine subjects (S01-S09) using 25 Ag / AgCl electrodes at a sampling rate of 250Hz. Four different motor imagery tasks were performed in two different sessions, including imagining movements of the left hand, right hand, feet, and tongue.

[0016] The present invention is described in detail below with reference to the accompanying drawings, which is a method for motor imagery EEG recognition based on a domain adaptation network of a multi-view convolutional Transformer.

[0017] Step (1): EEG signal acquisition and preprocessing.

[0018] The original EEG signal was filtered with a 50 Hz notch filter and then an overlapping third-order Butterworth filter was used to decompose the original EEG signal into four sub-band signals: 0-4 Hz, 2-10 Hz, 6-22 Hz, and 16-38 Hz.

[0019] Finally, the filtered data is z-score normalized to reduce non-stationarity:

[0020]

[0021] where x i and x o Represent the bandpass filtered data and the normalized data respectively. μ and σ 2 represents the mean and variance, which are calculated from the filtered data.

[0022] Step (2): Use multi-view convolutional Transformer to extract features.

[0023] The overall flow chart of the present invention is as follows Figure 1 There are two main steps in the feature extraction process. The first step uses a parallel spatiotemporal convolution module to extract spatiotemporal features from the preprocessed multi-band data.

[0024] First, the preprocessed data obtained in step (1) is fed into the parallel spatiotemporal convolution module, as Figure 2 As shown, the spatiotemporal features are initially extracted, and the module consists of two parallel one-dimensional convolutions. For temporal convolution, the kernel size is (k×1×n), where k represents the number of feature maps and n represents the length of the temporal kernel. In spatial convolution, the kernel size is (k×m×1), where m is usually equal to the number of channels C of the EEG signal. The temporal and spatial convolution results are then passed to the reshaped convolution layer to obtain two one-dimensional feature vectors, and the two vectors are concatenated to obtain the final spatiotemporal feature vector. Each convolution layer is followed by a batch normalization layer and an ELU activation function to enhance the nonlinear representation capability of the network. Compared with the disadvantage that traditional feature extractors may destroy the original feature structure information after applying spatial convolution to the temporal feature output, the parallel strategy used in the present invention can retain the integrity of the spatiotemporal feature structure to the greatest extent. Therefore, the use of parallel spatiotemporal convolution for each frequency branch can simultaneously retain temporal and spatial structural information, which is conducive to feature transfer between different domains.

[0025] Secondly, the spatiotemporal feature vector obtained by the parallel spatiotemporal convolution module is input into the Transformer module, such as Figure 3As shown, long-term dependencies in EEG signals are further extracted. These features are guided from the source domain to the target domain through the scaled dot product attention mechanism between the encoder and decoder in the Transformer. Specifically, the spatiotemporal features obtained in the previous step are first flattened in the channel direction, and then they are split into several slices of equal length along the time dimension, with a length of d, each slice representing a feature representation segment. The encoder and decoder of the Transformer are used to linearly transform the slices from the two domains into a query vector (Q) of the same size, a key vector (K) of the target domain, and a value vector (V) of the source domain. The long-term dependencies of the target domain are then evaluated by calculating the attention between the target domain and the target domain. Attention can be calculated by the following formula:

[0026]

[0027] On this basis, the multi-head attention mechanism (MHA) is used to improve the performance of the Transformer. The multi-head attention mechanism divides the generated Q, K, and V vectors into h subspaces, generating h different sets of Q, K, and V. Each head performs independent attention calculations, allowing the model to focus on different parts of the input sequence in different subspaces. The diversity of the multi-head attention mechanism can capture contextual information at different levels, thereby improving the model's understanding of the input information and providing more comprehensive contextual features for subsequent layers. The multi-head attention mechanism can be calculated using the following formula:

[0028] MHA(Q,K,V)=[head0;…;head h-1 ], (3)

[0029] Among them, head l =Attention(xW l Q ,xW l K ,xW l V ), x is the input data, W l Q ,W l K ,W l V are the weight matrices corresponding to the query vector Q, the key vector K of the target domain, and the value vector V of the source domain, respectively.

[0030] Step (3): Adversarial training using domain discriminator and classifier

[0031] To perform adversarial training, a domain discriminator and a classifier are constructed. They have similar structures, both consisting of two fully connected layers. The domain discriminator is used to determine whether the features of the input data come from the source domain or the target domain. The feedback from this classifier can be used to measure the difference in feature distribution between the two domains. In adversarial training, the feature extractor and the domain discriminator exist in opposition. The feature extractor's goal is to extract domain-invariant features that prevent the domain discriminator from distinguishing the source of the features (source or target domain). Through this adversarial training, the feature extractor gradually learns features that are applicable to both the source and target domains. The classifier is used to predict classification results. Training the classifier guides the feature extractor to learn features that can correctly distinguish categories, so that the domain-invariant features achieved by the discriminator are class separable and can better perform classification tasks. Specifically, the following goals are mainly achieved during the training process.

[0032] The first goal is to minimize the prediction loss of the classifier, and the loss function is as follows:

[0033]

[0034] Among them, N t 、N s are the number of target domains and source domains respectively, M is the number of categories, y ic is the actual value, is the predicted value.

[0035] The second goal is to guide the feature extractor to form an adversarial relationship with the discriminator by inverting the gradient layer, maximizing the domain prediction loss of the source domain data and the target domain data, and preliminarily aligning the distribution of the source domain data and the target domain data at the domain level. The loss function is as follows:

[0036]

[0037] in, is the expectation, x s is the source domain data; D(·) is the discriminator; B is the number of frequency bands; MBTCN (b) (·) is the multi-view convolutional Transformer feature extractor.

[0038] For the feature output of each frequency band branch, a classifier is also used to predict the classification result. The loss function is as follows:

[0039]

[0040] Among them, N is the number of data, M is the number of categories, and y ic is the true value, is the tag value.

[0041] The final objective function is:

[0042]

[0043] In order to verify the rationality of the motor imagery EEG recognition method based on the domain adaptation network of the multi-view convolutional Transformer proposed in this paper, a variety of methods were compared, as shown in Table 1.

[0044] Table 1 Classification results of similar methods on the BCI competition IV 2a dataset

[0045] method S01 S02 S03 S04 S05 S06 S07 S08 S09 average FBCSP 76 56.5 81.25 61 55 45.25 82.75 81.25 70.75 67.75 ssCSP 76.74 58.68 81.25 57.64 38.54 48.26 76.39 79.17 78.82 66.17 SSMM 82.64 60.76 85.76 67.01 58.68 54.51 90.97 81.25 79.51 73.45 ConvNet 76.39 55.21 89.24 74.65 56.94 54.17 92.71 77.08 76.39 72.53 MI-CNN 73.26 28.82 89.58 68.06 26.39 28.82 75.35 78.82 77.08 60.69 C2CM 87.5 65.28 90.28 66.67 62.5 45.49 89.58 83.33 79.51 74.46 SHNN 82.76 68.97 79.31 65.52 58.62 48.28 86.21 89.66 89.87 74.26 DRDA 83.19 55.14 87.43 75.28 62.29 57.15 86.18 83.61 82 74.74 GAT 88.89 61.11 93.4 71.86 50.35 60.07 89.58 87.5 86.46 76.58 The present invention 93.75 65.63 90.63 71.88 56.25 59.38 90.28 87.5 84.38 77.74

[0046] To comprehensively evaluate the performance of our framework, we compared our proposed method with state-of-the-art methods on the same dataset, including hand-crafted feature extraction methods FBCSP and ssCSP, SSMM, end-to-end convolutional neural networks (ConvNets), MI-CNNs, C2CMs, and SHNNs, as well as adversarial learning-based domain adaptation models DRDA and GAT. The classification performance for each subject, as well as the average results, are shown in Table 1. The proposed method demonstrated excellent performance on this dataset, achieving the highest average accuracy of 77.74%. This demonstrates that our method significantly outperforms hand-crafted feature extraction methods such as FBCSP and ssCSP, as well as end-to-end CNN-based models such as ConvNets and MI-CNNs. Despite combining the advantages of traditional feature extractors and CNNs, along with a fine-tuned model for each subject, C2CMs and SHNNs still fall short compared to our proposed method. Compared to DRDA and GAT, our proposed method not only focuses on the long-term dependencies of the target subject's EEG data using Transformer blocks but also leverages the frequency characteristics of the EEG, resulting in average accuracy improvements of 3.00% and 1.16%, respectively, which strongly validates our approach.

[0047] Overall, in order to address the challenges brought about by individual differences and instability of EEG signals, the present invention proposes a new adversarial domain adaptation network. This method uses multi-view parallel spatiotemporal convolution to extract the spatiotemporal structural information of EEG signals. In addition, the method also combines the Transformer module to extract long-term dependencies from EEG signals and aligns the feature distributions between the source domain and the target domain. The experimental results of the present invention clearly support the effectiveness of this method. Through the multi-view parallel spatiotemporal convolution module and the Transformer module, the present invention can successfully capture the cross-domain invariant features of EEG signals of different subjects, thereby realizing domain adaptation across subject tasks, and providing new ideas for subsequent adversarial domain adaptation network methods.

Claims

1. A motor imagery EEG recognition method based on domain adaptation network, characterized by: The following steps are involved: Step 1: Collect the subject's multi-channel motor imagery EEG signal and use the Butterworth filter bank to bandpass filter the EEG signal of each channel to obtain multiple EEG signals in different frequency bands. The EEG signal of the target subject is used as the target domain data, and the rest are used as the source domain data. Step 2: Based on the source domain data and target domain data, the source domain features and target domain features are obtained respectively through the multi-view convolutional Transformer feature extractor composed of a parallel spatiotemporal convolution module and a Transformer module. The specific implementation process is as follows: Parallel spatiotemporal convolution modules are used to extract temporal and spatial features of EEG signals in different frequency bands from the source and target domain data, respectively, and these features are summed up to obtain the spatiotemporal features of the EEG. After the spatiotemporal features are extracted, the Transformer module is used to extract the long-term dependencies contained therein. The long-term dependencies extracted from the spatiotemporal features of multiple frequency bands are summed up to obtain the final source and target domain features. The Transformer module is specifically implemented as follows: first, the obtained spatiotemporal features are flattened along the channel direction, and then they are split into several slices of equal length along the time dimension, with a length of d, where each slice represents a feature representation segment; The Transformer’s encoder and decoder linearly transform slices from both domains into a query vector Q, a key vector K of the target domain, and a value vector V of the source domain with the same size; the long-term dependency of the target domain is then evaluated by calculating the attention between the target and source domains; the attention is calculated by the following formula: On this basis, a multi-head attention mechanism is used to improve the performance of Transformer. The multi-head attention mechanism divides the generated Q, K, and V vectors into h subspaces, generating h different groups of Q, K, and V. Each head performs attention calculations independently. The multi-head attention mechanism is calculated using the following formula: MHA(Q,K,V)=[head0;…;head h-1 ] Among them, head l =Attention(xW l Q ,xW l K ,xW l V ), x is the input data, W l Q ,W l K ,W l V are the weight matrices corresponding to the query vector Q, the key vector K of the target domain, and the value vector V of the source domain; Step 3: Use source domain features and target domain features to perform adversarial domain adaptation to obtain a model that can adapt to target domain data; input the target domain test data into the model to identify the motor imagery category corresponding to the EEG signal.

2. The method for motor imagery EEG recognition based on domain adaptation network according to claim 1, characterized in that: The parallel spatiotemporal convolution module consists of two parallel one-dimensional convolutions; for temporal convolution, the kernel size is (k×1×n), where k represents the number of feature maps and n represents the temporal kernel length; in spatial convolution, the kernel size is (k×m×1), where m is equal to the number of channels of the EEG signal; The temporal and spatial convolution results are then passed to the reshape convolution layer to obtain two one-dimensional feature vectors, and the two vectors are concatenated to obtain the final spatiotemporal feature vector; each convolution layer is followed by a batch normalization layer and an ELU activation function.

3. The method for motor imagery EEG recognition based on domain adaptation network according to claim 2, characterized in that: The specific implementation process of step 3 is as follows: A domain discriminator and classifier are constructed. Both have the same structure, consisting of two fully connected layers. The domain discriminator determines whether the features of the input data come from the source domain or the target domain. Feedback from the classifier is used to measure the difference in feature distribution between the two domains. In adversarial training, the feature extractor and the domain discriminator are in opposition. The feature extractor aims to extract domain-invariant features that prevent the domain discriminator from distinguishing the source of the features. The classifier is used to predict the classification results. Classifier training guides the feature extractor to learn features that can correctly distinguish categories, making the domain-invariant features achieved by the discriminator classifiable. During the training process, there are several goals: The first goal is to minimize the prediction loss of the classifier, and the loss function is as follows: Among them, N t 、N s are the number of target domains and source domains respectively, M is the number of categories, y ic is the actual value, is the predicted value; The second goal is to guide the feature extractor to form an adversarial relationship with the discriminator by inverting the gradient layer, maximizing the domain prediction loss of the source domain data and the target domain data, and preliminarily aligning the distribution of the source domain data and the target domain data at the domain level. The loss function is as follows: in, is the expectation, x s is the source domain data; D(·) is the discriminator; B is the number of frequency bands; MBTCN (b) (·) is the multi-view convolutional Transformer feature extractor; For the feature output of each frequency band branch, a classifier is also used to predict the classification result. The loss function is as follows: Among them, N is the number of data, M is the number of categories, and y ic is the true value, is the tag value; The final objective function is: Among them, λ and ω G are the corresponding weight coefficients.

Images