A cross-subject EEG motor imagery classification method based on domain adaptive dual-flow transformer

Abstract

The application discloses a cross-subject EEG motor imagery classification method based on a domain adaptive dual-flow Transformer, acquires time-domain electroencephalogram signals containing source-subject EEG data and target-subject EEG data, and carries out preprocessing, and then constructs frequency-domain electroencephalogram signals corresponding to the time-domain electroencephalogram signals through a short-time Fourier transform; a model containing a dual-flow Transformer encoding network and a classifier is used to perform representation learning on the time-domain electroencephalogram signals and the frequency-domain electroencephalogram signals of each type of data and predict motor imagery classification results, specifically including: extracting time-domain representation and frequency-domain representation through intra-flow learning, and realizing bidirectional information interaction through cross-flow learning based on the time-domain representation and the frequency-domain representation to obtain joint time-frequency representation, and predicting motor imagery classification results through a classifier based on the joint time-frequency representation. The application can improve cross-subject discrimination ability while retaining feature transferability, thereby improving EEG motor imagery classification precision.

Description

Technical Field

[0001] This invention belongs to the field of brain-computer interface and electroencephalogram (EEG) signal technology, specifically relating to a cross-subject EEG motor imagery classification method based on domain-adaptive dual-stream Transformer. Background Technology

[0002] Electroencephalography (EEG) is widely used in brain-computer interface (BCI) systems for neural decoding tasks due to its non-invasiveness, low cost, and high temporal resolution. Among these, motor imagery classification, which aims to identify the corresponding category based on a subject's imagined limb movements, is one of the core problems in BCI research. With the development of deep learning, convolutional neural networks, hybrid convolutional-Transformer networks, and pure Transformer networks have all been applied to EEG motor imagery classification, achieving good results in in-subject scenarios. Examples include a Riemannian manifold and CNNLSTM-based EEG motor imagery classification method disclosed in patent application CN116820238A, and an FB2DCNNLSTM-based EEG motor imagery classification method and system disclosed in patent application CN118332408A.

[0003] However, in cross-subject scenarios, there are significant differences in EEG distribution among different subjects. Existing methods often only emphasize domain-invariant feature learning. Although this can improve transferability to some extent, it easily sacrifices individual features that are crucial for differentiating the target subject, resulting in limited classification accuracy. Therefore, there is a need for a cross-subject EEG motor imagery classification method that can simultaneously take into account the ability to model time-frequency information, cross-stream feature interaction, and the ability to co-model domain-independent and domain-related knowledge. Summary of the Invention

[0004] In view of the above, the present invention aims to solve the problems of insufficient discriminativeness and limited cross-subject generalization ability of existing cross-subject EEG motor imagery classification methods during the transfer process, and provides a cross-subject EEG motor imagery classification method based on domain adaptive dual-stream Transformer, so as to retain the discriminative information of the target subject while improving feature transferability.

[0005] To achieve the above-mentioned objectives, an embodiment provides a cross-subject EEG motor imagery classification method based on domain-adaptive dual-stream Transformer, comprising the following steps: After acquiring and preprocessing time-domain EEG signals containing source and target subject EEG data, frequency-domain EEG signals corresponding to the time-domain EEG signals are constructed using short-time Fourier transform. The model, incorporating a two-stream Transformer coding network and a classifier, performs representation learning on the time-domain and frequency-domain EEG signals of each data category and predicts the classification results for motor imagery. Specifically, this includes: Temporal and frequency domain representations are extracted through in-stream learning, and a joint time-frequency representation is obtained through bidirectional information interaction via cross-stream learning based on these representations. In the dual-stream Transformer coding network, the feedforward network of the Transformer block is replaced with a domain-adaptive expert hybrid module. This module includes independently configured source domain routers, target domain routers, and multiple experts. The source domain router outputs the original expert routing distribution for source domain samples, and the target domain router outputs the original expert routing distribution for target domain samples. Based on the feature similarity between source and target domain samples, cross-domain guided routing distributions are constructed for both. These distributions are then fused with the corresponding original expert routing distributions to obtain a corrected expert routing distribution. Sparse Top-k expert selection is performed on this corrected distribution, and the selected experts are used for representation learning based on input features. This allows for the sharing of domain-independent knowledge while retaining domain-related discriminative information during learning. Based on joint time-frequency representation, a classifier is used to predict the classification results of motion imagery.

[0006] Preferably, the time-domain EEG signal is preprocessed, including: artifact removal, bandpass filtering, standardization, and sliding window segmentation.

[0007] Preferably, the extraction of time-domain and frequency-domain representations through in-stream learning includes: Temporal convolution, spatial convolution, and one-dimensional convolution are performed sequentially on the temporal and frequency domain EEG signals of each data type to form embedding features suitable for Transformer encoding. In both the time-domain and frequency-domain streams, multi-layer Transformer modules are used to model long-range dependencies based on embedded features through a self-attention mechanism, resulting in enhanced time-domain and frequency-domain representations.

[0008] Preferably, a joint time-frequency representation is obtained through cross-current learning based on time-domain and frequency-domain representations to achieve bidirectional information interaction, including: Using time-domain representations as queries and frequency-domain representations as keys and values, time-domain stream features are updated through a cross-attention mechanism. Frequency domain representation is used as the query, and time domain representation is used as the key and value. Frequency domain flow features are updated through a cross-attention mechanism. The time-domain flow features and frequency-domain flow features after bidirectional interaction are fused to obtain a joint time-frequency representation for final classification.

[0009] Preferably, based on the feature similarity relationship between source domain samples and target domain samples, cross-domain guided routing distributions for source domain samples and target domain samples are constructed respectively, including: For the input features of the source domain samples Input features of the target domain samples Using cosine similarity And combined with temperature coefficient Calculate the similarity weights from the source domain to the target domain. and the similarity weights from the target domain to the source domain : Using two similarity weights and the original expert route distribution of the peer domain, a cross-domain guided route distribution for source domain samples and target domain samples is constructed. and : in, and These are the original expert route distributions for the target domain samples and the source domain samples, respectively. i and j These are the source domain sample index and the target domain sample index, respectively, where m is the number of samples.

[0010] Preferably, the cross-domain guided routing distribution is fused with the corresponding original expert routing distribution to obtain a corrected expert routing distribution, including: in, and These are the corrected expert route distributions for the source domain samples and the target domain samples, respectively. The fusion coefficient has a value of [value missing]. .

[0011] Preferably, the model is optimized before being applied. The loss function used for parameter optimization includes: cross-entropy classification loss calculated on the source and target domains based on the motion imagery classification results, time-frequency consistency constraint constructed on the motion imagery classification results output by the time-domain flow branch and the frequency-domain flow branch, and cross-domain routing consistency constraint loss constraining the difference between the original expert routing distribution and the cross-domain guidance distribution.

[0012] Preferably, the time-frequency consistency constraint adopts KL divergence.

[0013] Preferably, the cross-domain routing consistency constraint loss is expressed as: : in, and Samples from the source domain i The corresponding original expert routing distribution and cross-domain bootstrapping distribution, and Samples of the target domain i The corresponding original expert routing distribution and cross-domain bootstrapping distribution, Let be the KL divergence, and m and n be the number of samples in the source and target domains, respectively.

[0014] Compared with the prior art, the beneficial effects of the present invention include at least the following: This invention combines dual-stream time-frequency modeling, cross-stream interaction, and domain-adaptive expert selection to enhance inter-domain transfer capabilities while preserving key discriminative information of the target subjects. It can significantly improve the classification performance of cross-subject EEG motor imagery and achieve classification results superior to existing methods on multiple public datasets. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a flowchart of the cross-subject EEG motion imagery classification method based on domain-adaptive dual-stream Transformer provided in the embodiment; Figure 2 This is a schematic diagram of the model structure including a two-stream Transformer coding network and a classifier provided in the embodiment; Figure 3 This is a schematic diagram of the structure of the in-stream Transformer module provided in the embodiment; Figure 4 A schematic diagram of the cross-stream Transformer module provided in the embodiment; Figure 5 A schematic diagram of the domain-adaptive expert hybrid module provided in the embodiment. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not limit the scope of protection of this invention.

[0018] like Figure 1As shown in the figure, an embodiment of the present invention proposes a cross-subject EEG motor imagery classification method based on domain-adaptive dual-stream Transformer, which includes the following steps: S1. Acquire time-domain EEG signals containing source subject EEG data and target subject EEG data, preprocess them, and then construct frequency-domain EEG signals corresponding to the time-domain EEG signals through short-time Fourier transform.

[0019] In this embodiment, source subject EEG data and target subject EEG data are first acquired and divided into source domain samples and target domain samples. These data are presented in the time domain and collectively referred to as time-domain EEG signals.

[0020] The time-domain EEG signals were then preprocessed, including artifact removal, bandpass filtering, standardization, and sliding window segmentation. More specifically, bilateral mastoid electrodes were removed from each dataset, and baseline correction and artifact removal were performed on the original time-domain EEG signals. Next, a fourth-order Butterworth bandpass filter was used to retain components in the 1Hz to 45Hz frequency band to preserve EEG information related to motor imagery and cognitive activity. Then, the time-domain EEG signals were z-score standardized. Finally, the EEG signals were segmented using a 1-second sliding window with a 50% overlap rate to increase the number of training samples and enhance the model's ability to model non-stationary EEG sequences.

[0021] Finally, a short-time Fourier transform is performed on the preprocessed time-domain EEG signal to construct its frequency-domain representation, thus obtaining the frequency-domain EEG signal corresponding to the time-domain EEG signal. The time-domain EEG signal serves as the time-domain input to the model, and the frequency-domain EEG signal serves as the frequency-domain input.

[0022] S2 utilizes a model containing a two-stream Transformer coding network and a classifier to perform representation learning on the time-domain and frequency-domain EEG signals of each class of data and predict the classification results of motor imagery.

[0023] like Figure 2 As shown, the model includes a two-stream Transformer coding network and a classifier. The two-stream Transformer coding network is used for representation learning, specifically extracting time-domain and frequency-domain representations through intra-stream learning, and obtaining a joint time-frequency representation by achieving bidirectional information interaction based on the time-domain and frequency-domain representations through cross-stream learning. The classifier is used for prediction, predicting the motion image classification result based on the joint time-frequency representation.

[0024] In this embodiment, temporal and frequency domain representations are extracted through in-stream learning. This includes: performing temporal convolution, spatial convolution, and one-dimensional convolution sequentially on the temporal and frequency domain EEG signals of each data type to form embedding features suitable for Transformer encoding, providing a unified representation for subsequent Transformer encoding; and modeling long-range temporal dependencies based on the embedding features using a self-attention mechanism within the temporal and frequency streams, respectively, to obtain enhanced temporal and frequency domain representations. The in-stream encoding structure can be found in [reference needed]. Figure 2 and Figure 3 .

[0025] In this embodiment, a joint time-frequency representation is obtained through cross-stream learning based on time-domain and frequency-domain representations to achieve bidirectional information interaction. This includes: using the time-domain representation as a query and the frequency-domain representation as a key and value, updating the time-domain stream features through a cross-attention mechanism. , ,in, For the temporal flow features corresponding to the source domain samples, The target domain samples are defined as temporal flow features; the frequency domain representation is used as the query, and the temporal representation is used as the key and value, respectively. The frequency domain flow features are updated through a cross-attention mechanism. , ,in, For the frequency domain flow features corresponding to the source domain samples, The target domain samples correspond to frequency domain flow features, thereby enabling bidirectional information exchange and complementary learning. The cross-flow interaction structure can be found in [reference needed]. Figure 2 and Figure 4 Finally, the time-domain and frequency-domain flow features, after bidirectional interaction, are concatenated and fused through a unified self-attention mechanism to obtain a joint time-frequency representation for final classification. and Among them, the joint time-frequency representation corresponding to the source domain samples for and The fusion results, the joint time-frequency representation of the target domain samples for and The fusion result; In this embodiment, the feedforward network of the Transformer block in the dual-stream Transformer coding network is replaced with a Domain Adaptive Expert Hybrid Module (DA-MOE), such as... Figure 5 As shown, the domain adaptive expert hybrid module includes independently configured source domain routers, target domain routers, and multiple experts, each of which uses a lightweight network.

[0026] Let the input features of the source domain samples be... The input features of the target domain samples are The number of experts is .

[0027] First, the source domain router The original expert route distribution of the output source domain samples Target domain router Output the original expert route distribution of the target domain samples : Then, based on the feature similarity relationship between the source domain samples and the target domain samples, cross-domain guided routing distributions for the source domain samples and the target domain samples are constructed respectively, specifically including: For the input features of the source domain samples Input features of the target domain samples Using cosine similarity And combined with temperature coefficient Calculate the similarity weights from the source domain to the target domain. and the similarity weights from the target domain to the source domain : Using two similarity weights and the original expert route distribution of the peer domain, a cross-domain guided route distribution for source domain samples and target domain samples is constructed. and : in, i and j These are the source domain sample index and the target domain sample index, respectively, where m is the number of samples.

[0028] Next, the cross-domain guided route distribution is merged with the corresponding original expert route distribution to obtain the corrected expert route distribution, which specifically includes: in, and These are the corrected expert route distributions for the source domain samples and the target domain samples, respectively. The fusion coefficient has a value of [value missing]. .

[0029] Finally, sparse Top-k expert selection is performed on the corrected expert routing distribution, and the selected experts are used for representation learning based on input features. This allows for the sharing of domain-independent knowledge while preserving domain-related discriminative information during learning. Taking the source domain samples as an example, the final set of activated experts is: This mechanism enables samples from different domains to share cross-domain guided knowledge through feature similarity, while retaining the original discriminative information relevant to their respective domains, thereby reducing the negative transfer impact of irrelevant domain knowledge. The final expert output is: in Indicates the first A network of experts express The One portion, The output features are those of the experts; the same applies to the target domain samples.

[0030] In this embodiment, the model is optimized before being applied, and the loss function used for parameter optimization includes: based on joint time-frequency representation. and Predicted motion imagery classification results and Cross-entropy classification loss calculated separately in the source and target domains. and ,in, and The true classification labels are given in the source and target domains, respectively, and the total cross-entropy classification loss is given. ; Simultaneously, the motion imagination classification results output by the time-domain flow branch and the frequency-domain flow branch are analyzed. and , and Constructed time-frequency consistency constraints and ,in, and Based on time-domain flow features , Predicted motor imagery classification results and Based on frequency domain flow characteristics , Predicted motion imagery classification results, total time-frequency consistency constraint The time-domain flow branch is better at preserving global structure and domain-invariant information and has strong transferability; the frequency-domain flow branch is better at capturing class discrimination patterns in specific frequency bands and has strong discriminative power. Therefore, the frequency-domain flow branch is used to guide the time-domain flow branch in the source domain, and the time-domain flow branch is used to guide the frequency-domain flow branch in the target domain.

[0031] The loss function used also includes a cross-domain routing consistency constraint loss that constrains the difference between the original expert route distribution and the cross-domain guidance distribution. To further narrow the gap between the original expert route distribution and the cross-domain guided route distribution, the expression is: in, Let be the Kullback-Leibler divergence, and m and n be the number of samples in the source and target domains, respectively. Finally, the classification loss is used. Time-frequency consistency loss Cross-domain routing consistency constraints The weighted sum is the total loss. Represented as: in and Using the weight coefficients, the network parameters are updated through backpropagation to obtain a cross-subject EEG motor imagery classification model that combines transferability and discriminativeness.

[0032] Application examples To verify the effectiveness of the method of this invention, experiments were conducted on three publicly available EEG motor imagery datasets: BCI Competition IV-2a, BCI Competition IV-2b, and OpenBMI. BCI IV-2a included 9 subjects, 22 channels, and 4 types of motor imagery tasks; BCI IV-2b included 9 subjects, 3 channels, and 2 types of motor imagery tasks; OpenBMI included 54 subjects, 62 channels, downsampled to 250Hz, and included left and right hand motor imagery tasks. All data underwent the same preprocessing procedures as described above.

[0033] This invention employs leave-one-out cross-validation for evaluation: one subject is selected as the target domain at each iteration, and the remaining subjects are used as the source domain; 80% of the target domain samples are used for training, and 20% are used for testing. During training, the Adam optimizer is used with a batch size of 64, a learning rate of 0.0001, and a maximum training duration of 300 epochs. An early stopping strategy is employed to prevent overfitting. Comparison methods include existing methods such as DeepConvNet, EEGNet, MMCNN, DRDA, DJ-DAN, and DA-QuadViT, with the evaluation metric being percentage accuracy.

[0034] Tables 1 and 2 show the comparative experimental results and ablation experimental results of the method of the present invention on public datasets.

[0035] Table 1: Comparative Experiment Results of EEG Motor Imagery Classification Across Participants Table 2: Results of ablation experiments across subject EEG motor imagery classification The results show that the average accuracy of the invention described in Table 1 on the BCI IV-2a and BCI IV-2b datasets reached 85.4% and 89.3%, respectively, which are improvements of 1.3% and 1.5% compared to DA-QuadViT. Table 2 further shows that cross-stream learning, domain adaptive expert hybrid module (DA-MoE), and time-frequency consistency loss can all bring stable gains. Among them, the invention achieved an accuracy of 82.1% on the OpenBMI dataset, and the improvement after adding DA-MoE was the most significant, indicating that the invention can simultaneously enhance the time-frequency complementary representation ability and the cross-subject domain adaptation ability.

[0036] The specific embodiments described above illustrate the technical solution and beneficial effects of the present invention in detail. It should be understood that the above description is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, additions, and equivalent substitutions made within the scope of the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A cross-subject EEG motor imagery classification method based on domain-adaptive dual-stream Transformer, characterized in that, Includes the following steps: After acquiring and preprocessing time-domain EEG signals containing source and target subject EEG data, frequency-domain EEG signals corresponding to the time-domain EEG signals are constructed using short-time Fourier transform. The model, incorporating a two-stream Transformer coding network and a classifier, performs representation learning on the time-domain and frequency-domain EEG signals of each data category and predicts the classification results for motor imagery. Specifically, this includes: Temporal and frequency domain representations are extracted through in-stream learning, and a joint time-frequency representation is obtained through bidirectional information interaction via cross-stream learning based on these representations. In the dual-stream Transformer coding network, the feedforward network of the Transformer block is replaced with a domain-adaptive expert hybrid module. This module includes independently configured source domain routers, target domain routers, and multiple experts. The source domain router outputs the original expert routing distribution for source domain samples, and the target domain router outputs the original expert routing distribution for target domain samples. Based on the feature similarity between source and target domain samples, cross-domain guided routing distributions are constructed for both. These distributions are then fused with the corresponding original expert routing distributions to obtain a corrected expert routing distribution. Sparse Top-k expert selection is performed on this corrected distribution, and the selected experts are used for representation learning based on input features. This allows for the sharing of domain-independent knowledge while retaining domain-related discriminative information during learning. Based on joint time-frequency representation, a classifier is used to predict the classification results of motion imagery.

2. The cross-subject EEG motor imagery classification method based on domain-adaptive dual-stream Transformer according to claim 1, characterized in that, Preprocessing of time-domain EEG signals includes: artifact removal, bandpass filtering, standardization, and sliding window segmentation.

3. The cross-subject EEG motor imagery classification method based on domain-adaptive dual-stream Transformer according to claim 1, characterized in that, In-stream learning is used to extract temporal and frequency domain representations, including: Temporal convolution, spatial convolution, and one-dimensional convolution are performed sequentially on the temporal and frequency domain EEG signals of each data type to form embedding features suitable for Transformer encoding. In both the time-domain and frequency-domain streams, multi-layer Transformer modules are used to model long-range dependencies based on embedded features through a self-attention mechanism, resulting in enhanced time-domain and frequency-domain representations.

4. The cross-subject EEG motor imagery classification method based on domain-adaptive dual-stream Transformer according to claim 1, characterized in that, Joint time-frequency representations are obtained through cross-current learning based on time-domain and frequency-domain representations to achieve bidirectional information exchange, including: Using time-domain representations as queries and frequency-domain representations as keys and values, time-domain stream features are updated through a cross-attention mechanism. Frequency domain representation is used as the query, and time domain representation is used as the key and value. Frequency domain flow features are updated through a cross-attention mechanism. The time-domain flow features and frequency-domain flow features after bidirectional interaction are fused to obtain a joint time-frequency representation for final classification.

5. The cross-subject EEG motor imagery classification method based on domain-adaptive dual-stream Transformer according to claim 1, characterized in that, Based on the feature similarity relationship between source domain samples and target domain samples, cross-domain guided route distributions for source domain samples and target domain samples are constructed respectively, including: For the input features of the source domain samples Input features of the target domain samples Using cosine similarity And combined with temperature coefficient Calculate the similarity weights from the source domain to the target domain. and the similarity weights from the target domain to the source domain : Using two similarity weights and the original expert route distribution of the peer domain, a cross-domain guided route distribution for source domain samples and target domain samples is constructed. and : in, and These are the original expert route distributions for the target domain samples and the source domain samples, respectively. i and j These are the source domain sample index and the target domain sample index, respectively, where m is the number of samples.

6. The cross-subject EEG motor imagery classification method based on domain-adaptive dual-stream Transformer according to claim 5, characterized in that, The cross-domain guided route distribution is merged with the corresponding original expert route distribution to obtain the corrected expert route distribution, including: in, and These are the corrected expert route distributions for the source domain samples and the target domain samples, respectively. The fusion coefficient has a value of [value missing]. .

7. The cross-subject EEG motor imagery classification method based on domain-adaptive dual-stream Transformer according to claim 1, characterized in that, The model was optimized before being applied. The loss function used for parameter optimization included: cross-entropy classification loss calculated on the source and target domains based on the motion imagery classification results, time-frequency consistency constraint constructed on the motion imagery classification results output by the time-domain flow branch and the frequency-domain flow branch, and cross-domain routing consistency constraint loss constraining the difference between the original expert routing distribution and the cross-domain guidance distribution.

8. The cross-subject EEG motor imagery classification method based on domain-adaptive dual-stream Transformer according to claim 7, characterized in that, The time-frequency consistency constraint uses KL divergence.

9. The cross-subject EEG motor imagery classification method based on domain-adaptive dual-stream Transformer according to claim 7, characterized in that, The cross-domain routing consistency constraint loss is expressed as follows: : in, and Samples from the source domain i The corresponding original expert routing distribution and cross-domain bootstrapping distribution, and Samples of the target domain i The corresponding original expert routing distribution and cross-domain bootstrapping distribution, Let be the KL divergence, and m and n be the number of samples in the source and target domains, respectively.

Images