Specialized Transform model for motor imagery electroencephalogram decoding and hierarchical transfer learning method
By specializing the Transformer model and using hierarchical transfer learning, the problems of low signal-to-noise ratio, large individual differences, and noise pollution in motor imagery EEG decoding were solved, improving decoding accuracy and model adaptability, reducing the need for personalized calibration, and realizing a high-performance brain-computer interface system.
Patent Information
- Application Number
- CN202511656922.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-13
- Publication Date
- 2026-02-10
AI Technical Summary
Existing technologies for decoding motor imagery EEG suffer from problems such as low signal-to-noise ratio, large individual differences, severe noise pollution, and insufficient model generalization ability, resulting in low decoding accuracy and the need for a large amount of personalized calibration.
A specialized Transformer model, including the MSCHA module, KATS encoder, and classification module, is adopted. Through multi-scale temporal feature extraction, hybrid attention fusion, spatial feature integration and downsampling, combined with a global-local fusion attention mechanism and a specialized KANS layer, the long- and short-range dependencies of EEG signals are captured, and the model is adapted through a hierarchical transfer learning method.
It improved the decoding accuracy within subjects, reduced the need for personalized calibration, and achieved a high-performance and highly adaptable brain-computer interface system.
Smart Images
Figure CN121502472A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of brain-computer interface (BCI) technology, specifically to a specialized Transformer model and hierarchical transfer learning method for decoding motor imagery EEG. Background Technology
[0002] Brain-computer interface (BCI) technology has expanded new methods for connecting the brain and external devices. BCI can monitor, record, and interpret motor signals from the cerebral cortex and translate them into control commands, enabling control of external devices and breaking away from the traditional reliance on nerves and muscles. Electroencephalography (EEG) technology uses scalp electrodes to record brain electrical signals, and its millisecond-level temporal resolution, non-invasive detection, low cost, and ease of operation have made it an important tool for studying neural activity. The motor imagery (MI) paradigm can induce characteristic neural electrical activity in the sensorimotor cortex by allowing subjects to actively imagine limb movements. The analysis and use of these spontaneous signals directly related to motor intention are important technical means for EEG-BCI in fields such as neurorehabilitation and prosthetic control.
[0003] Despite the immense potential of MI-EEG in the field of brain-computer interfaces, several key challenges remain in accurately decoding user signals. EEG signals inherently possess an extremely low signal-to-noise ratio, making them highly susceptible to noise contamination from electromyographic activity, environmental electromagnetic interference, and eye-movement artifacts, directly impacting signal quality and decoding accuracy. Furthermore, in motor imagery tasks, EEG signals exhibit significant inter-individual variability, and even the same subject may produce markedly different neural responses at different times. These issues further complicate EEG decoding, necessitating personalized recalibration of the BCI system before each reuse.
[0004] Traditional machine learning methods are widely used in MI-EEG signal classification, typically following a two-stage processing flow: feature extraction and classifier construction. In the feature extraction stage, Common Spatial Patterns (CSP) and its improved algorithms, such as Filter Bank CSP (FBCSP), are commonly used for spatial filtering and feature extraction in specific frequency bands. Short Time Fourier Transform (STFT) divides the time-domain signal into fixed-duration windows and performs a Fourier transform within each window to extract time-frequency features. In the classification stage, Support Vector Machines (SVM) demonstrate good generalization ability on small sample data, while Decision Trees (DT) possess excellent ability to distinguish sample categories by recursively partitioning the feature space to build a branching structure. These are all widely used as classifiers for motion image signals.
[0005] While these methods are highly effective, their heavy reliance on manual feature design makes them sensitive to individual differences, thus limiting generalization ability. Deep learning methods, on the other hand, automatically extract task features directly from raw EEG signals, avoiding the limitations of manual feature design and improving model performance through end-to-end training. For example, Schirrmeister et al. developed an end-to-end convolutional neural network that learns hierarchical feature representations directly from raw EEG signals, outperforming FBCSP-based methods. Lawhern's lightweight model, EEGNet, employs depthwise separable convolutions to achieve efficient feature extraction with fewer parameters and demonstrates robustness across various BCI paradigms. Wang et al. investigated transform-domain CNN methods based on STFT and CWT for classification tasks of motor imagery EEG signals.
[0006] However, while single-scale CNNs perform well in decoding motor imagery EEG signals, their spatiotemporal feature extraction capabilities are insufficient, making it difficult to effectively handle individual differences. Furthermore, limited receptive fields may affect the modeling of long-range dependencies, thus reducing classification accuracy. To address these limitations, researchers have combined CNNs with recurrent neural networks (RNNs) and long short-term memory networks (LSTMs). For example, after extracting spatial features using CNNs, LSTMs are used to capture long-term temporal associations, improving the decoding accuracy of dynamic MI processes. Meanwhile, Transformer-based models effectively capture global spatiotemporal dependencies of EEG signals using self-attention mechanisms. This architecture can model long-range interactions between different brain regions, significantly improving feature representation capabilities. In addition, the Conformer model proposed by Song et al. integrates CNNs and Transformers, first extracting local time-frequency features and then modeling global dependencies through attention mechanisms, effectively overcoming the limitations of CNNs in modeling long-range associations and achieving excellent performance in MI-EEG decoding. Training deep learning models requires a large amount of high-quality labeled data, but collecting MI-EEG data is time-consuming and labor-intensive. Furthermore, existing Transformer-based methods still have shortcomings in balancing local and global features, and tend to overlook key details, thus limiting further performance improvements.
[0007] Therefore, we propose a specialized Transformer model and a hierarchical transfer learning method for decoding motor imagery EEG to address the above problems. Summary of the Invention
[0008] (a) Technical problems to be solved
[0009] To address the shortcomings of existing technologies, this invention provides a specialized Transformer model and hierarchical transfer learning method for decoding motor imagery EEG, thus solving the problems mentioned in the background section.
[0010] (II) Technical Solution
[0011] To achieve the above objectives, the present invention specifically adopts the following technical solution:
[0012] A specialized Transformer model for decoding motor imagery EEG includes an MSCHA module, a KATS encoder, and a classification module connected in sequence.
[0013] The MSCHA module, as the front end of the model, performs preliminary but comprehensive feature extraction and enhancement on the input raw MI-EEG signal, providing high-quality feature sequences for the subsequent KATS module. The MSCHA module includes a multi-scale temporal feature extraction unit, a hybrid attention fusion unit, and a spatial feature integration and downsampling unit.
[0014] The KATS encoder consists of four identical KATS modules stacked together. The KATS module is the core encoding unit of the model, which performs deep temporal dependency modeling and nonlinear transformation on the feature sequence. Each KATS module adopts a global-local fusion attention mechanism to capture long-range and short-range dependencies at the same time, and uses a B-spline-based specialized KANS layer to accurately fit the key nonlinear dynamics in the EEG.
[0015] The classification module is responsible for converting the deep feature sequence output by the KATS module into the final category prediction. The classification module first integrates the features through a lightweight convolutional encoder, and then feeds the obtained feature vector into a fully connected layer with a Softmax activation function, thereby outputting the predicted probability of each motion image category and completing the end-to-end classification task.
[0016] Furthermore, the multi-scale temporal feature extraction unit includes three parallel two-dimensional convolutional branches. The three branches use temporal convolutional kernels of sizes (1, 15), (1, 31), and (1, 51) respectively to extract temporal features from different temporal resolutions, and the output feature maps are spliced in the channel dimension.
[0017] Furthermore, the hybrid attention fusion unit includes three parallel paths: a raw information path, a channel attention path, and a spatiotemporal attention path. The raw information path retains the output features of the multi-scale feature extraction unit. The channel attention path uses the ECA module to process features and capture cross-channel interactions. The spatiotemporal attention path uses the CBAM module to process features and enhance key information of the channel and spatial dimensions. The outputs of the three parallel paths respectively carry feature maps of the raw information, channel-dimensional enhanced information, and spatiotemporal enhanced information, and are stitched together in the channel dimension to achieve the aggregation of multi-view information.
[0018] Furthermore, the spatial feature integration and downsampling unit includes a spatial convolutional layer with a kernel size of (C,1), where C is the number of MI-EEG signal electrode channels, to integrate the spatial correlation between electrodes; a one-dimensional average pooling layer with kernel size P and stride S to downsample the features, reducing sequence length and computational complexity; and a dimension transpose operation to adjust the feature dimension to the KATS encoder adaptation format.
[0019] Furthermore, the global-local fusion attention mechanism includes parallel global attention streams and local attention streams. The global attention stream uses a standard self-attention mechanism to calculate the global context of the entire sequence. The local attention stream uses two parallel 1D convolutional branches to process the key matrix and capture local temporal patterns. The outputs of the two streams are added together to obtain the fusion attention features that are dependent on both long and short ranges.
[0020] Furthermore, the specialized KANS layer employs cubic B-spline basis functions, and its forward propagation expression is:
[0021]
[0022] It is a fixed basic activation function. These are the corresponding linear transformation weights. It is the i-th k-th B-spline basis function. is the corresponding learnable coefficient, G is the number of grid intervals, and the KANS layer is used to capture the local and non-smooth dynamic characteristics of ERD / ERS in MI-EEG signals.
[0023] A hierarchical transfer learning method for a specialized Transformer model used in motor imagery EEG decoding includes the following steps:
[0024] Step 1: Data preparation and augmentation; Divide the original MI-EEG trials into M equal-length non-overlapping segments in the time dimension, generate augmented samples according to the positional random replacement rule, and maintain the original augmented sample ratio of 1:1.
[0025] Step 2: Source domain pre-training. Train the complete Transformer model on a large-scale source domain MI-EEG signal dataset to learn general feature extraction capabilities.
[0026] Step 3: Layered progressive fine-tuning. The model is divided into 5 functional levels from bottom to top. Based on the calibration data of the target subjects, one of the 5 fine-tuning schemes is used to update the model parameters to achieve knowledge transfer.
[0027] Step 4: Decode the prediction. Apply the fine-tuned model to the MI-EEG signal of the target subject and output the prediction result of the motor imagery category.
[0028] Furthermore, the five functional layers in step 3, from bottom to top, are as follows: the bottom layer is a general temporal feature extraction layer, which consists of multi-scale convolutional modules; above it is a feature enhancement layer responsible for enhancement and integration, which includes a hybrid attention mechanism; next is a general sequence relation layer for modeling sequence dependencies, namely a global-local attention module; then above that is a specific nonlinear mapping layer for learning individualized patterns, implemented by the KANS module; and finally, the top layer is the classification decision layer, which is responsible for the final classification task.
[0029] Furthermore, the five fine-tuning schemes in step 3 are as follows:
[0030] Option 1: Freeze the parameters of levels 1 to 4, and only update the weights of level 5;
[0031] Option 2: Freeze levels 1 to 3, and update the weights of levels 4 and 5;
[0032] Option 3: Freeze levels 1 and 2, and update the weights of levels 3, 4, and 5;
[0033] Option 4: Freeze only level 1, and update the weights of levels 2 to 5;
[0034] Option 5: Update all parameters in the model.
[0035] (III) Beneficial Effects
[0036] Compared with existing technologies, this invention provides a specialized Transformer model and hierarchical transfer learning method for decoding motor imagery EEG, which has the following beneficial effects:
[0037] This invention features targeted designs at multiple levels, including feature enhancement, dependency modeling, and nonlinear representation. The model of this invention has a high average accuracy within subjects and its performance comprehensively surpasses that of many advanced baseline models. More importantly, the hierarchical fine-tuning strategy proposed in this invention can efficiently adapt the pre-trained model to new subjects. Only a small amount of calibration data is needed to improve cross-subject decoding performance to a high level. This invention provides an effective technical framework for developing high-performance and highly adaptable brain-computer interface systems. Attached Figure Description
[0038] Figure 1 This is a schematic diagram of the method steps of the present invention. Detailed Implementation
[0039] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0040] Example
[0041] like Figure 1 As shown in the figure, an embodiment of the present invention proposes a specialized Transformer model for decoding motor imagery EEG, comprising an MSCHA module, a KATS encoder, and a classification module connected in sequence. The MSCHA module, as the front end of the model, performs preliminary but comprehensive feature extraction and enhancement on the input raw MI-EEG signal, providing a high-quality feature sequence for the subsequent KATS module. The MSCHA module includes a multi-scale temporal feature extraction unit, a hybrid attention fusion unit, and a spatial feature integration and downsampling unit.
[0042] The multi-scale temporal feature extraction unit includes three parallel two-dimensional convolutional branches. The three branches use temporal convolutional kernels of sizes (1, 15), (1, 31), and (1, 51) to extract temporal features from different temporal resolutions. The output feature maps are concatenated in the channel dimension.
[0043] The hybrid attention fusion unit includes three parallel paths: the original information path, the channel attention path, and the spatiotemporal attention path. The original information path retains the output features of the multi-scale feature extraction unit. The channel attention path uses the ECA module to process features and capture cross-channel interactions. The spatiotemporal attention path uses the CBAM module to process features and enhance key information of the channel and spatial dimensions. The outputs of the three parallel paths respectively carry feature maps of the original information, channel-dimensional enhanced information, and spatiotemporal-dimensional enhanced information, and are stitched together in the channel dimension to achieve the aggregation of multi-view information.
[0044] The spatial feature integration and downsampling unit includes a spatial convolutional layer with a kernel size of (C,1), where C is the number of MI-EEG signal electrode channels, integrating the spatial correlation between electrodes; a one-dimensional average pooling layer with kernel size P and stride S, which downsamples the features to reduce sequence length and computational complexity; and a dimension transpose operation to adjust the feature dimension to the KATS encoder adaptation format.
[0045] The KATS encoder consists of four identical stacked KATS modules. The KATS module is the core encoding unit of the model, which performs deep temporal dependency modeling and nonlinear transformation on the feature sequence. Each KATS module adopts a global-local fusion attention mechanism to capture long-range and short-range dependencies at the same time, and uses a B-spline-based specialized KANS layer to accurately fit the key nonlinear dynamics in the EEG.
[0046] The global-local fusion attention mechanism includes parallel global attention streams and local attention streams. The global attention stream uses a standard self-attention mechanism to calculate the global context of the entire sequence. The local attention stream uses two parallel 1D convolutional branches to process the key matrix and capture local temporal patterns. The outputs of the two streams are added together to obtain the fused attention features that depend on both long and short ranges.
[0047] The classification module is responsible for converting the deep feature sequence output by the KATS module into the final category prediction. The classification module first integrates the features through a lightweight convolutional encoder, and then feeds the resulting feature vector into a fully connected layer with a Softmax activation function, thereby outputting the predicted probability of each motion image category and completing the end-to-end classification task.
[0048] The specialized KANS layer uses cubic B-spline basis functions, and the forward propagation expression is:
[0049]
[0050] It is a fixed basic activation function. These are the corresponding linear transformation weights. It is the i-th k-th B-spline basis function. is the corresponding learnable coefficient, G is the number of grid intervals, and the KANS layer is used to capture the local and non-smooth dynamic characteristics of ERD / ERS in MI-EEG signals.
[0051] A hierarchical transfer learning method for a specialized Transformer model used in motor imagery EEG decoding includes the following steps:
[0052] Step 1: Data preparation and augmentation; Divide the original MI-EEG trials into M equal-length non-overlapping segments in the time dimension, generate augmented samples according to the positional random replacement rule, and maintain the original augmented sample ratio of 1:1.
[0053] Step 2: Source domain pre-training. Train the complete Transformer model on a large-scale source domain MI-EEG signal dataset to learn general feature extraction capabilities.
[0054] Step 3: Layered progressive fine-tuning. The model is divided into 5 functional levels from bottom to top. Based on the calibration data of the target subjects, one of the 5 fine-tuning schemes is used to update the model parameters to achieve knowledge transfer.
[0055] The five functional layers, from bottom to top, are as follows: the bottom layer is a general temporal feature extraction layer, composed of multi-scale convolutional modules; above that is a feature enhancement layer responsible for enhancement and integration, which includes a hybrid attention mechanism; next is a general sequence relation layer for modeling sequence dependencies, namely a global-local attention module; then there is a specific nonlinear mapping layer for learning individualized patterns, implemented by the KANS module; and finally, the top layer is the classification decision layer, responsible for the final classification task.
[0056] The five fine-tuning schemes are as follows:
[0057] Option 1: Freeze the parameters of levels 1 to 4, and only update the weights of level 5;
[0058] Option 2: Freeze levels 1 to 3, and update the weights of levels 4 and 5;
[0059] Option 3: Freeze levels 1 and 2, and update the weights of levels 3, 4, and 5;
[0060] Option 4: Freeze only level 1, and update the weights of levels 2 to 5;
[0061] Option 5: Update all parameters in the model
[0062] Step 4: Decode the prediction. Apply the fine-tuned model to the MI-EEG signal of the target subject and output the prediction result of the motor imagery category.
[0063] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A specialized Transformer model for decoding motor imagery EEG, characterized in that: It includes the MSCHA module, KATS encoder, and classification module connected in sequence; The MSCHA module, as the front end of the model, performs preliminary but comprehensive feature extraction and enhancement on the input raw MI-EEG signal, providing high-quality feature sequences for the subsequent KATS module. The MSCHA module includes a multi-scale temporal feature extraction unit, a hybrid attention fusion unit, and a spatial feature integration and downsampling unit. The KATS encoder consists of four identical KATS modules stacked together. The KATS module is the core encoding unit of the model, which performs deep temporal dependency modeling and nonlinear transformation on the feature sequence. Each KATS module adopts a global-local fusion attention mechanism to capture long-range and short-range dependencies at the same time, and uses a B-spline-based specialized KANS layer to accurately fit the key nonlinear dynamics in the EEG. The classification module is responsible for converting the deep feature sequence output by the KATS module into the final category prediction. The classification module first integrates the features through a lightweight convolutional encoder, and then feeds the obtained feature vector into a fully connected layer with a Softmax activation function, thereby outputting the predicted probability of each motion image category and completing the end-to-end classification task.
2. A specialized Transformer model for decoding motor imagery EEG according to claim 1, characterized in that: The multi-scale temporal feature extraction unit includes three parallel two-dimensional convolutional branches. The three branches use temporal convolutional kernels of sizes (1, 15), (1, 31), and (1, 51) respectively to extract temporal features from different temporal resolutions, and the output feature maps are spliced in the channel dimension.
3. A specialized Transformer model for decoding motor imagery EEG according to claim 1, characterized in that: The hybrid attention fusion unit includes three parallel paths: a raw information path, a channel attention path, and a spatiotemporal attention path. The raw information path retains the output features of the multi-scale feature extraction unit. The channel attention path uses the ECA module to process features and capture cross-channel interactions. The spatiotemporal attention path uses the CBAM module to process features and enhance key information of the channel and spatial dimensions. The outputs of the three parallel paths respectively carry feature maps of the raw information, channel-dimensional enhanced information, and spatiotemporal enhanced information, and are stitched together in the channel dimension to achieve the aggregation of multi-view information.
4. A specialized Transformer model for decoding motor imagery EEG according to claim 1, characterized in that: The spatial feature integration and downsampling unit includes a spatial convolutional layer with a kernel size of (C,1), where C is the number of MI-EEG signal electrode channels, integrating the spatial correlation between electrodes; a one-dimensional average pooling layer with kernel size P and stride S, which downsamples the features to reduce sequence length and computational complexity; and a dimension transpose operation to adjust the feature dimension to the KATS encoder adaptation format.
5. A specialized Transformer model for decoding motor imagery EEG according to claim 1, characterized in that: The global-local fusion attention mechanism includes parallel global attention streams and local attention streams. The global attention stream uses a standard self-attention mechanism to calculate the global context of the entire sequence. The local attention stream uses two parallel 1D convolutional branches to process the key matrix and capture local temporal patterns. The outputs of the two streams are added together to obtain the fused attention features that depend on both long and short ranges.
6. A specialized Transformer model for decoding motor imagery EEG according to claim 1, characterized in that: The specialized KANS layer uses cubic B-spline basis functions, and the forward propagation expression is: It is a fixed basic activation function. It is its corresponding linear transformation weight. It is the i-th k-th B-spline basis function. is the corresponding learnable coefficient, G is the number of grid intervals, and the KANS layer is used to capture the local and non-smooth dynamic characteristics of ERD / ERS in MI-EEG signals.
7. A hierarchical transfer learning method based on a specialized Transformer model for motor imagery EEG decoding as described in any one of claims 1-6, characterized in that: Includes the following steps: Step 1: Data preparation and augmentation; Divide the original MI-EEG trials into M equal-length non-overlapping segments in the time dimension, generate augmented samples according to the positional random replacement rule, and maintain the original augmented sample ratio of 1:
1. Step 2: Source domain pre-training. Train the complete Transformer model on a large-scale source domain MI-EEG signal dataset to learn general feature extraction capabilities. Step 3: Layered progressive fine-tuning. The model is divided into 5 functional levels from bottom to top. Based on the calibration data of the target subjects, one of the 5 fine-tuning schemes is used to update the model parameters to achieve knowledge transfer. Step 4: Decode the prediction. Apply the fine-tuned model to the MI-EEG signal of the target subject and output the prediction result of the motor imagery category.
8. A hierarchical transfer learning method for a specialized Transformer model for motor imagery EEG decoding according to claim 7, characterized in that: In step 3, the five functional layers, from bottom to top, are as follows: the bottom layer is the general temporal feature extraction layer, which consists of multi-scale convolutional modules; above it is the feature enhancement layer responsible for enhancement and integration, which includes a hybrid attention mechanism; next is the general sequence relation layer for modeling sequence dependencies, i.e., the global-local attention module; then above that is the specific nonlinear mapping layer for learning individualized patterns, implemented by the KANS module; and finally, the top layer is the classification decision layer, which is responsible for the final classification task.
9. A hierarchical transfer learning method for a specialized Transformer model for motor imagery EEG decoding according to claim 7, characterized in that: The five fine-tuning schemes in step 3 are as follows: Option 1: Freeze the parameters of levels 1 to 4, and only update the weights of level 5; Option 2: Freeze levels 1 to 3, and update the weights of levels 4 and 5; Option 3: Freeze levels 1 and 2, and update the weights of levels 3, 4, and 5; Option 4: Freeze only level 1, and update the weights of levels 2 to 5; Option 5: Update all parameters in the model.