Electroencephalogram-myoelectricity collaborative limb movement intention decoding method and system based on symmetric cross-modal attention network

Abstract

The invention provides an electroencephalogram-myoelectricity collaborative limb movement intention decoding method and system based on a symmetric cross-modal attention network, and belongs to the technical field of neural rehabilitation engineering and movement intention decoding. Comprising the following steps: acquiring an electroencephalogram signal and an electromyographic signal to be decoded; inputting the electroencephalogram signals into an electroencephalogram channel specificity feature extraction network, and extracting multi-scale space-time oscillation features; the electromyographic signals are input into an electromyographic channel specificity feature extraction network, and dynamic time sequence mode features are extracted; inputting the electroencephalogram features and the myoelectricity features into a symmetric cross-modal attention module, carrying out bidirectional feature interaction and calibration, and generating fusion enhancement features; the symmetric cross-modal attention module realizes mutual enhancement and alignment between the electroencephalogram signals and the electromyographic signals by taking own features as queries and taking another modal feature as a key and a value through a cross attention mechanism; and inputting the fusion enhancement feature into a classifier, and decoding to obtain a corresponding motion intention category.

Description

Technical Field

[0001] This invention relates to the fields of neurorehabilitation engineering and motor intention decoding technology, and in particular to a method and system for decoding limb motor intention based on symmetrical cross-modal attention network-based EEG-EMG coordinated limb motor intention. Background Technology

[0002] Stroke-induced upper limb motor dysfunction severely restricts patients' independence in activities of daily living (ADL). Brain-computer interface (BCI) technology, by decoding central nervous system signals to drive external devices and induce neural plasticity, holds significant potential in the field of rehabilitation. Currently, the active rehabilitation BCI paradigm based on motor imagery (MI) (MI-BCI) is the mainstream, but its research mainly focuses on distinguishing bilateral limb movements (such as left and right hand), and the decoding paradigm is relatively simple. Clinical stroke patients are mostly unilateral hemiplegic, so decoding the intentions of multi-degree-of-freedom fine movements (such as single-handed grasping and extension) of the affected limb has more practical rehabilitation value. However, different fine movements of unilateral limbs mainly activate the ipsilateral sensorimotor cortex, and the resulting electroencephalogram (EEG) signals are highly overlapping in spatial distribution. In addition, the EEG signals themselves are non-stationary, have low spatial resolution, and are easily interfered with, making it difficult to achieve high-precision, multi-category fine movement decoding (such as fine movements with similar intentions, like "grasping a ball" and "grasping a cup") using only single-modality EEG.

[0003] To improve decoding performance, deep learning techniques (such as EEGNet[1], FBCNet[2], ATCNet[3], etc.) have been applied to EEG feature learning. Although a single EMG signal can reflect the muscle execution state, it lacks sufficient intention information before the initiation of movement or in the imagination task. In unilateral multi-task scenarios, the performance improvement of a single-modal network is limited by the low signal-to-noise ratio and spatial pattern overlap. Therefore, the introduction of peripheral electromyographic signals (EMG) that are complementary to EEG signals to form a multimodal hybrid BCI (Hybrid BCI) has become a research hotspot. EMG directly reflects muscle execution information and can enhance cortical activation and rehabilitation participation in motor imagination. However, existing EEG-EMG multimodal fusion methods have significant shortcomings: First, they are insufficient in handling the heterogeneity of the physical properties of the two signals (low-frequency oscillations and multi-scale characteristics of EEG vs. high-frequency random bursts and local dynamics of EMG), and often use general network structures, which leads to the loss or aliasing of key information; Second, they lack effective solutions to the problems of "spatiotemporal misalignment" (neural conduction delay from cortical intention generation to muscle execution) and "insufficient deep interaction" (most existing fusion strategies are unidirectional or static and lack bidirectional dynamic calibration mechanisms). Although some studies have attempted to introduce architectures such as Transformer for feature integration (such as NeuroFusion-Trans[4], AC-DSHResNet[5], E2FNet[6], etc.), they still fail to explicitly model the inherent fine-grained coupling relationship between "intention-execution", which limits the decoding robustness and accuracy in complex motion tasks. Summary of the Invention

[0004] In view of this, the present invention provides a method and system for decoding limb motor intentions based on a symmetrical cross-modal attention network (CCI) and electromyography (sEMG). This method is used for multimodal fusion of EEG and sEMG signals, and leverages their complementarity to overcome the accuracy bottleneck of single-modal methods. It overcomes the shortcomings of existing single-modal BCI and simple multimodal fusion methods in decoding fine motor intentions of unilateral upper limbs, such as low accuracy, poor robustness, and insufficient ability to handle heterogeneous signals and spatiotemporal misalignment. This method can be applied to neurorehabilitation, intelligent prosthetic control, and next-generation brain-computer interface systems. It can effectively fuse EEG and EMG signals to achieve high-precision and highly robust decoding of complex movements.

[0005] The technical solution adopted by the embodiments of the present invention to solve its technical problem is as follows:

[0006] The first aspect of this invention provides a method for decoding limb movement intentions based on a symmetric cross-modal attention network using EEG-EMG coordination, comprising:

[0007] Acquire the EEG and EMG signals to be decoded;

[0008] The EEG signal is input into the EEG channel-specific feature extraction network to extract multi-scale spatiotemporal oscillation features as EEG features; the EEG channel-specific feature extraction network includes a depthwise separable convolution module and a holed spatial pyramid pooling module.

[0009] The electromyographic signal is input into the electromyographic channel-specific feature extraction network to extract dynamic temporal pattern features as electromyographic features; the electromyographic channel-specific feature extraction network includes a dilated residual temporal convolutional network.

[0010] EEG and EMG features are input into a symmetrical cross-modal attention module for bidirectional feature interaction and calibration to generate fused and enhanced features. The symmetrical cross-modal attention module uses its own features as queries and the other modality's features as keys and values ​​through a cross-attention mechanism to achieve mutual enhancement and alignment between EEG and EMG signals.

[0011] The fused and enhanced features are input into the classifier, and the corresponding motion intent category is obtained by decoding.

[0012] Preferably, the symmetric cross-modal attention module performs bidirectional feature interaction and calibration, specifically including:

[0013] First attention path: Using EEG features as queries and EMG features as keys and values, calculate attention weights and output EEG enhancement features calibrated by EMG features;

[0014] The second attention path: using electromyography (EMG) features as queries and electroencephalography (EEG) features as keys and values, calculates attention weights and outputs EMG enhancement features calibrated by EEG features; the EEG enhancement features and EMG enhancement features are fused to obtain the fused enhancement features.

[0015] Preferably, the brainwave channel-specific feature extraction network utilizes the cavity spatial pyramid pooling module to capture rhythmic oscillation information at different time scales in the brainwave signal by setting convolutional layers with different cavitation rates in parallel.

[0016] Preferably, the dilated residual temporal convolutional network in the electromyographic channel-specific feature extraction network increases the receptive field and captures long-range temporal dependencies in the electromyographic signal by stacking causal convolutional layers with dilation factors.

[0017] Preferably, the movement intention category includes at least one of the following: arm extension in different spatial directions, hand gripping of different objects, and wrist pronation or supination.

[0018] A second aspect of the present invention provides an EEG-EMG coordinated limb movement intention decoding system based on a symmetrical cross-modal attention network, comprising:

[0019] The signal acquisition module is used to simultaneously acquire the user's electroencephalogram (EEG) and electromyogram (EMG) signals;

[0020] The EEG feature extraction module, consisting of a depthwise separable convolution module and a hollow spatial pyramid pooling module, is used to receive the EEG signals and extract multi-scale spatiotemporal oscillation features.

[0021] The electromyography feature extraction module, composed of a dilated residual temporal convolutional network, is used to receive the electromyography signal and extract dynamic temporal pattern features.

[0022] The symmetrical cross-modal attention fusion module is used to receive the EEG features and EMG features, perform feature interaction and calibration through a bidirectional cross-attention mechanism, and output fused and enhanced features.

[0023] The intent decoding module is used to receive the fused enhanced features and decode and output the corresponding motion intent category.

[0024] Preferably, the symmetric cross-modal attention fusion module includes:

[0025] The first cross-attention submodule is configured to use EEG features as queries and EMG features as keys and values, and output EEG enhancement features calibrated by EMG.

[0026] The second cross-attention submodule is configured to use electromyography (EMG) features as queries and electroencephalography (EEG) features as keys and values, and output EMG-calibrated EMG enhancement features; the feature fusion submodule is used to fuse the EMG enhancement features with the EMG enhancement features.

[0027] Preferably, the movement intention category includes fine motor actions corresponding to one upper limb, including at least one of arm extension direction, hand grasp type, and wrist rotation direction.

[0028] Preferably, the system is part of a brain-computer interface rehabilitation training system, and the output of the intent decoding module is used to drive external rehabilitation equipment or provide neurofeedback.

[0029] As can be seen from the above technical solution, the EEG-EMG co-modal limb movement intention decoding method and system based on a symmetric cross-modal attention network provided in this invention first acquires the EEG and EMG signals to be decoded; the EEG signals are input into an EEG channel-specific feature extraction network to extract multi-scale spatiotemporal oscillation features as EEG features; the EEG channel-specific feature extraction network includes a depthwise separable convolutional module and a dilated spatial pyramid pooling module; the EMG signals are input into an EMG channel-specific feature extraction network to extract dynamic temporal pattern features as EMG features; the EMG channel-specific feature extraction network includes a dilated residual temporal convolutional network; the EEG and EMG features are input into a symmetric cross-modal attention module for bidirectional feature interaction and calibration to generate fused enhanced features; the symmetric cross-modal attention module uses a cross-attention mechanism, with its own features as queries and the other modality's features as keys and values, to achieve mutual enhancement and alignment between the EEG and EMG signals; the fused enhanced features are input into a classifier to decode the corresponding movement intention category. Cross-modal fusion of EMG and EEG signals is achieved by constructing a bidirectional query path based on a multi-head attention mechanism, which includes two independent and parallel cross-attention computational flows: EEG→EMG and EMG→EEG. This enables deep heterogeneous feature interaction and solves the spatiotemporal mismatch problem in heterogeneous signal fusion. Attached Figure Description

[0030] Figure 1 This is a diagram of the CATA-Net model architecture of the present invention.

[0031] Figure 2 The proposed CATA-Net model is compared with existing state-of-the-art methods under different experimental paradigms.

[0032] Figure 3 A schematic diagram of the subject-specific accuracy distribution for the motor imagery grasping task (MI-Grasping).

[0033] Figure 4 This diagram illustrates the subject-specific performance and inter-session fluctuations of CATA-Net in the motor imagery grasping (MI-Grasping) task.

[0034] Figure 5 Evaluation of CATA-Net's cross-session generalization performance and time stability in grasping tasks.

[0035] Figure 6 This is the classification confusion matrix of the CATA-Net model in complex and fine action recognition.

[0036] Figure 7 Visualization of the CATA-Net feature manifold based on t-SNE (using subject 8 as an example).

[0037] Figure 8 This diagram illustrates a comparison of the task-specific performance of CATA-Net and benchmark models under real motion and motion imagination paradigms. Detailed Implementation

[0038] The technical solution and effects of the present invention will be further described in detail below with reference to the accompanying drawings.

[0039] This invention aims to overcome the shortcomings of existing single-modal BCI and simple multimodal fusion methods in decoding fine motor intentions of the unilateral upper limb, such as low accuracy, poor robustness, and insufficient ability to handle heterogeneous signals and spatiotemporal misalignment. It proposes a method and system that can effectively fuse EEG and EMG signals to achieve high-precision and highly robust decoding of complex movements. Accordingly, this invention provides an EEG-EMG multimodal fusion decoding architecture based on an intention-execution coordination mechanism—CATA-Net (Cross-modal Asymmetric Transformer Attention Network). The core of this architecture lies in: specific processing of heterogeneous signals: different feature extraction modules are designed to process non-stationary EEG spatiotemporal signals and dynamic EMG temporal signals respectively; establishing a bidirectional mapping channel: a symmetric cross-modal attention module (SCMA) is introduced to simulate the physiological coordination mechanism between cortical motor intention and muscle execution activity, achieving mutual calibration and enhancement of the two features in deep semantics; and providing a specific technical implementation path from data to decoding results.

[0040] refer to Figure 1 The model shown employs a two-stream architecture, consisting of a modality-specific feature encoder, an SCMA module, and an attention decision module. In the feature extraction stage performed by the modality-specific feature encoder, the EEG branch integrates deep temporal convolution and ASPP to effectively capture multi-scale spatiotemporal dependencies in non-stationary neural signals; the EMG branch uses Dilated Res-TCN to extract robust local dynamic temporal patterns. The core SCMA module achieves mutual calibration and deep fusion of heterogeneous information after unifying feature dimensions through a bidirectional scaled dot product attention mechanism. Finally, the fused representation input decision module completes high-precision classification of various fine motor actions such as arm extension, grasping, and twisting.

[0041] EEG feature extraction branch:

[0042] EEG exhibits characteristics of non-stationarity, low signal-to-noise ratio, and high-dimensional spatiotemporal coupling. The EEG feature extraction branch of CATA-Net is used for EEG feature extraction, simultaneously capturing short-term high-frequency oscillations and long-term low-frequency dependencies. The EEG branch employs a hierarchical architecture based on depthwise separable convolutions and integrates the ASPP module to achieve multi-scale feature aggregation. Input data is represented as tensors. ,in Represents the number of electrode channels. Let be the number of sampling points. First, in order to extract frequency domain features from the original signal, the model uses a multi-kernel time-domain convolutional layer to filter each channel independently. Let be... For the first Each channel at time The input signal, the first The output of each time-domain filter Discrete convolution can be defined as: (1)

[0043] in, Indicates the first The weight vector of each temporal convolution kernel. The value is set to 64 to cover a complete oscillation period of approximately 250 ms within the 4-40 Hz frequency band. This is a bias term. This layer is configured with... Each convolutional kernel is used, and L2 regularization and MaxNorm constraints (maximum norm of 0.6) are applied to prevent overfitting. Subsequently, to capture the spatial correlations between different brain regions and reduce feature dimensionality, a deep convolutional layer is introduced. This layer does not perform inter-channel summation but instead applies a spatial filter to each temporal feature map. Expand each time channel to A spatial feature map. The spatial aggregation process is described as follows: (2)

[0044] The depth multiplier here is set to This expands the number of output feature maps to After batch normalization, the exponential linear unit (ELU) is used as the activation function to address neuron death and accelerate convergence. The ELU function is defined as follows: (3)

[0045] in This is a hyperparameter, typically set to 1.0. Then... The average pooling layer reduces the temporal resolution, and Dropout (with a ratio of 0.5) is introduced at the end of the feature map to enhance the model's generalization ability.

[0046] To further capture multi-scale contextual information in non-stationary EEG signals, this branch introduces an improved ASPP module at the end of the encoder. Traditional convolutional operations are limited by a fixed receptive field, making it difficult to simultaneously adapt to feature extraction of different frequency components. ASPP, by introducing dilated convolution, exponentially expands the receptive field without increasing the number of parameters. For the input feature sequence... and convolution kernel expansion rate The dilated convolution is defined as: (4)

[0047] when At this point, the operation degenerates into a standard convolution. In this architecture, the ASPP module contains three parallel depthwise separable convolutional branches, with dilation rates set to [values ​​to be filled in]. The kernel size is uniformly set to This multi-branch design allows the model to be designed separately using... Capture local detail features, and with and Capture long-range temporal dependency features. The outputs of each branch are fused through a concatenation operation and then processed through a... The point convolutional layers are used to integrate channels, ultimately generating feature vectors containing rich multi-scale spatiotemporal information. This is used by subsequent cross-modal attention modules. All convolutional layers are initialized uniformly using He, ensuring effective gradient propagation during the early stages of deep network training.

[0048] Electromyography feature extraction branch:

[0049] EMG directly reflects the superposition and evolution of motor unit action potentials (MUAPs) over time, and its characteristics essentially include the intensity of muscle activation, coordination patterns, and dynamic temporal dependencies. To robustly decode motor execution commands from high-frequency, random, and transient EMG signals, this invention provides a dedicated feature extraction branch based on a Dilated Res-TCN to robustly extract the dynamic temporal patterns inherent in muscle activation. This branch, by introducing depthwise separable convolutions and an exponentially increasing dilation rate, aims to construct a long-term receptive field that covers the entire movement cycle at low computational cost, while preserving local temporal details.

[0050] Let the preprocessed electromyography input sequence be... ,in The number of time sampling points, This represents the number of channels for the electromyography (EMG) sensor. First, the input is projected onto a high-dimensional feature space through an initial convolutional layer with a kernel size of 1, generating a feature tensor. ,in The number of filters represents the number of channels. This operation enhances information exchange between channels while keeping the timing length constant.

[0051] The core feature extraction process is as follows: This model consists of cascaded residual dilated convolutional blocks, with a set depth. To effectively capture long-range temporal dependencies without causing resolution degradation or parameter explosion, dilated convolution operations are used within each block. For the first... Layer input sequence The dilated convolution operation on the one-dimensional time axis is defined as follows: (5)

[0052] in, Represents the convolution kernel. The kernel size is the convolution kernel size. For the first The void ratio of each residual block. To construct a hierarchical temporal context awareness, the model employs an exponential expansion strategy, namely... ( This allows the effective receptive field of the network to grow exponentially with the number of layers, thereby ensuring that the top-level features can cover a sufficiently long historical period of information to determine the current motion state.

[0053] Furthermore, to optimize parameter efficiency and mitigate overfitting, the dilated convolution operation described above is implemented using depthwise separable convolution. This mechanism decomposes standard convolution into two steps: depthwise convolution and pointwise convolution. Assume the input feature map has [number of channels]. Depthwise convolution first applies temporal filtering independently to each channel: (6)

[0054] Subsequently, pointwise convolution is performed... convolution kernel Linearly combine the features of each channel: (7)

[0055] in and These represent the weights of the depthwise convolutional kernel and the pointwise convolutional kernel, respectively. In the code implementation, each residual block contains two consecutive depthwise separable convolutional layers, followed by batch normalization, a linear activation function, and dropout. Regularization layers are added to enhance the model's generalization ability and training stability.

[0056] To address the vanishing gradient problem in deep networks and facilitate efficient backpropagation of gradients, residual connections are introduced into each convolutional block. Let... Let represent a two-layer convolution transformation within the block, then the ... Output of each block It can be represented as: (8)

[0057] This structure allows raw information to flow directly to subsequent layers, forcing the network to focus on learning the residual mapping of the input. Ultimately, after... Layer stacking processing results in EMG branch outputs containing high-level feature sequences rich in temporal dynamics and muscle synergy patterns. The feature sequence will then be downsampled in the time dimension using global average pooling to match the feature resolution of the EEG branch, providing an aligned execution layer representation for subsequent cross-modal attention fusion.

[0058] Symmetric Cross-Modal Attention Module (SCMA):

[0059] Although single-modal feature extraction branches can capture the neural intention patterns of cortical EEG and the muscle execution details of EMG respectively, direct splicing of the two features often fails to establish accurate semantic alignment due to neural conduction delays and signal heterogeneity. To deeply explore the intrinsic coupling relationship of the "intention-execution" mechanism within the feature space, this study constructs an SCMA module. This module aims to address the spatiotemporal asynchrony problem of multimodal signals by using a bidirectional query mechanism to recalibrate and focus the feature expression of one modality using the feature distribution of one modality. The symmetric cross-modal attention module establishes a bidirectional, dynamic interactive channel between the EEG and EMG streams through a cross-attention mechanism. It achieves mutual mapping, calibration, and enhancement between cortical intention (EEG features) and muscle execution (EMG features) signals through the attention mechanism. Each modality uses its own features as queries and the other's features as keys and values ​​to achieve mutual enhancement and alignment between EEG and EMG signals. The fused enhanced features are then input into a classifier to decode and obtain the corresponding motor intention category.

[0060] set up They are respectively after feature extraction branches and The feature sequence after average pooling downsampling. To map heterogeneous features to a unified semantic space, a fully connected layer is first used to project both features to the same hidden layer dimension. And introduce the nonlinear activation function ReLU to enhance the expressive power of features. The core of SCMA lies in two parallel multi-head attention mechanisms

[27] (MHA): one uses EEG as a query to retrieve complementary information in EMG, and the other uses EMG as a query to trace back the corresponding intent in EEG. For any two modal inputs and (here) The computation process based on scaled dot product attention can be defined as follows: (9)

[0061] in, , , These represent the query matrix, key matrix, and value matrix, respectively. It is a learnable linear projection weight matrix. The dimension of the key vector, used here as a scaling factor. This is used to prevent the gradient vanishing problem caused by excessively large dot product results.

[0062] To capture rich cross-modal interaction information from different representation subspaces, the model employs a multi-head mechanism. Specifically, for Multiple parallel attention heads (as set in this study) ), No. The output for each head is calculated as follows: (10)

[0063] The outputs of all heads are concatenated and linearly transformed to generate the final cross-modal features. In the SCMA module, this computation process is applied symmetrically in both directions:

[0064] 1) The first attention path is an EEG-guided EMG enhancement path: using EEG features as the query and EMG features as the key and value, the attention weight is calculated and the EEG enhancement features calibrated by EMG features are output. As a query As keys and values. The output of this path. By using the motion intention distribution of EEG as an index, the muscle activation patterns most relevant to the intention in the EMG signal are weighted and aggregated, thereby suppressing background electromyographic noise that is irrelevant to the specific action.

[0065] 2) The second attention path is an EMG-guided EEG focusing path: using EMG features as the query and EEG features as the key and value, it calculates the attention weight and outputs the EMG enhancement features calibrated by the EEG features, letting... As a query As keys and values. The output of this path. By using actual detected muscle activity as feedback, the specific cortical oscillation region that triggers the action in the EEG signal is focused in reverse, effectively alleviating the spatial ambiguity of the EEG signal.

[0066] Ultimately, attention output in both directions (EEG enhancement characteristics) and (EMG enhancement features) are compressed into fixed-length vectors through a global average pooling layer, and then concatenated along the channel dimension to form a fused feature vector containing both intent and execution verification information. (i.e., fusion enhancement features). This fused vector is then fed into a classifier containing Dropout (ratio 0.5) and fully connected layers for the final motion class probability prediction. This symmetrical interactive design not only ensures the integrity of information flow but also significantly improves the model's robustness in complex unilateral motion decoding through explicit bidirectional calibration.

[0067] Attention-based decision-making module:

[0068] fused feature vector output by the symmetric cross-modal attention module Although it integrates the intent representation of EEG and the execution details of EMG, it still resides in a high-dimensional feature manifold space, making it difficult to directly apply to discrete motion category determination. To map this multimodal hybrid representation to a specific action label space, a decision module based on nonlinear transformation and regularization strategies was constructed. First, to mitigate the overfitting phenomenon that easily occurs when training deep neural networks on high-dimensional physiological signal data (especially EEG / EMG datasets with limited sample size) and to introduce sparsity into the feature layers, the model implements a Dropout regularization strategy before and after the fully connected layers. This process can be mathematically formalized as applying a Bernoulli-distributed mask transformation to the input feature vector. Assume the input to the decision module is... The features after Dropout processing It can be represented as: (11)

[0069] in, This represents the Hadamard product (i.e., element-wise multiplication). Let be a mask vector, whose elements follow the parameter . Bernoulli distribution, For the discard rate. In the configuration of this study, Setting it to 0.5 means that about half of the neuron connections are randomly blocked during training, thus forcing the network to learn more robust distributed feature representations rather than relying on specific local co-adaptation patterns.

[0070] The regularized feature vectors are then fed into a hidden layer containing 128 neurons, where affine transformations and nonlinear activation functions map the features to a more discriminative decision space. The output of this layer... The calculation is as follows:

[0071] (12)

[0072] In this formula, This is the weight matrix. For bias vectors, This corresponds to the ReLU activation function, which aims to solve the vanishing gradient problem and accelerate model convergence. After another Dropout layer with a ratio of 0.5, the features are finally input to the output layer.

[0073] The output layer consists of a fully connected layer and a Softmax normalization function, used to generate the posterior probability distributions belonging to each motion category. Let... The number of target categories for the task (in this experimental setup, for tasks involving arm extension, grasping, and wrist twisting), (The values ​​are 6, 3, or 2 respectively), for the first... Predicted probabilities for each category Defined as: (13)

[0074] in, It is an unnormalized logits vector. and These represent the weights and biases of the output layer, respectively. The Softmax function ensures that the output vector satisfies... and The probability axioms constrain this. Model parameter optimization is based on minimizing the true label distribution. With the predicted probability distribution The classification cross-entropy loss function between them: (14)

[0075] In the formula This indicates the batch size (set to 8). For the first Each sample belongs to category The true labels (one-hot encoded) are obtained. The loss function is updated through backpropagation using the Adam optimizer, and the decision module can adaptively adjust the decision boundary, thereby achieving high-precision parsing of complex unilateral upper limb movement intentions in the multimodal feature space.

[0076] Detailed implementation methods and calculation process examples:

[0077] The following describes a specific embodiment of the EEG-EMG co-operational limb movement intention decoding method based on a symmetric cross-modal attention network using CATA-Net. This embodiment uses the multimodal upper limb movement dataset published by Jeong et al. The implementation steps include:

[0078] Step S1: Acquire the EEG and EMG signals to be decoded; multimodal physiological data acquisition and preprocessing includes:

[0079] Data source: 25 healthy right-handed subjects were selected from the dataset, and their experimental task data (arm extension, grasping a ball, wrist pronation) and two motor imagery (imagining grasping a ball, imagining wrist pronation) were selected as the decoding target of this embodiment.

[0080] Signal extraction and downsampling: 60 channels of EEG and 7 channels of forearm sEMG signals were extracted from the original 2500 Hz sampled data. To balance computational efficiency and information preservation, the EEG and EMG signals were downsampled to 500 Hz.

[0081] Data segmentation: Starting from the task prompt start time, a data segment of length 3 seconds (1500 sampling points) is extracted, with each trial as a sample. Each sample is then standardized (z-score).

[0082] Dataset splitting: Data from 20 subjects (approximately 8,000 samples) was used as the training set, and data from 5 subjects (approximately 2,000 samples) was used as an independent test set to verify the model's cross-subject generalization ability.

[0083] Step S2, Modality-Specific Feature Extraction (CATA-Net Forward Operation): The EEG signal is input into the EEG channel-specific feature extraction network to extract multi-scale spatiotemporal oscillation features as EEG features.

[0084] Input: A sample containing an EEG data matrix of dimensions [60, 1500] and an EMG data matrix of dimensions [7, 1500].

[0085] EEG feature extraction branch:

[0086] Operation: First, a depthwise separable convolutional layer is used to perform lightweight initial feature extraction from both the spatial (60 channels) and temporal dimensions. Then, the features are fed into the Spatial Pyramid Pooling (ASPP) module.

[0087] The ASPP operation process involves using multiple dilated convolutional kernels with different dilation rates (e.g., dilation rates of 1, 6, and 12) in parallel to convolve the input features, capturing spatial contextual information of cortical activity at multiple scales. Finally, the output features from each parallel branch are concatenated along the channel dimension and fused using a 1x1 convolution to output a high-dimensional EEG feature tensor F_eeg. This step resolves the spatial ambiguity problem in EEG.

[0088] EMG feature extraction branch:

[0089] Operation: Input EMG data into a dilated residual temporal convolutional network (Dilated Res-TCN).

[0090] The Dilated Res-TCN operation process: This network consists of multiple stacked residual blocks, each containing dilated convolutional layers with an exponentially increasing dilation rate (e.g., 1, 2, 4, 8…). Through this structure, the network can capture long-range, dynamic temporal dependencies in muscle activation patterns, such as the onset, duration, and extinction patterns of muscle contraction, with an exponentially expanded receptive field. The output is a high-dimensional EMG feature tensor. .

[0091] Step S3: Symmetrical cross-modal attention feature fusion (core innovation point operation), inputting electromyographic signals into the electromyographic channel-specific feature extraction network to extract dynamic temporal pattern features as electromyographic features:

[0092] Operation: and Feed it into the Symmetric Cross-Modal Attention Module (SCMA).

[0093] SCMA calculation process:

[0094] Cross-attention computation inputs EEG and EMG features into a symmetrical cross-modal attention module for bidirectional feature interaction and calibration, generating fused enhanced features:

[0095] First attention path: EEG as query, EMG as key and value: Calculate the attention weights of EEG features on EMG features. This process simulates "how cortical intent focuses on and modulates specific muscle execution patterns." Formulated as: = Softmax(( * ) * ( * )^T / sqrt(d))* ( * ),in , , For learnable projection matrices, For feature dimensions.

[0096] Second attention path: EMG as query, EEG as key and value: Calculate the attention weights of EMG features on EEG features. This process simulates "how peripheral executive feedback influences and corrects central motor commands". Represented as: Attn_m2e = Softmax(( * ) * ( * ')^T / sqrt( )) *( * ').

[0097] The obtained attention features and the original features are then subjected to residual connections and layer normalization to obtain the enhanced EEG features. 'and EMG features '.

[0098] Finally, ' and The features are concatenated along the channel dimension and fused through a fully connected layer to output the final joint feature representation. .

[0099] Step S4: Classification and Intent Decoding

[0100] Operation: Combine features Input a classifier consisting of a fully connected layer and a Softmax function.

[0101] Operation: The classifier outputs a probability distribution vector. These correspond to five task categories (three real movements + two imagined movements). The category with the highest probability is taken as the movement intention ultimately decoded by the system.

[0102] Step S5: Model Training and Optimization

[0103] Loss function: Use the standard cross-entropy loss function.

[0104] Optimization process: The Adam optimizer is used for iterative training on the training set with a batch size of 32. The accuracy on the test set is monitored during training, and an early stopping strategy is employed to prevent overfitting.

[0105] Accordingly, this invention provides an EEG-EMG co-modal limb movement intention decoding system based on a symmetrical cross-modal attention network, used to implement the aforementioned EEG-EMG co-modal limb movement intention decoding method based on a symmetrical cross-modal attention network. The system includes:

[0106] The signal acquisition module is used to simultaneously acquire the user's electroencephalogram (EEG) and electromyogram (EMG) signals;

[0107] Signal preprocessing module: performs downsampling, filtering, segmentation, and normalization operations;

[0108] CATA-Net decoding module: It has built-in EEG feature extraction sub-network, EMG feature extraction sub-network, symmetric cross-modal attention module and classifier as described above, and is used to perform S2 to S4 operations;

[0109] Intent output module: Converts the decoded intent category into control commands and outputs them to external devices (such as prostheses, robots, or computers).

[0110] The CATA-Net decoding module includes:

[0111] The EEG feature extraction module consists of a depthwise separable convolution module and a hollow spatial pyramid pooling module, which is used to receive EEG signals and extract multi-scale spatiotemporal oscillation features.

[0112] The electromyography feature extraction module, composed of a dilated residual temporal convolutional network, is used to receive electromyography signals and extract dynamic temporal pattern features.

[0113] The symmetrical cross-modal attention fusion module receives EEG and EMG features, performs feature interaction and calibration through a bidirectional cross-attention mechanism, and outputs fused and enhanced features. The symmetrical cross-modal attention fusion module includes:

[0114] The first cross-attention submodule is configured to use EEG features as queries and EMG features as keys and values, and output EEG enhancement features calibrated by EMG.

[0115] The second cross-attention submodule is configured to use EMG features as queries and EEG features as keys and values, and output EMG enhancement features calibrated by EEG. The feature fusion submodule is used to fuse EEG enhancement features with EMG enhancement features.

[0116] This system is part of a brain-computer interface rehabilitation training system. The output of the intent decoding module is used to drive external rehabilitation equipment or provide neurofeedback.

[0117] Technical effects and experimental results:

[0118] Under the data and experimental settings of the embodiment, the system achieved the following technical effects:

[0119] High decoding accuracy: As shown in Table 1, CATA-Net achieves an average decoding accuracy of 98.3% on three real motion tasks, which is significantly better than the single-modal baseline model.

[0120] Table 1. Comparison of classification performance (accuracy %) for three motion tasks under real motion and motion imagination conditions.

[0121] Figure 2 The paper compares the Kappa coefficients of the CATA-Net model with existing methods under different experimental paradigms. (a) represents the Kappa values ​​for the stretching, grasping, and twisting tasks in the Real Motion (RM) mode, and (b) represents the Kappa values ​​for the grasping and twisting tasks in the Motion Imagining (MI) mode. In the RM mode, CATA-Net achieves Kappa values ​​exceeding 0.88 in the stretching, grasping, and twisting tasks, indicating high consistency and reliability. Particularly in the twisting task under the Motion Imagining mode, CATA-Net achieves a Kappa value of 0.50, significantly outperforming most unimodal models (typically between 0.20 and 0.30). This verifies the effectiveness of the proposed method in processing weak motion imagining signals and effectively addresses the problem of low consistency in motion imagining tasks using traditional methods.

[0122] Breaking through the bottleneck of imagination tasks: As shown in Table 2, in the highly challenging motion imagination task (grasping and twisting), the method of this invention improves the accuracy by 7.7% and 14.7% respectively compared to the current state-of-the-art single-modal EEG decoding model ATCNet. This directly proves that the bidirectional mapping established by SCMA effectively compensates for the lack or weakness of EMG signals in the imagination task, and enhances the decoding capability by utilizing the intent information of EEG.

[0123] Figure 3 and Figure 4 The figures show the subject-specific accuracy distribution and inter-session variability of the motor imagery grasping task. CATA-Net exhibits high median and upper quartile values ​​across different subjects. Although the standard deviation is slightly high, this reflects that the model can exceed its performance limits rather than being unstable. The figures show that some subjects (such as S8 and S12) achieved near 100% accuracy, indicating that the method of this invention can fully utilize high-quality neuromuscular signals and provide better decoding performance for subjects with good BCI control potential.

[0124] Figure 5 The cross-session generalization performance and temporal stability of CATA-Net in the grasping task were evaluated. (a) shows the accuracy change of the model from early to subsequent sessions in Real Motion (RM) mode; (b) shows the performance retention relative to the same-session benchmark in Motion Imagining (MI) mode. In RM mode, even with a one-week interval, the model accuracy decreased by only 2.8%-3.5%, maintaining an average accuracy of 96.2%, indicating that the proposed method can learn intrinsic neuromuscular control patterns that do not drift over time. In MI mode, although the signal is more unstable, the performance degradation was still controlled within 5.8%, validating the potential of the proposed method to reduce the need for frequent recalibration in practical applications.

[0125] Table 2. Performance statistics and significance tests of different models in the motor imagery grasping task (MI-Grasping) (N=75)

[0126] (Note: The superscript * indicates that the upper right corner is a symbol) The upper right corner mark ** indicates (Statistical tests are based on pairwise comparisons between CATA-Net and other models.)

[0127] Strong generalization ability: It performs stably on test sets across subjects, indicating that the model has learned universal neuromuscular co-representations rather than individual-specific noise.

[0128] refer to Figure 6 The CATA-Net model is shown in the classification confusion matrix for complex fine motion recognition, where (a) represents the real motion grasping task, (b) represents the real motion wrist twisting task, (c) represents the motion imagination grasping task, and (d) represents the motion imagination wrist twisting task. Figure 6 The classification confusion matrix shows that CATA-Net achieved recognition rates of over 96% and 98% in real-motor grasping and wrist twisting tasks, respectively, indicating that high-precision decoding can be achieved directly using electromyographic signals. In the motor imagery task, although some confusion exists, CATA-Net still maintains a diagonal advantage of over 62%, far exceeding the random level, proving that the method of this invention can effectively utilize EEG intention information to compensate for missing muscle information.

[0129] refer to Figure 7The visualization of the CATA-Net feature manifold based on t-SNE (taking subject 8 as an example) shows the feature distribution of arm extension, grasping and wrist twisting tasks in the real motion (RM) mode, respectively, and (d) and (e) show the feature distribution of grasping and wrist twisting tasks in the motor imagery (MI) mode, respectively. Figure 7 This demonstrates that CATA-Net can map highly overlapping neuromuscular signals to well-defined clusters in a high-dimensional feature space. In real-world motion modes, the arm extension task exhibits six distinctly separate, compact clusters, while the wrist twisting task displays a characteristic "V-shaped" topology, visually demonstrating the model's successful separation of neural representations corresponding to different motion intentions. Even in motion imagination mode, the model maintains feature discriminability, validating the feature decoupling capability of the proposed method.

[0130] Figure 8 The performance comparison between CATA-Net and the baseline model is presented in different tasks. In the real-motor mode, as the task transitions from spatially guided extension to force-controlled grasping and twisting, the accuracy of CATA-Net gradually improves, indicating that the method of this invention can effectively integrate the spatial coding of EEG and the fine force control mechanism of EMG. In the motor imagery mode, CATA-Net achieves an accuracy of 75.0% in the wrist twisting task, which is 5.9% and 9.0% higher than the best multimodal and unimodal baselines, respectively, confirming the significant advantage of the method of this invention in processing weak motor intention signals.

[0131] This invention provides a complete technical solution encompassing heterogeneous data acquisition, specific processing, bidirectional attention fusion, and final classification decision-making. It successfully transforms the physiological "intent-execution coordination" mechanism into a computable and optimizable deep learning model. This solution specifically addresses the technical problem of insufficient accuracy in fine motor intent decoding in existing technologies. Verification using publicly available datasets demonstrates a clear and significantly superior technical effect compared to existing methods, providing a practical technical path for building high-performance, naturally interactive BCI systems.

[0132] The EEG-EMG collaborative decoding method and system (CATA-Net) based on a symmetric cross-modal attention network proposed in this invention has achieved the following outstanding technical effects in practical implementation:

[0133] Significant improvement in decoding accuracy: In the most challenging unilateral upper limb motor imagery (MI) decoding task, CATA-Net achieves state-of-the-art (SOTA) classification accuracy. Particularly in decoding fine motor skills (such as wrist twisting and grasping different objects), its performance shows a significant and statistically significant (p<0.01) improvement compared to existing advanced monomodal EEG decoding models (such as ATCNet) and multimodal fusion models (such as NeuroFusion-Trans and FAConformer). For example, in the MI wrist twisting task, the accuracy is improved by 5.9 percentage points compared to the second-best model; in the MI grasping task, the average accuracy is improved by 7.7% compared to the pure EEG model.

[0134] Excellent cross-session robustness and generalization ability: CATA-Net demonstrates outstanding stability in addressing the non-stationarity of EEG / EMG signals over time (across sessions). In rigorous cross-session evaluations (trained with early session data and tested with subsequent session data), the model's performance degradation is minimal. For example, in real-world motor tasks, the accuracy drop-off across a week of sessions is less than 3.5%; even in the more unstable motor imagery task, the performance degradation is kept below 5.8%. This demonstrates that the system can learn intrinsic neuromuscular control patterns that do not drift over time, significantly reducing the need for frequent recalibration in practical applications.

[0135] Effective Adaptation to Individual Differences and Unleashing Performance Potential: The system demonstrates strong adaptability when facing different subjects. For subjects with good BCI control potential, CATA-Net, through its deep interaction mechanism, can fully explore and integrate effective neuromuscular features, achieving near 100% decoding accuracy in some sessions, breaking through the performance "ceiling" of traditional methods. This indicates that the system has the ability to help users maximize the potential of their physiological signals, providing possibilities for personalized rehabilitation training.

[0136] Enhanced resolution of weak intention signals: In challenging scenarios involving only motor imagery without actual muscle contraction, traditional multimodal methods often fail due to the excessively weak electromyographic (EMG) signals. CATA-Net, through its Symmetric Cross-Modal Attention (SCMA) mechanism, can proactively "search" for and enhance residual weak patterns in EMG related to specific actions, and vice versa, utilizing the intention features of EEG. This bidirectional dynamic calibration mechanism enables the system to effectively extract discriminative features even under low signal-to-noise ratio conditions, thereby significantly improving the accuracy and reliability of resolving weak motor intentions (verified by a higher Kappa consistency coefficient).

[0137] Clear interpretability and feature decoupling capability: Feature visualization analysis confirms that CATA-Net can map raw, highly overlapping neuromuscular signals to well-defined clusters in a high-dimensional feature space. For example, in the wrist twisting task of motor imagination, the feature manifold exhibits an orthogonal "V"-shaped topology corresponding to antagonistic muscle groups. This intuitively demonstrates that the model successfully learns and separates the essential neural representations corresponding to different motor intentions, rather than simply fitting data, enhancing the credibility and scientific rigor of the method.

[0138] In summary, the effectiveness of CATA-Net lies in its innovative dual-stream architecture and symmetrical bidirectional fusion mechanism designed for heterogeneous signal characteristics. This effectively addresses core challenges in unilateral upper limb fine motor decoding, such as overlapping neural representations, signal non-stationarity, and significant individual variability. It not only achieves leading levels in various quantitative indicators but, more importantly, makes breakthroughs in key dimensions crucial for practical applications, including robustness, generalization, and efficiency in utilizing weak signals. This provides an effective core technical solution for developing a new generation of highly reliable, user-friendly, and personalized brain-computer interface rehabilitation systems.

[0139] The above-disclosed embodiments are merely preferred embodiments of the present invention and should not be construed as limiting the scope of the invention. Those skilled in the art will understand that implementing all or part of the above-described embodiments and making equivalent changes in accordance with the claims of the present invention are still within the scope of the invention.

[0140] [1] VJ Lawhern, AJ Solon, NR Waytowich, SM Gordon, CPHung, and BJ Lance, “EEGNet: a compact convolutional neural network for EEG-based brain–computer interfaces,” J. Neural Eng., vol. 15, no. 5, p.056013, 2018.

[0141] [2] R. Mane et al., "FBCNet: A multi-view convolutional neural network for brain-computer interface," ArXiv Prepr. ArXiv210401233, 2021.

[0142] [3] H. Altaheri, G. Muhammad, and M. Alsulaiman, “Physics-informedattention temporal convolutional network for EEG-based motor imageryclassification,” IEEE Trans. Ind. Inform., vol. 19, no. 2, pp. 2249–2258,2022.

[0143] [4] T. Sultan, G. Liu, P. Sikorski, S. Alshathri, W. El-Shafai, andM. Babaiasl, “NeuroFusion-Trans: A Novel Transformer-Based EEG-EMG FusionModel for Assistive Robotics,” IEEE Access, 2025.

[0144] [5] X. Tang, J. Li, X. Li, H. Zhang, X. Dang, and B. Chen, “Dual-Stream Hybrid Network Based on Global and Local Spectral Fusion for DecodingEEG and sEMG Fusion Signals,” IEEE Sens. J., 2025.

[0145] [6] G. Jiang, K. Wang, Q. He, and P. Xie, “E^2FNet: An EEG and EMG-Based Fusion Network for Hand Motion Intention Recognition,” IEEE Sens. J.,2024.

Claims

1. A method for electroencephalography-electromyography synergistic limb movement intention decoding based on symmetric cross-modal attention network, characterized in that, The method comprises the following steps: obtaining an electroencephalogram signal and an electromyogram signal to be decoded; inputting the electroencephalogram signal into an electroencephalogram channel-specific feature extraction network to extract multi-scale spatio-temporal oscillation features as electroencephalogram features; the electroencephalogram channel-specific feature extraction network comprises a depth separable convolution module and a hollow spatial pyramid pooling module; inputting the electromyogram signal into an electromyogram channel-specific feature extraction network to extract dynamic temporal pattern features as electromyogram features; the electromyogram channel-specific feature extraction network comprises a hollow residual temporal convolution network; inputting the electroencephalogram features and the electromyogram features into a symmetric cross-modal attention module to perform bidirectional feature interaction and calibration to generate fusion enhanced features; the symmetric cross-modal attention module uses cross-attention mechanism to take its own features as queries and the features of the other modality as keys and values to realize mutual enhancement and alignment between the electroencephalogram signal and the electromyogram signal; and inputting the fusion enhanced features into a classifier to obtain corresponding movement intention categories.

2. The electroencephalography-electromyography synergic limb movement intention decoding method based on symmetric cross-modal attention network according to claim 1, wherein, The symmetric cross-modal attention module performs bidirectional feature interaction and calibration, specifically including: a first attention path: taking the electroencephalogram features as queries and the electromyogram features as keys and values to calculate attention weights and output electroencephalogram enhanced features calibrated by the electromyogram features; a second attention path: taking the electromyogram features as queries and the electroencephalogram features as keys and values to calculate attention weights and output electromyogram enhanced features calibrated by the electroencephalogram features; and fusing the electroencephalogram enhanced features and the electromyogram enhanced features to obtain the fusion enhanced features.

3. The EEG-EMG synergic limb movement intention decoding method based on symmetric cross-modal attention network according to claim 1, characterized in that, The electroencephalogram channel-specific feature extraction network uses the hollow spatial pyramid pooling module to set convolution layers with different hole rates in parallel to capture rhythm oscillation information of different time scales in the electroencephalogram signal.

4. The electroencephalography-electromyography synergic limb movement intention decoding method based on symmetric cross-modal attention network according to claim 1, wherein, The hollow residual temporal convolution network in the electromyogram channel-specific feature extraction network stacks causal convolution layers with expansion factors to increase the receptive field and capture long-range temporal dependencies in the electromyogram signal.

5. The EEG-EMG synergic limb motor intention decoding method based on symmetric cross-modal attention network according to any one of claims 1-4, characterized in that, The movement intention categories include at least one of the following: stretching of the arm in different spatial directions, grasping mode of the hand on different objects, pronation or supination twist of the wrist.

6. An electroencephalography-electromyography synergic limb movement intention decoding system based on symmetric cross-modal attention network, characterized in that, The system for implementing the method of any one of claims 1-5 comprises: a signal acquisition module for synchronously acquiring electroencephalogram signals and electromyogram signals of a user; an electroencephalogram feature extraction module composed of a depth separable convolution module and a hollow spatial pyramid pooling module, configured to receive the electroencephalogram signals and extract multi-scale spatio-temporal oscillation features; an electromyogram feature extraction module composed of a hollow residual temporal convolution network, configured to receive the electromyogram signals and extract dynamic temporal pattern features; a symmetric cross-modal attention fusion module configured to receive the electroencephalogram features and the electromyogram features, perform feature interaction and calibration through bidirectional cross-attention mechanism, and output fusion enhanced features; an intention decoding module configured to receive the fusion enhanced features and decode and output corresponding movement intention categories.

7. The EEG-EMG synergic limb movement intention decoding system based on symmetric cross-modal attention network according to claim 6, wherein, The symmetric cross-modal attention fusion module comprises: a first cross-attention submodule configured to take the electroencephalogram features as queries and the electromyogram features as keys and values to output electroencephalogram enhanced features calibrated by the electromyogram; and The second cross-attention sub-module is configured to take the electromyographic features as queries, take the electroencephalographic features as keys and values, and output electroencephalographic-calibrated electromyographic enhanced features; and the feature fusion sub-module is configured to fuse the electroencephalographic enhanced features and the electromyographic enhanced features.

8. The EEG-EMG synergic limb motor intention decoding system based on symmetric cross-modal attention network according to claim 6 or 7, characterized in that, The motion intention category includes fine movements corresponding to a unilateral upper limb, including at least one of an arm extension direction, a hand grip type, and a wrist rotation direction.

9. The EEG-EMG synergic limb movement intention decoding system based on symmetric cross-modal attention network according to claim 6, wherein, The system is part of a brain-computer interface rehabilitation training system, and the output of the intention decoding module is used to drive an external rehabilitation device or provide neural feedback.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program, when executed by a processor, implements the steps of the method of any one of claims 1 to 5.

Images