EEG-EMG fusion rehabilitation control and evaluation method based on multi-band time-space cross attention

Abstract

The invention belongs to the field of physiological signal processing, and particularly designs an EEG-EMG fusion rehabilitation control and evaluation method based on multiband space-time cross attention, which is characterized by comprising the following steps: step 1, data acquisition and preprocessing; step 2, multiband spatial-temporal feature extraction; 3, multi-band spatial-temporal feature fusion is carried out; 4, performing cross attention fusion; and step 5, classification control. According to the method, through a multi-band time-space cross attention mechanism, the time-space correlation between EEG and EMG signals can be accurately captured, and the accuracy and robustness of a rehabilitation system in action normal form classification are remarkably improved. According to the method, the decoding precision of the motion intention is effectively improved, the control capability of rehabilitation equipment on fine motion is enhanced, and more accurate technical support is provided for personalized rehabilitation.

Description

Technical Field

[0001] This invention belongs to the field of physiological signal processing, and specifically relates to an EEG-EMG fusion rehabilitation control and assessment method based on multi-band spatiotemporal cross-attention. Background Technology

[0002] The applications of EEG (electroencephalography) and EMG (electromyography) signals in rehabilitation are gradually increasing, especially in the control of rehabilitation equipment such as mechanical prostheses, exoskeletons, and functional electrical stimulation. Existing rehabilitation systems acquire EEG or EMG signals and use algorithms to decode motor intentions, thereby achieving control of rehabilitation equipment. These systems promote neuroplasticity through training, improve motor and sensory functions, and accelerate the rehabilitation process.

[0003] However, existing EEG–EMG fusion methods still have several problems and limitations. Traditional EEG–EMG fusion methods mostly rely on manual feature extraction, performing fusion at the feature level, or combining the output of a multimodal classifier through a decision layer. While these methods can improve classification accuracy to some extent, they face several major problems: First, manual feature extraction methods rely too heavily on manual design for signal processing and fusion, making it difficult to capture the complex spatiotemporal correlations between EEG and EMG signals; second, many early methods lacked deep fusion of EEG and EMG signals during feature fusion, failing to effectively consider the interaction relationships between different signal modalities.

[0004] With the development of deep learning technology, fusion methods in recent years have shifted towards model-level fusion based on neural networks. This approach can directly perform end-to-end joint representation learning on the original signals or intermediate features, and it has achieved significant progress. For example, the CNN-LSTM fusion architecture proposed by Zhang et al. improved the detection accuracy of lower limb movements by compensating for the neuromuscular delay between EEG and EMG; the CNN-based model proposed by Tryon et al. achieved elbow movement classification by using EEG-EMG spectral input. Nevertheless, most existing fusion methods still have several key shortcomings: First, many methods rely on traditional movement paradigms and cannot fully reflect the complexity and diversity of natural human movement; second, most methods ignore the spatiotemporal coupling of EEG and EMG signals, failing to effectively capture the temporal dynamics and cross-channel spatial correlations during movement; finally, existing fusion methods mostly use a single frequency band or a fixed time window for signal processing, failing to fully utilize the multi-frequency physiological information in EEG signals. These problems limit their application effectiveness in practical rehabilitation scenarios. Summary of the Invention

[0005] To address the problems existing in current technologies, particularly the shortcomings of existing EEG-EMG fusion methods, this invention proposes a rehabilitation control and assessment method based on multi-band spatiotemporal cross-attention EEG-EMG fusion. This multi-band spatiotemporal cross-attention fusion method effectively overcomes the limitations of existing technologies. By deeply fusing EEG and EMG signals, it significantly improves the decoding accuracy of motor intentions and the control capabilities of rehabilitation equipment, providing reliable technical support for precise rehabilitation training.

[0006] Technical solution A method for EEG-EMG fusion rehabilitation control and assessment based on multi-band spatiotemporal cross-attention includes the following steps: Step 1: Data Acquisition and Preprocessing; Step 2: Multi-band spatiotemporal feature extraction; Step 3: Multi-band spatiotemporal feature fusion; Step 4: Cross-attention fusion; Step 5: Categorization and control.

[0007] Specifically, in step 1, The EEG and EMG acquisition devices simultaneously acquire brain and muscle signals. After data acquisition, the data is preprocessed by power supply notch filtering and bandpass filtering before downsampling.

[0008] The electroencephalogram (EEG) acquisition device includes: wet electrodes, dry electrodes, wired devices, and Bluetooth wireless devices; the electromyogram (EMG) acquisition device includes: electrode lead type and wristband type devices.

[0009] Specifically, in step 2, After preprocessing, the EEG and EMG signals are subjected to spatiotemporal feature extraction through the EEG Spatio-Temporal Feature Convolution (EEG STFConv) module and the EMG Spatio-Temporal Feature Convolution (EMG STFConv) module, respectively.

[0010] For EEG signals, multi-band processing is employed. Wavelet packet decomposition (WPD) is used to decompose the EEG signal into multiple frequency sub-bands, and the spatiotemporal features of each frequency band are extracted. After time-domain and spatial-domain convolution processing, the spatiotemporal feature representation of each frequency band is obtained.

[0011] The EMG signal is then directly processed by the spatiotemporal convolution module to extract spatiotemporal features.

[0012] Ultimately, the spatiotemporal feature sequences of both EEG and EMG were of similar size. ,in For batch size, For the number of channels, This represents the time length after feature extraction.

[0013] Specifically, both EEG and EMG spatiotemporal convolutional feature extractors include temporal and spatial feature extractors. The EEG / EMG temporal feature extractor consists of several cascaded one-dimensional convolutional layers and max-pooling layers to obtain temporal feature representations. The EEG / EMG spatial feature extractor first performs a dimensionality transformation, then passes through several cascaded two-dimensional convolutional layers and max-pooling layers, and finally performs another dimensionality transformation to obtain spatial feature representations.

[0014] Specifically, in step 3, The output of the EEG spatiotemporal convolution module is provided to the multi-band feature fusion module, which fuses spatiotemporal features from different frequency bands to enhance the representational capability of the EEG signal. An adaptive weighting mechanism is used to weight and fuse features from different frequency bands, ultimately generating a comprehensive EEG spatiotemporal feature representation, which is further used for subsequent cross-attention fusion and motion intent decoding.

[0015] Specifically, in step 4, Cross-attention fusion stage: This stage is based on the cross-attention mechanism, modality Features With mode Features The attention score is calculated and then compared with the modality. of The updated result was obtained through calculation. Fusion characteristics: in, , Modal Features and modes feature, , , The weight matrix is ​​a learnable matrix. For modality The query matrix, , Modal The key value matrix, To calculate the attention score, For the feature vector dimension, For the updated modality The fusion characteristics.

[0016] The modal cross-attention approach for EEG and EMG processes the temporal and spatial feature sequences of EEG using two separate cross-attention modules. One branch uses EEG as the Q-query modality and EMG as the K-key and V-value, while the other branch uses EMG as the Q-query modality and EEG as the K-key and V-value. A multi-head cross-attention mechanism is employed to update and fuse the temporal and spatial features of EEG and EMG respectively, thereby extracting deep-level correlation information between the two modalities. in, Representing the The fusion features of individual attention heads, among which For each attention head, the feature dimensions, This represents a feedforward neural network. The weight matrix is ​​a learnable matrix. Represents the fusion of multi-head cross-attention Modal characteristics, This indicates the cross-attention fusion module. Characteristic sequences representing EEG and EMG, This represents the matrix concatenation operation.

[0017] Specifically, in step 5, Classification and control stage: These fused features are spliced ​​together, classified through a fully connected layer, achieving effective fusion of EEG and EMG signals and action decoding. The probability of each action class is then output via Sotfmax. : in, These are time feature sequences of EEG and EMG, respectively. Spatial feature sequences of EEG and EMG, respectively. For learnable fully connected weights, For time-crossing attention fusion module, For the spatial cross-attention fusion module, the following is obtained: It is a probability vector.

[0018] The loss function used in the classification network is the cross-entropy loss. in, For cross-entropy loss, The number of categories representing actions. What category does it represent? This indicates whether it belongs to that category. This represents the probability of obtaining that category.

[0019] The categorizable range of movements covers movements of varying fineness, including left and right hand movements (2 categories), arm extension movements in different directions (4 categories), lifting movements of different weights (3 categories), and grasping movements of different objects (4 categories), among other movement paradigms.

[0020] Rehabilitation assessment phase: Spatial and temporal attention scores obtained during the brain-motor modal cross-attention fusion algorithm are used to assess rehabilitation. The spatial dimension reflects the degree of correlation between brain electrical channels (different brain regions) and muscle electrical channels (different muscles), while the temporal dimension reflects the distribution of brain-motor responses during movement execution. The attention scores of healthy subjects and patients at different stages during movement execution are analyzed and compared to conduct rehabilitation assessment.

[0021] Beneficial effects This invention utilizes a multi-band spatiotemporal cross-attention mechanism to accurately capture the spatiotemporal correlation between EEG and EMG signals, significantly improving the accuracy and robustness of rehabilitation systems in movement paradigm classification. The method effectively enhances the decoding accuracy of motor intentions, strengthens the control capabilities of rehabilitation equipment over fine motor skills, and provides more precise technical support for personalized rehabilitation. Attached Figure Description

[0022] Figure 1 This is a framework diagram of the rehabilitation control and assessment method based on EEG and EMG fusion with multi-band spatiotemporal cross-attention, as described in this invention. Figure 2 This is a schematic diagram of the adaptive multi-band feature fusion module according to an embodiment of the present invention; Figure 3 This is a schematic diagram of the EEG and EMG spatiotemporal convolution feature extractor in an embodiment of the present invention. Detailed Implementation

[0023] The technical solution provided in this application will be further described below with reference to specific embodiments and accompanying drawings. The advantages and features of this application will become clearer from the following description.

[0024] The present invention aims to solve the following technical problems: Existing fusion methods lack sufficient spatiotemporal coupling modeling: Most existing EEG–EMG fusion methods fail to effectively capture the spatiotemporal coupling between EEG and EMG signals, and cannot fully utilize the temporal dynamics and cross-channel spatial correlation between the two, resulting in insufficient accuracy of motion intent decoding and robustness of rehabilitation device control.

[0025] Limitations of manual feature extraction and fixed signal processing methods: Traditional EEG–EMG fusion methods rely on manual feature extraction, making the methods overly dependent on manual design for signal processing and fusion, and unable to adaptively capture the complex spatiotemporal patterns of the signal. Furthermore, many existing methods use fixed time windows or single frequency bands for signal processing, failing to fully exploit the multi-frequency band information in the EEG signal, thus limiting the accuracy of motion decoding.

[0026] Lack of adequate modeling of the complexity of natural motion: Most existing fusion methods are based on traditional motion paradigms, which are difficult to fully reflect the complexity and diversity of natural human motion, resulting in poor adaptability of the models in practical applications, especially when faced with diverse and daily activities.

[0027] This invention proposes an EEG and EMG fusion method based on multi-band spatiotemporal cross-attention. By deeply fusing EEG and EMG signals, it can more accurately capture the spatiotemporal coupling relationship between the two, effectively improve the decoding accuracy of movement intention and the control capability of rehabilitation equipment, thereby solving the problems of insufficient spatiotemporal modeling, single signal processing method, and insufficient modeling of the complexity of natural movement in the existing technology.

[0028] Example A method for EEG-EMG fusion rehabilitation control and assessment based on multi-band spatiotemporal cross-attention, such as Figure 1 This includes the following steps: Step S1: Data Acquisition and Preprocessing Data acquisition: A 64-channel EEG system and an 8-channel EMG armband were used to simultaneously acquire EEG and EMG signals during upper limb movement, with a sampling rate of 250 Hz.

[0029] Data preprocessing: Denoising of EEG and EMG signals was performed, including 50 Hz notch filtering and bandpass filtering (0.5–42 Hz for EEG, 10–500 Hz for EMG), and a 4-second data window before and after the motion task was extracted and standardized to eliminate individual differences and baseline drift.

[0030] Step S2: Multi-band spatiotemporal feature extraction EEG signal multi-band decomposition: Wavelet packet decomposition (WPD) is used to decompose the EEG signal into multiple frequency bands (δ, θ, α, β), and the spatiotemporal features of each frequency band are extracted.

[0031] Spatiotemporal convolution processing: For each frequency band, the EEG STFConv module is used to extract spatiotemporal features, including temporal convolution and spatial convolution operations, to extract the spatiotemporal feature sequence of each frequency band.

[0032] Spatiotemporal feature extraction of EMG signals: EMG signals are directly subjected to spatiotemporal convolution through the EMG STFConv module to extract their spatiotemporal features.

[0033] EEG and EMG spatiotemporal convolutional feature extractors, such as Figure 3 As shown.

[0034] In this embodiment, the input and output parameters of the EEG / EMG signal spatiotemporal feature extractor are shown in Table 1 and Table 2.

[0035] Table 1. Specific input and output parameters of the EEG temporal / spatial feature extractor. Table 2. Specific input and output parameters of the EMG temporal / spatial feature extractor. Step S3: Multi-band feature fusion (e.g.) Figure 2 ) Feature fusion: A comprehensive spatiotemporal feature representation of EEG is obtained by fusing features from different frequency bands using an adaptive weighting mechanism. This fusion operation dynamically adjusts the weights based on the contribution of different frequency bands to the current task.

[0036] Feature Dimension Compression: The spatiotemporal feature sequences of EEG and EMG are compressed and fused to form a unified spatiotemporal feature representation.

[0037] Step S4: Spatiotemporal Cross-Attention Fusion Cross-attention mechanism: This mechanism utilizes cross-attention modules (including temporal cross-attention CAF-T and spatial cross-attention CAF-S) to capture the spatiotemporal dependencies between EEG and EMG signals. Within this module, the spatiotemporal features of the EEG and EMG signals interact, updating their respective representations through an attention mechanism and enhancing the interactive information between them.

[0038] Bidirectional information fusion: EEG and EMG signals are fused bidirectionally in both the time and spatial domains to optimize the coupling effect between signals and enhance decoding accuracy.

[0039] Step S5: Classification and Rehabilitation Control Motion intent classification: The fused EEG and EMG features are input into a fully connected layer, and the class probability is output through the Softmax activation function for motion intent classification.

[0040] Rehabilitation equipment control: Control rehabilitation equipment (such as exoskeletons, prostheses, functional electrical stimulation devices, etc.) according to the classification results to provide precise motor feedback.

[0041] Rehabilitation assessment: The patient's rehabilitation progress is assessed by evaluating the accuracy of motor intention classification and real-time feedback from equipment control, and rehabilitation strategies are dynamically adjusted.

[0042] To verify the advantages of the present invention, this embodiment collected data from 12 subjects (labeled as S01-S12, of which S09-S12 were stroke patients). Data on the subjects' arm extension and grabbing tasks were collected, and 64 EEG-EMG synchronous data were collected for each subject for each action.

[0043] Complete the following comparisons: 1) The spatiotemporal convolution method in this invention was compared with other single-modal methods. 2) The method of this invention was compared with the current advanced EEG-EMG fusion methods. The experimental comparison results of the two task data are shown in Tables 3 and 4. It can be seen that the average accuracy of the method proposed in this invention reached 84.2% and 95.2% respectively, which is higher than other EEG-EMG fusion methods, demonstrating the benefits of this invention.

[0044] Table 3. Comparison of single-modal and EEG-EMG fusion methods in arm extension tasks. Table 4. Comparison of single-modal and EEG-EMG fusion methods in arm extension tasks. The inventiveness of this invention is mainly reflected in the following: 1. This invention innovatively solves the problem of insufficient spatiotemporal modeling in the fusion of EEG and EMG signals by introducing a multi-band spatiotemporal cross-attention mechanism. Through multi-band analysis, it can dynamically adapt to the different frequency band characteristics of EEG signals, enhance the ability to capture the complex spatiotemporal coupling relationship between EEG and EMG, thereby improving the accuracy of motion intent decoding and the control performance of rehabilitation equipment; 2. This invention can effectively model the complexity and diversity of natural human movements, especially when facing diverse daily activities, providing high adaptability. This solves the problem that existing methods cannot fully reflect the complexity of natural human movements, and improves the flexibility and robustness of the model in practical applications; 3. The weights obtained by the present invention through spatiotemporal cross-attention can reflect the degree of coupling between EEG and EMG in time and space, providing a powerful reference for rehabilitation assessment and neurofeedback regulation.

[0045] The above description is merely a description of preferred embodiments of this application and is not intended to limit the scope of this application in any way. Any changes or modifications made by those skilled in the art based on the above-disclosed technical content should be considered as equivalent and valid embodiments and fall within the scope of protection of the technical solution of this application.

Claims

1. A method for EEG-EMG fusion rehabilitation control and assessment based on multi-band spatiotemporal cross-attention, characterized in that, Includes the following steps: Step 1: Data Acquisition and Preprocessing; Step 2: Multi-band spatiotemporal feature extraction; Step 3: Multi-band spatiotemporal feature fusion; Step 4: Cross-attention fusion; Step 5: Categorization and control.

2. The EEG-EMG fusion rehabilitation control and assessment method based on multi-band spatiotemporal cross-attention as described in claim 1, characterized in that, Step 1 is as follows: The EEG and EMG acquisition devices simultaneously acquire brain and muscle signals. After data acquisition, the data is downsampled after preprocessing by power supply notch filtering and bandpass filtering. The electroencephalogram (EEG) acquisition device includes: wet electrodes, dry electrodes, wired devices, and Bluetooth wireless devices; the electromyogram (EMG) acquisition device includes: electrode lead type and wristband type devices.

3. The EEG-EMG fusion rehabilitation control and assessment method based on multi-band spatiotemporal cross-attention as described in claim 1, characterized in that, Step 2 is as follows: After preprocessing, the EEG and EMG signals are subjected to spatiotemporal feature extraction through the EEG spatiotemporal convolution module and the EMG spatiotemporal convolution module, respectively. For EEG signals, multi-band processing is adopted. The EEG signal is decomposed into multiple frequency sub-bands by wavelet packet decomposition, and the spatiotemporal features of each frequency band are extracted. After the signal of each frequency band is processed by time domain and spatial domain convolution, the spatiotemporal feature representation of that frequency band is obtained. EMG signals are directly processed by a spatiotemporal convolution module to extract spatiotemporal features; Ultimately, the spatiotemporal feature sequences of both EEG and EMG were of similar size. ,in For batch size, For the number of channels, This represents the time length after feature extraction.

4. The EEG-EMG fusion rehabilitation control and assessment method based on multi-band spatiotemporal cross-attention as described in claim 3, characterized in that, Both EEG and EMG spatiotemporal convolutional feature extractors include temporal feature extractors and spatial feature extractors. The EEG / EMG temporal feature extractor consists of several cascaded one-dimensional convolutional layers and max pooling layers to obtain temporal feature representations. The EEG / EMG spatial feature extractor first performs a dimensionality transformation, then passes through several cascaded two-dimensional convolutional layers and max pooling layers, and finally performs another dimensionality transformation to obtain spatial feature representations.

5. The EEG-EMG fusion rehabilitation control and assessment method based on multi-band spatiotemporal cross-attention as described in claim 1, characterized in that, Step 3 specifically involves: The output of the EEG spatiotemporal convolution module is provided to the multi-band feature fusion module, which fuses the spatiotemporal features of different frequency bands to enhance the representation capability of the EEG signal. Through an adaptive weighting mechanism, the features of different frequency bands are weighted and fused to finally generate a comprehensive EEG spatiotemporal feature representation, which is further used for subsequent cross-attention fusion and motion intent decoding.

6. The EEG-EMG fusion rehabilitation control and assessment method based on multi-band spatiotemporal cross-attention as described in claim 1, characterized in that, Step 4 specifically involves: The cross-attention fusion stage is based on the cross-attention mechanism, modality Features With mode Features The attention score is calculated and then compared with the modality. of The updated result was obtained through calculation. Fusion characteristics: in, , Modal Features and modes feature, , , The weight matrix is ​​a learnable matrix. For modality The query matrix, , Modal The key value matrix, To calculate the attention score, For the feature vector dimension, For the updated modality The fusion characteristics; The modal cross-attention approach for EEG and EMG processes the temporal and spatial feature sequences of EEG using two separate cross-attention modules. One branch uses EEG as the Q-query modality and EMG as the K-key and V-value, while the other branch uses EMG as the Q-query modality and EEG as the K-key and V-value. A multi-head cross-attention mechanism is employed to update and fuse the temporal and spatial features of EEG and EMG respectively, thereby extracting deep-level correlation information between the two modalities. in, Representing the The fusion features of individual attention heads, among which For each attention head, the feature dimensions, This represents a feedforward neural network. The weight matrix is ​​a learnable matrix. Represents the fusion of multi-head cross-attention Modal characteristics, This indicates the cross-attention fusion module. Characteristic sequences representing EEG and EMG, This represents the matrix concatenation operation.

7. The EEG-EMG fusion rehabilitation control and assessment method based on multi-band spatiotemporal cross-attention as described in claim 1, characterized in that, Step 5 specifically involves: Classification and control stage: The spliced ​​and fused features are classified through a fully connected layer to achieve effective fusion of EEG and EMG signals and action decoding. The probability of each action class is then output via Sotfmax. : in, These are time feature sequences of EEG and EMG, respectively. Spatial feature sequences of EEG and EMG, respectively. For learnable fully connected weights, For time-crossing attention fusion module, For the spatial cross-attention fusion module, the following is obtained: It is a probability vector; The loss function used in the classification network is the cross-entropy loss. in, For cross-entropy loss, The number of categories representing actions. What category does it represent? This indicates whether it belongs to that category. This represents the probability of obtaining that category; Rehabilitation assessment phase: Spatial and temporal attention scores obtained during the brain-motor modal cross-attention fusion algorithm are used to assess rehabilitation. The spatial dimension reflects the degree of correlation between the brain electrical channels and the electromyographic channels, while the temporal dimension reflects the distribution of brain-motor responses during the execution of the movement. The attention scores of healthy subjects and patients at different stages during the execution of the movement are analyzed and compared to conduct rehabilitation assessment.

Images