A rehabilitation control and evaluation method and system based on fusion of EEG and EMG

Abstract

The embodiment of the application provides a rehabilitation control and evaluation method and system based on EEG and EMG fusion, and relates to the technical field of biological signal processing. The rehabilitation control and evaluation method based on EEG and EMG fusion comprises the following steps: acquiring a first electroencephalogram signal and a first electromyogram signal of a user collected currently; obtaining a reference electromyogram signal output by a generator based on the first electroencephalogram signal, the first electromyogram signal and the pre-trained generator; fusing the first electroencephalogram signal and the reference electromyogram signal to obtain a target biological signal used for action classification, and determining a current motion intention of the user based on the target biological signal.

Description

Technical Field

[0001] This invention relates to the field of biosignal processing technology, specifically to a rehabilitation control and assessment method and system based on the fusion of EEG and EMG. Background Technology

[0002] Electroencephalogram (EEG) and electromyogram (EMG) signals are common bio-information sources available for the human body, reflecting physiological states and behavioral intentions. Consequently, they are finding increasingly widespread application in the field of rehabilitation. For example, rehabilitation training systems can collect a patient's EEG or EMG signals, use algorithms to decode their motor intentions, and thus control rehabilitation devices such as prosthetics, exoskeletons, and functional electrical stimulation (fED) devices, assisting patients in their rehabilitation training. Through repeated training and active reinforcement, this type of rehabilitation training can promote neuroplasticity, improve motor and sensory functions, and accelerate the rehabilitation process.

[0003] However, most existing motor intention decoding technologies employ single-modal EEG decoding or single-modal EMG decoding. The former is limited by low spatial resolution, low signal-to-noise ratio, and susceptibility to artifacts, making it difficult to stably represent the fine-grained motor intentions of patients. While the latter more directly reflects muscle output, for patients with stroke, spinal cord injury, peripheral nerve injury, and abnormal muscle tone, the measured electromyography often contains pathological components such as spasticity, synergistic contraction, abnormal high-frequency noise, and insufficient muscle strength, thus failing to accurately reflect the patient's original motor intentions. Summary of the Invention

[0004] The purpose of this invention is to provide a rehabilitation control and assessment method and system based on EEG and EMG fusion. After acquiring the user's first EEG signal and first EMG signal, the shared semantic features obtained from the first EEG signal and first EMG signal are input into the generator to obtain a healthy prior reference EMG signal. This generation process is equivalent to constructing an ideal muscle response reference channel outside the patient's abnormal EMG, which can suppress non-intentional components caused by factors such as spasticity, muscle weakness, and abnormal co-contraction, and retain only the neural drive information related to the real motor intention. Subsequently, the first EEG signal and the reference EMG signal are fused across modally to obtain a more accurate target biosignal. Then, the motor intention recognition based on the target biosignal can more accurately identify the user's motor intention.

[0005] To achieve the above objectives, the present invention provides a rehabilitation control and assessment method based on the fusion of EEG and EMG, comprising: acquiring the first electroencephalogram (EEG) signal and the first electromyogram (EMG) signal of the currently collected user; Based on the first EEG signal, the first EMG signal, and the pre-trained generator, a reference EMG signal output by the generator is obtained. The first EEG signal is fused with the reference EMG signal to obtain a target biosignal for action classification, and the user's current movement intention is determined based on the target biosignal.

[0006] This invention also provides a rehabilitation control and assessment system based on EEG and EMG fusion, comprising: an EEG acquisition module, an EMG acquisition module, and a fusion decoding module; the EEG acquisition module is used to acquire the user's EEG signals and send them to the fusion decoding module; the EMG acquisition module is used to acquire the user's EMG signals and send them to the fusion decoding module; the fusion decoding module is used to execute the above-described rehabilitation control and assessment method based on EEG and EMG fusion.

[0007] The present invention also provides a computer-readable storage medium, which is a non-volatile or non-transient storage medium, on which a computer program is stored, and the computer program is executed by a processor to perform the steps of the rehabilitation control and assessment method based on the fusion of EEG and EMG as described above.

[0008] In one embodiment, the generator is trained as follows: Acquire multiple second EEG signals and multiple second EMG signals, along with corresponding action tags; The second EEG signal and the second EMG signal corresponding to the same time window and the same action label are projected into the same shared semantic space to obtain positive training sample pairs; the second EEG signal and the second EMG signal corresponding to different action labels are projected into the same shared semantic space to obtain negative training sample pairs. Construct a generator that takes EEG and EMG signals projected into the same shared semantic space as input and healthy EMG signals as output; The generator is trained using the positive training sample pairs and the negative training sample pairs to obtain the final generator, which uses the sum of the time-domain error term and the frequency-domain error term as the training loss term during the training process.

[0009] In one embodiment, the generator includes: a channel-sharing module and a channel-independent module; Based on the first EEG signal, the first EMG signal, and the pre-trained generator, a reference EMG signal output by the generator is obtained, including: The first EEG signal and the first EMG signal are projected onto a shared semantic space to obtain shared semantic features; The channel sharing module extracts action-related features that are action-related but unrelated to electromyography channels from the shared semantic features. The waveforms of each electromyographic channel are reconstructed based on the action-related features by the independent channel module to obtain reference electromyographic signals for multiple electromyographic channels.

[0010] In one embodiment, the first electroencephalogram (EEG) signal is fused with the reference electromyogram (EMG) signal to obtain a target biosignal for action classification, including: The corresponding feature information is extracted from the first electroencephalogram (EEG) signal and the reference electromyogram (EMG) signal, respectively. The feature information includes: temporal features and spatial features. The temporal and spatial features of the first EEG signal and the reference EMG signal are fused to obtain the target biosignal for action classification.

[0011] In one embodiment, the temporal and spatial features of the first electroencephalogram (EEG) signal and the reference electromyogram (EMG) signal are fused to obtain a target biosignal for action classification, including: Based on the cross-attention mechanism, the temporal features of the first EEG signal and the temporal features of the reference EMG signal are fused to obtain temporal cross-attention features. Based on the cross-attention mechanism, the spatial features of the first EEG signal and the spatial features of the reference EMG signal are fused to obtain spatial cross-attention features. The target biological signal is obtained by combining the temporal cross-attention features and the spatial cross-attention features.

[0012] In one embodiment, the temporal features of the first EEG signal and the temporal features of the reference EMG signal are fused based on a cross-attention mechanism to obtain temporal cross-attention features, including: The temporal cross-attention feature is obtained by fusing the temporal features of the first EEG signal as the Q vector and the temporal features of the reference EMG signal as the K and V vectors; or... The temporal cross-attention feature is obtained by fusing the temporal features of the reference electromyography signal as the Q vector and the temporal features of the first electroencephalogram (EEG) signal as the K and V vectors.

[0013] In one embodiment, spatial features of the first EEG signal and spatial features of the reference EMG signal are fused based on a cross-attention mechanism to obtain spatial cross-attention features, including: The spatial cross-attention feature is obtained by fusing the spatial features of the first EEG signal as the Q vector and the spatial features of the reference EMG signal as the K and V vectors; or... The spatial cross-attention features are obtained by fusing the spatial features of the reference electromyography signal as the Q vector and the spatial features of the first electroencephalogram (EEG) signal as the K and V vectors.

[0014] In one embodiment, the method further includes: Calculate the temporal attention score between the temporal features of the first EEG signal and the temporal features of the reference EMG signal, and the spatial attention score between the spatial features of the first EEG signal and the spatial features of the reference EMG signal.

[0015] In one embodiment, after acquiring the first electroencephalogram (EEG) signal and the first electromyogram (EMG) signal of the currently collected user, the method further includes: The first EEG signal and the first EMG signal are preprocessed respectively, and the preprocessing includes at least one of the following: bandpass filtering, notch filtering to remove power frequency, artifact suppression, time synchronization, segmentation, standardization, and data augmentation. Attached Figure Description

[0016] Figure 1 This is a schematic diagram of the structure of the rehabilitation control and assessment system based on EEG and EMG fusion applied in the rehabilitation control and assessment method based on EEG and EMG fusion in the first embodiment of the present invention. Figure 2 This is a flowchart of the rehabilitation control and assessment method based on the fusion of EEG and EMG in the first embodiment of the present invention. Figure 3 This is a schematic diagram of the motion intent recognition process in the first embodiment of the present invention; Figure 4 This is a schematic diagram of the reaching hand action task and the lifting weight action task in the first embodiment of the present invention; Figure 5 This is a schematic diagram of the generator in the first embodiment of the present invention; Figure 6 This is a schematic diagram of feature extraction from the first EEG signal and the reference EMG signal in the first embodiment of the present invention. Detailed Implementation

[0017] The embodiments of the present invention will be described in detail below with reference to the accompanying drawings to provide a clearer understanding of the purpose, features, and advantages of the present invention. It should be understood that the embodiments shown in the drawings are not intended to limit the scope of the present invention, but are merely illustrative of the essential spirit of the technical solution of the present invention.

[0018] In the following description, certain specific details are set forth for the purpose of illustrating various disclosed embodiments in order to provide a thorough understanding of the various disclosed embodiments. However, those skilled in the art will recognize that embodiments may be practiced without one or more of these specific details. In other instances, well-known apparatuses, structures, and techniques associated with this application may not have been shown or described in detail to avoid unnecessarily obscuring the description of the embodiments.

[0019] Unless the context requires otherwise, throughout the specification and claims, the word “comprising” and its variations, such as “including” and “having”, shall be understood to have an open, inclusive meaning, that is, to be interpreted as “including, but not limited to”.

[0020] Throughout this specification, references to "an embodiment" or "an embodiment" indicate that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment. Therefore, the appearance of "in an embodiment" or "an embodiment" in various places throughout the specification does not necessarily refer to the same embodiment. Furthermore, a particular feature, structure, or characteristic may be combined in any manner in one or more embodiments.

[0021] The singular forms “a” and “the” used in this specification and the appended claims include plural references unless otherwise expressly stated herein. It should be noted that the term “or” is generally used to include the meaning of “or / and” unless otherwise expressly stated herein.

[0022] In the following description, in order to clearly demonstrate the structure and working method of the present invention, a number of directional terms will be used. However, terms such as "front", "back", "left", "right", "outer", "inner", "outer", "inner", "up", and "down" should be understood as convenient terms and not as limiting terms.

[0023] Existing EEG-EMG fusion studies typically employ feature splicing, shallow attention weighting, or conventional two-branch networks for joint classification. Common problems include: first, directly using the patient's raw EMG data as effective input easily incorporates pathological noise into the discrimination space; second, the lack of explicit modeling of a stable physiological mapping from EEG to ideal muscle response makes the model more prone to fitting individual differences and pathological states rather than learning true motor neural drives; and third, in cross-domain scenarios involving training in healthy individuals and testing in patients, significant inter-domain differences lead to poor model generalization performance, making it difficult to truly apply in clinical practice.

[0024] Most existing signal reconstruction methods focus on signal restoration, interpolation, or compression reconstruction within a single modality, lacking a design that treats cross-modal reconstruction as a "representation constraint," making it difficult to separate the true motor intent from abnormal patient signals using health priors. Furthermore, existing methods do not adequately consider model interpretability and clinical feasibility, lacking stable outputs that can directly serve rehabilitation control.

[0025] Therefore, there is an urgent need to propose a new approach that differs from the traditional "simple brain-muscle fusion". This approach would allow the model to first learn the relatively stable neural conduction patterns between the brain and muscles in a healthy state, and then transfer these patterns to the patient's side to generate an electromyographic representation that is closer to the healthy prior. Based on this representation, the model would be deeply fused with the patient's EEG, thereby achieving robust, accurate, and interpretable decoding of fine motor intentions.

[0026] The technical problem to be solved by this invention is: to address the problems of pathological noise pollution, difficulty in cross-domain transfer between healthy individuals and patients, insufficient ability to distinguish fine motor categories, and poor stability of clinical control signals in existing EEG-EMG fusion methods in patient groups, and to provide a brain-computer rehabilitation fine motor intention decoding method and system based on health prior generation and comparative reconstruction.

[0027] Specifically, the present invention aims to solve at least some of the following problems: 1. How to learn a more stable “central motor command-peripheral muscle response” mapping from synchronized EEG-EMG samples from healthy individuals, rather than just learning superficial statistical correlations; 2. How to use the above mapping to generate healthy prior electromyographic representations from patients' EEG signals that are free from pathological interference and retain true motor intentions; 3. How to achieve higher accuracy, stronger generalization, and better interpretability in multiple categories of tasks such as fine motor skills, grasping, stretching, and lifting in patients; 4. How to make the decoding results serve as a stable control input for devices such as rehabilitation robots, exoskeletons, and FES (functional electrical stimulation therapy devices) to support closed-loop clinical rehabilitation applications.

[0028] The first embodiment of this invention relates to a rehabilitation control and assessment method based on the fusion of EEG and EMG, applied to a fusion decoding module in a rehabilitation control and assessment system based on the fusion of EEG and EMG. This fusion decoding module can be deployed in electronic devices such as mobile phones, computers, cloud servers, edge devices, exoskeletons, rehabilitation robots, FES devices, and brain-computer interfaces, i.e., it is installed in the chip of the device, such as the processor chip of a rehabilitation robot. Please refer to [reference needed]. Figure 1The rehabilitation control and assessment system based on EEG and EMG fusion includes: an EEG acquisition module 1, an EMG acquisition module 2, and a fusion decoding module 3. The fusion decoding module 3 is communicatively connected to both the EEG acquisition module 1 and the EMG acquisition module 2. The EEG acquisition module 1 can be a dry electrode or wet electrode acquisition device, such as an EEG acquisition cap. After the user wears the cap, the acquisition electrodes contact the user's scalp to acquire EEG signals. The EMG acquisition module 2 can be an electrode lead type, such as a wristband device. After the user wears the wristband device, the EMG acquisition electrodes are attached to the user's arm to acquire EMG signals from the user's arm muscles.

[0029] The specific process of the rehabilitation control and assessment method based on the fusion of EEG and EMG in this embodiment is as follows: Figure 2 As shown; please refer to. Figure 3 This is a schematic diagram of the entire motion intent recognition process.

[0030] Step 101: Obtain the first EEG signal and the first EMG signal of the currently collected user.

[0031] Specifically, after the user wears the EEG acquisition module and the EMG acquisition module, the EEG acquisition module and the EMG acquisition module respectively acquire the first EEG signal and the first EMG signal in real time, and send the acquired first EEG signal and the first EMG signal to the fusion decoding module 3 in real time.

[0032] The fusion decoding module 3 preprocesses the received first EEG signal and first EMG signal respectively to obtain the first EEG signal and first EMG signal used subsequently; the preprocessing includes at least one of the following: bandpass filtering, notch filtering to remove power frequency, artifact suppression, time synchronization, segmentation, standardization and data augmentation.

[0033] The preprocessed first EEG signal and the first EMG signal are then time-aligned, and the time-aligned first EEG signal and the first EMG signal are then further processed.

[0034] The first EEG signal is characterized as B1 represents the number of electromyographic signals, C1 represents the number of electroencephalogram (EEG) channels, and T1 represents the time series length (number of sampling points) of the EEG signal. The first electromyographic signal is represented as: B2 represents the number of electromyographic signals, C2 represents the number of electromyographic channels, and T2 represents the time series length (number of sampling points) of the electromyographic signals; usually B1=B2.

[0035] Step 102: Based on the first EEG signal, the first EMG signal, and the pre-trained generator, obtain the reference EMG signal output by the generator.

[0036] Specifically, the pre-trained generator takes the first EEG signal and the first EMG signal as input directly or indirectly, and takes the reference EMG signal as output. The reference EMG signal is a healthy prior EMG representation. For example, the generator includes: a channel-sharing module and a channel-independent module with communication connections; The first EEG signal and the first EMG signal are projected onto a shared semantic space to obtain shared semantic features; The channel sharing module extracts action-related features that are action-related but unrelated to electromyography channels from the shared semantic features. The waveforms of each electromyographic channel are reconstructed based on the action-related features by the independent channel module to obtain reference electromyographic signals for multiple electromyographic channels.

[0037] Specifically, feature extraction is first performed on the first EEG signal and the first EMG signal. For example, the first EEG signal and the first EMG signal are respectively input into the corresponding dual-stream spatiotemporal encoder to extract the EEG temporal features and EEG spatial features from the first EEG signal, forming EEG-related features; and the EMG temporal features and EMG spatial features from the first EMG signal, forming EMG-related features. The dual-stream spatiotemporal encoder can be composed of sequentially connected one-dimensional convolution, two-dimensional convolution, max pooling layer, normalization, and nonlinear activation modules, used to extract fusionable representations while preserving the original physiological characteristics of each modality.

[0038] Subsequently, EEG-related features (EEG temporal features Z) were analyzed. e t Spatial characteristics of brainwaves Z e s ) and electromyographic related characteristics (electromyographic time characteristics Z) m t Spatial characteristics of electromyography Z m s The features extracted from the two modalities are projected onto the same shared semantic space, mapping them to the same latent semantic features. That is, the EEG-related features and EMG-related features are transformed into the same dimension embedding through the projection layer Proj to obtain the shared representation of the EEG-related features and EMG-related features, i.e., shared semantic features. By aligning the EEG-related features and EMG-related features at the semantic level, the representation gap between cortical commands (EEG) and muscle output (EMG) is reduced. The shared semantic features subsequently input into the generator are action intention-related features, rather than the original features that are not mixed with noise.

[0039] Shared semantic features are input into the channel sharing module, which extracts action-related features that are action-related but independent of electromyographic channels from the shared semantic features. The channel sharing module includes a linear transformation, a fully connected layer, and a temporal convolutional layer (or attention convergence layer) connected in sequence. After the shared semantic features are input into the channel sharing module, the channel sharing module can extract action-driven latent variables shared across channels. These action-driven latent variables are used to represent common neural drive information that is related to a specific action category but does not depend on a specific electromyographic channel, which is the action-related feature.

[0040] Action-related features are input into a channel-independent module, which contains multiple electromyographic (EMG) channel branches. The action-related features are broadcast to each EMG channel branch, and each EMG channel branch has corresponding channel embedding parameters. Thus, the action-related features are combined with the corresponding channel embedding parameters within each EMG channel branch, and the waveform is reconstructed through deconvolution, upsampling convolution, or a temporal decoding network to obtain the EMG waveform of that EMG channel. The multi-channel EMG waveform forms the reference EMG signal. The generator references the EMG features of healthy individuals during training, so the reference EMG signal also contains the EMG features of healthy individuals, which is the healthy prior EMG representation.

[0041] The generator is trained as follows: Acquire multiple second EEG signals and multiple second EMG signals, along with corresponding action tags; The second EEG signal and the second EMG signal corresponding to the same time window and the same action label are projected into the same shared semantic space to obtain positive training sample pairs; the second EEG signal and the second EMG signal corresponding to different action labels are projected into the same shared semantic space to obtain negative training sample pairs. Construct a generator that takes EEG and EMG signals projected into the same shared semantic space as input and healthy EMG signals as output; The generator is trained using the positive training sample pairs and the negative training sample pairs to obtain the final generator, which uses the sum of the time-domain error term and the frequency-domain error term as the training loss term during the training process.

[0042] The training process is detailed below: Step S1: First, collect sample data from the subject. The subject wears an EEG acquisition module and an EMG acquisition module. While the subject performs corresponding action tasks according to preset action prompts, second EEG signals and second EMG signals are simultaneously collected and labeled with corresponding action tags. The prompting actions include, but are not limited to: upper limb extension direction, grasping type, different weight-bearing lifting movements, finger flexion and extension movements, and other fine motor skills. Then, corresponding action tags can be assigned to the collected second EEG signals and second EMG signals of the subject, with the action tags indicating the action category. Figure 4 For example, a 2-second prompt time is given first, and the action task is prompted by voice or display. Then, a 4-second action task execution time is given, followed by a 2-second rest time, after which the subsequent action task can be carried out. The reaching action task includes four directions: up, down, left, and right. The lifting action task can be divided into three weight levels: 500g, 1000g, and 1500g.

[0043] This allows for the establishment of a multimodal sample library, which includes second electroencephalogram (EEG) signals, second electromyogram (EMG) signals, and corresponding action tags. The library includes multiple subjects, ranging from healthy subjects without muscle damage to non-healthy subjects with muscle damage.

[0044] Step S2 involves preprocessing the second EEG signal and the second EMG signal. Preprocessing methods include, but are not limited to, bandpass filtering, notch filtering to remove power frequency, artifact suppression, time synchronization, segmentation, standardization, and data augmentation.

[0045] Step S3 involves pairing the second EEG signal and the second EMG signal within the same time window to obtain multiple paired samples. The EEG and EMG signals in the paired samples can correspond to the same action label or different action labels. Each paired sample includes: positive training sample pairs consisting of second EEG signals and second EMG signals corresponding to the same time window and the same action label; and negative training sample pairs consisting of second EEG signals and second EMG signals corresponding to the same time window and different action labels. In other words, paired samples can be either positive or negative training sample pairs.

[0046] Step S4 involves inputting the second EEG signal and the second EMG signal from each paired sample into the corresponding dual-stream spatiotemporal encoder to extract the EEG temporal and spatial features from the second EEG signal, and the EMG temporal and spatial features from the second EMG signal. The dual-stream spatiotemporal encoder can be composed of sequentially connected one-dimensional convolution, two-dimensional convolution, pooling, normalization, and nonlinear activation modules, used to extract fusionable representations while preserving the original physiological characteristics of each modality. The same dual-stream spatiotemporal encoder is used in both the generator training and generator application phases.

[0047] The EEG-related features (including EEG temporal and spatial features) and EMG-related features (including EMG temporal and spatial features) of the second EEG signal and the second EMG signal in the same paired sample are simultaneously projected into the same shared semantic space, thereby aligning the EEG-related features and EMG-related features in the paired sample. This alignment can be achieved using a learning loss function, preferably in the form of an InfoNCE loss function, denoted as L. align =InfoNCE(ze , z m );z e Characterizing EEG-related features, z m It characterizes electromyographic (EMG) related features; thus, EEG and EMG related features corresponding to the same action label in paired samples are brought close to each other, while EEG and EMG related features corresponding to different action labels are moved away from each other. This enables consistent motion representation across modalities and provides a more stable cross-modal representation.

[0048] Based on the above process, shared semantic feature samples corresponding to each paired sample can be obtained. These shared semantic feature samples are action intention-related features, and are the original features without noise.

[0049] Step S5: Construct an EEG-to-EMG generator based on health priors. The generator can generate corresponding health prior reference EMG signals based on brain-muscle shared semantic features. The reference EMG signals contain health prior EMG features.

[0050] For example, please refer to Figure 5 The generator includes: a channel-sharing module and a channel-independent module for communication connections; The channel sharing module is used to extract action-related features that are action-related but independent of electromyographic channels from shared semantic features; The independent channel module contains multiple electromyographic (EMG) channel branches. Each EMG channel branch has corresponding channel embedding parameters. The EMG waveform of the EMG channel can be obtained through waveform reconstruction by deconvolution, upsampling convolution, or a temporal decoding network. Each EMG channel branch is connected to a channel-specific ConvTranspose1d deconvolution decoding branch. The waveform is reconstructed by the deconvolution decoding branch to ensure that the differentiated muscle activation patterns of each EMG channel can be recovered. The EMG waveforms of multiple EMG channels form a multi-channel health prior EMG output.

[0051] Therefore, the generator can be trained using shared semantic feature samples corresponding to all paired samples. During training, a composite constraint loss is used as the training loss term. The composite constraint loss can be determined by comparing the reference electromyography (EMG) signal reconstructed by the generator with the actual EMG signal. The composite constraint loss L... rec Includes: time-domain error term L time With frequency domain error term L freq Time-domain error term L time To apply mean square error or absolute error constraints in the time domain, and frequency domain error term L... freq To employ spectral loss constraints based on Short-Time Fourier Transform (STFT) in the frequency domain, L rec =λ1×L time +λ2×L freqThe sum of λ1 and λ2 is 1 to ensure that the generated result maintains both waveform consistency and spectral physiological rationality.

[0052] Through training, a generator that meets the training loss requirements can be obtained.

[0053] It is evident that the shared semantic features obtained from the first EEG signal and the first EMG signal are input into the generator, which can obtain a healthy prior reference EMG signal. This generation process is equivalent to constructing an ideal muscle response reference channel outside the patient's abnormal EMG, which can suppress unintentional components caused by factors such as spasticity, muscle weakness, and abnormal co-contraction, and retain only the neural drive information related to the real motor intention.

[0054] Step 103: The first EEG signal is fused with the reference EMG signal to obtain a target biosignal for action classification, and the user's current movement intention is determined based on the target biosignal.

[0055] Specifically, corresponding feature information is extracted from the first EEG signal and the reference EMG signal, respectively. The feature information includes: temporal features and spatial features. Here, the EEG temporal features and EEG spatial features in the first EEG signal and the EMG temporal features and EMG spatial features in the reference EMG signal can be extracted using the dual-stream spatiotemporal encoder in step 102.

[0056] Specifically, the dual-stream spatiotemporal encoder includes a temporal feature extraction module and a spatial feature extraction module, which can be a CNN convolutional neural network.

[0057] The fusion decoding module extracts the temporal feature Z of the first EEG signal from the first EEG signal and the reference EMG signal respectively through the temporal feature extraction module. e t The time characteristics Z of the reference electromyographic signal m t The temporal feature extraction module includes a one-dimensional convolutional layer (Conv1D), a batch normalization layer (BN), an exponential linear unit layer (ELU), a max pooling layer (MaxPool), and a dropout layer, connected in sequence; it can output temporal features.

[0058] The spatial feature Z of the first EEG signal was extracted from the first EEG signal and the reference EMG signal using the spatial feature extraction module. e s Spatial characteristics Z of reference electromyography signal m sThe spatial feature extraction module includes a series of interconnected 2D convolutional layers (Conv2D), batch normalization (BN), exponential linear unit (ELU), max pooling (MaxPool), and dropout layers, which can output spatial dimensional features.

[0059] For example, please refer to Figure 6 The first temporal feature extraction module for extracting temporal features from the first EEG signal includes a first convolutional layer, a first max-pooling layer, a second convolutional layer, and a second max-pooling layer connected in sequence. Specifically, the input to the first convolutional layer is the first EEG signal. The kernel of the first convolutional layer is (8, 4), and its output data is , The data is input into the first max pooling layer, which has a pooling kernel of 4, and its output data is... , The data is input into the second convolutional layer, which has a kernel size of (16, 8), and its output data is... , The data is input into the second max-pooling layer, which has four pooling kernels. Its output data is the temporal characteristics of the EEG signal. .

[0060] The second time feature extraction module for extracting time features from the reference electromyography signal includes a third convolutional layer, a third max-pooling layer, a fourth convolutional layer, a fourth max-pooling layer, a fifth convolutional layer, and a fifth max-pooling layer connected in sequence; specifically, the input of the third convolutional layer is the electromyography signal. The kernel of the third convolutional layer is (3, 64), and its output data is , The data is input into the third max pooling layer, which has a pooling kernel of 4, and its output data is... , The data is input into the fourth convolutional layer, which has a kernel size of (3, 64), and its output data is... , The data is input into the fourth max pooling layer, which has a pooling kernel of 4, and its output data is... , The data is input into the fifth convolutional layer, which has a kernel size of (3, 64), and its output data is... , The data is input into the fifth max-pooling layer, which has a pooling kernel of 4. Its output data is the temporal characteristics of the electromyographic signal. .

[0061] The spatial feature extraction module for extracting spatial features from the first EEG signal and the reference EMG signal includes a sixth convolutional layer, a sixth max-pooling layer, a seventh convolutional layer, and a seventh max-pooling layer connected in sequence. That is, the spatial feature extraction modules for extracting spatial features from the first EEG signal and the reference EMG signal have the same structure; there can be two separate spatial feature extraction modules for extracting spatial features from the first EEG signal and the reference EMG signal, which will be described separately below: From the first brainwave signal When extracting spatial features, first analyze the first EEG signal. Performing a reshape operation can alter the primary EEG signal. The tensor, without altering the data of the first EEG signal, is the EEG signal after the reshape operation. ; The data is input into the sixth convolutional layer, which consists of two convolutional kernels (1, 2), and its output data is... , The data is input into the sixth max pooling layer, whose pooling kernel is (1,8), and its output data is... , The data is fed into the seventh convolutional layer, which consists of four convolutional kernels (1, 4), and its output data is... , The data is input into the seventh max pooling layer, whose pooling kernel is (1,8), and its output data is... Then on Perform a reshape operation to obtain the spatial features of the first EEG signal. .

[0062] From the reference electromyography signal When extracting spatial features, first analyze the reference electromyography signal. Performing a reshape operation can alter the reference electromyographic signal. The tensor, but without changing the data of the first EEG signal, the reference EMG signal after the reshape operation is... ; The data is input into the sixth convolutional layer, which consists of two convolutional kernels (1, 2), and its output data is... , The data is input into the sixth max pooling layer, whose pooling kernel is (1,8), and its output data is... , The data is fed into the seventh convolutional layer, which consists of four convolutional kernels (1, 4), and its output data is... , The data is input into the seventh max pooling layer, whose pooling kernel is (1,8), and its output data is... Then on Perform a reshape operation to obtain the temporal characteristics of the reference electromyography signal. .

[0063] The temporal and spatial features of the first EEG signal and the reference EMG signal are fused to obtain the target biosignal for action classification.

[0064] Specifically, based on a multimodal cross-attention mechanism, the temporal and spatial features of the first EEG signal and the reference EMG signal are fused, including: the temporal feature Z of the first EEG signal is fused using the cross-attention mechanism. e t The time characteristics Z of the reference electromyographic signal m t Feature fusion is performed to obtain temporal cross-attention features; based on the cross-attention mechanism, the spatial features Z of the first EEG signal are... e s Spatial characteristics Z of reference electromyography signal m s Feature fusion is performed to obtain spatial cross-attention features; the target biological signal is obtained by combining the temporal cross-attention features and the spatial cross-attention features.

[0065] Among them, the time feature Z of the first EEG signal is based on the cross-attention mechanism. e t The time characteristics Z of the reference electromyographic signal m t Feature fusion was performed to obtain temporal cross-attention features, including: the temporal feature Z of the first EEG signal. e t As the Q vector, the time feature Z of the reference electromyographic signal m t The temporal cross-attention features are obtained by fusing the K and V vectors; or, the temporal features Z of the reference electromyographic signal are used. m t As the Q vector, the time feature Z of the first EEG signal e t The temporal cross-attention features are obtained by fusing the K and V vectors.

[0066] Based on the cross-attention mechanism, the spatial features Z of the first EEG signal are analyzed. e s Spatial characteristics Z of reference electromyography signal m sFeature fusion is performed to obtain spatial cross-attention features, including: spatial features Z of the first EEG signal. e s As the Q vector, the spatial features Z of the reference electromyography signal are used. m s The spatial cross-attention features are obtained by fusing the K and V vectors; or, the spatial features Z of the reference electromyographic signal are used. m s As the Q vector, the spatial features Z of the first EEG signal e s The spatial cross-attention features are obtained by fusing the K and V vectors.

[0067] In one example, when fusing the temporal cross-attention features using the temporal features of the first EEG signal as the Q vector and the temporal features of the reference EMG signal as the K and V vectors, the spatial cross-attention features are obtained by fusing the spatial features of the reference EMG signal as the Q vector and the spatial features of the first EEG signal as the K and V vectors; or, when fusing the temporal cross-attention features using the temporal features of the reference EMG signal as the Q vector and the temporal features of the first EEG signal as the K and V vectors, the spatial cross-attention features are obtained by fusing the spatial features of the first EEG signal as the Q vector and the spatial features of the reference EMG signal as the K and V vectors.

[0068] For example, the time-crossing attention feature is obtained by fusing the time features of the first EEG signal as the Q vector (query vector), the time features of the reference EMG signal as the K vector (key vector), and the V vector (value vector), including: using the time features of the first EEG signal as the Q vector, calculating the attention score with the K vector of the time features of the reference EMG signal, and then combining it with the V vector of the time features of the reference EMG signal to calculate the time-crossing attention feature.

[0069] Q-vector of the time characteristics of the first EEG signal ,in This represents the preset linear transformation weight matrix. Indicates the temporal characteristics of the first EEG signal; K-vector of time characteristics of reference electromyographic signal ,in This represents the preset linear transformation weight matrix. Indicates the temporal characteristics of the reference electromyographic signal; V vector of reference electromyographic signal time characteristics ,in This represents the preset linear transformation weight matrix. Indicates the temporal characteristics of the reference electromyographic signal; Attention score is calculated by using the temporal characteristics of the first EEG signal as the Q vector and the temporal characteristics of the reference EMG signal as the K vector. ; in, Representing the feature dimension of the K vector, based on The output dimension is set accordingly; This represents the scaling factor.

[0070] Temporal Cross-Attention Features .

[0071] In some embodiments, the process of cross-modal fusing the reference electromyography (EMG) signal into the first electroencephalogram (EEG) signal is as follows: through the EEG-EMG cross-modal information interaction module (CrossTransformer) E-M This is achieved by fusing the features of the first EEG signal and the features of the reference EMG signal through a cross-attention module of two branches; in one of the cross-attention modules, the time feature Z of the first EEG signal is used as the basis for the fusion. e t The query modality Q and the time feature Z based on the reference electromyography signal are used as the query modality Q. m t By obtaining the K-key value and V-value, a cross-modal fusion temporal feature corresponding to the first EEG signal can be obtained; in the cross-attention module of another branch, the spatial feature Z of the first EEG signal is used. e s The query modality Q and the spatial feature Z based on the reference electromyography signal. m s By obtaining the K-key value and the V-value, a cross-modal fusion spatial feature corresponding to the first EEG signal can be obtained; the combination of the cross-modal fusion temporal feature and the cross-modal fusion spatial feature corresponding to the first EEG signal is the first EEG signal of cross-modal fusion.

[0072] The process of cross-modal fusing the first EEG signal into the reference EMG signal is as follows: through the EMG-EEG cross-modal information interaction module (CrossTransformer) M-E This is achieved by fusing the features of the first EEG signal and the features of the reference EMG signal through a cross-attention module of two branches; in one of the cross-attention modules, the time feature Z of the reference EMG signal is used as the basis for the fusion. m t As the query modality Q, and the time feature Z based on the first EEG signal e t By obtaining the K-key value and V-value, a cross-modal fusion temporal feature corresponding to the reference EMG signal can be obtained; in the cross-attention module of another branch, the spatial feature Z of the reference EMG signal is used.m s As the query modality Q, and the spatial feature Z based on the first EEG signal e s By obtaining the K-key value and V-value, a cross-modal fusion spatial feature corresponding to the reference electromyography (EMG) signal can be obtained; the combination of the cross-modal fusion temporal feature and the cross-modal fusion spatial feature corresponding to the reference EMG signal is the reference EMG signal for cross-modal fusion.

[0073] Each branch's cross-attention module comprises, in sequence: a cross-modal attention module (Cross-modal Attention), a residual connection and layer normalization module (Add&LN), a feedforward network module, and another residual connection and layer normalization module (Add&LN). Figure 3 In this model, each branch's cross-modal attention module consists of H sequentially connected cross-modal attention layers.

[0074] The first EEG signal after cross-modal fusion is combined with the reference EMG signal after cross-modal fusion to form the target biological signal. This allows for deep feature correlation between the first EEG signal and the reference EMG signal. Then, the first EEG signal after cross-modal fusion is spliced ​​with the reference EMG signal after cross-modal fusion to obtain a multimodal signal, which is the final target biological signal.

[0075] In this embodiment, the target biosignal integrates the characteristics of the first EEG signal and the reference EMG signal. It has the advantages of the first EEG signal being less prone to fatigue and unaffected by neuromuscular pathway damage, and the reference EMG signal having a high signal-to-noise ratio and being easy to decode. The characteristics of the first EEG signal and the reference EMG signal are deeply fused, and the resulting target physiological signal helps to improve the accuracy and stability of action paradigm classification. Furthermore, the multi-dimensional fusion analysis results of the EEG signal can provide more effective information for user rehabilitation assessment and improve the efficiency of rehabilitation training.

[0076] Subsequently, the user's current movement intention can be determined based on the target biosignal. For example, the user's movements can be classified based on the target biosignal to obtain a movement classification result. Specifically, the movement classification result obtained based on the target biosignal indicates the target movement currently parsed. For example, it can be a fine motor category such as upper limb extension direction, grasping type, different weight-bearing lifting movements, and finger flexion and extension movements. For example, when a user performs an action task, the user's true movement intention can be obtained through parsing; The final current movement intention can serve as a control command to drive exoskeletons, rehabilitation robots, functional electrical stimulation devices, virtual rehabilitation training systems, or assessment terminals; and can update thresholds, confidence levels, or individualized parameters based on online feedback to achieve closed-loop rehabilitation training.

[0077] In this embodiment, the fusion decoding module can also be obtained through pre-training when executing step 103. The fusion decoding module is preferably a dual-path cross-transformer or other cross-attention network, which completes cross-modal deep fusion through bidirectional query / key-value interaction and outputs action categories or continuous control commands. During training, the classification loss used is L. cls =CrossEntropy( In some scenarios, the contrast learning process, generator, and fusion decoding module can be trained together, and the total training loss is L. total =L cls +α•L align +β•L rec .

[0078] After training, a contrastive learning module, a generator, and a fusion decoding module that meet the loss requirements can be obtained.

[0079] Furthermore, the temporal attention score between the temporal characteristics of the first EEG signal and the temporal characteristics of the reference EMG signal, as well as the spatial attention score between the spatial characteristics of the first EEG signal and the spatial characteristics of the reference EMG signal, can be calculated separately; the temporal attention score and the spatial attention score can be used as reference information for rehabilitation assessment.

[0080] As can be seen from the above, this embodiment provides a method for decoding fine motor intentions in neuromuscular rehabilitation based on health prior generation and comparative reconstruction. Its core lies not in simply splicing EEG and EMG, but in constructing a neural drive bridging path that integrates EEG, health prior EMG, and fusion decoding. Through this path, the relatively stable brain-muscle coupling patterns in healthy individuals are first learned, and then corresponding health prior EMG representations are generated on the patient's side using EEG. This serves as an intermediate layer to filter pathological interference, ultimately achieving accurate recognition of true motor intentions. The core innovations of this embodiment are at least reflected in the following aspects: By using healthy prior electromyography (EMG) generation, pathological noise in the patient's original EMG is effectively filtered out, making the model more focused on the true motor intention rather than abnormal muscle performance. A brain-computer rehabilitation decoding approach with "healthy prior EMG generation" as an intermediate bridge is proposed, no longer using the patient's original EMG as the only muscle representation. We propose a joint learning mechanism of "contrast alignment + cross-modal reconstruction" to enable the model to learn cross-modal shared representations that are closer to real neural drives. Through contrast alignment and reconstruction constraints, we establish a more stable brain-muscle semantic correspondence in the shared space, thereby improving the generalization ability across subjects and across healthy / patient domains. A patient-side health prior transfer strategy is proposed to realize the knowledge transfer from healthy people to patients and improve the stability of fine motor decoding. Specifically, it can achieve higher classification accuracy and clearer feature separation in fine motor tasks such as stretching, lifting and grasping, which is suitable for fine-grained rehabilitation motor recognition. The generated health-prior electromyography has good physiological rationality and interpretability, which can help doctors observe the recovery of cortex-muscle coupling and improve clinical trust. The decoded output can be directly converted into rehabilitation control commands, which is convenient for integration with exoskeletons, robots, FES and digital rehabilitation platforms, and has clear application value. Thus, a complete technical solution covering methods, systems, equipment, media and closed-loop rehabilitation application scenarios can be simultaneously provided, which is convenient for clinical and industrial implementation.

[0081] The second embodiment of the present invention relates to a rehabilitation control and assessment system based on the fusion of EEG and EMG. Please refer to [reference needed]. Figure 1 The rehabilitation control and assessment system based on the fusion of EEG and EMG includes: EEG acquisition module 1, EMG acquisition module 2, and fusion decoding module 3. The fusion decoding module 3 is communicatively connected to EEG acquisition module 1 and EMG acquisition module 2, respectively.

[0082] The EEG acquisition module 1 is used to acquire the user's EEG signals and send them to the fusion decoding module 3. The EEG acquisition module 1 can be a dry electrode or wet electrode acquisition device, such as an EEG acquisition cap. After the user wears the EEG acquisition cap, the acquisition electrodes in it contact the user's scalp to acquire EEG signals.

[0083] The electromyography (EMG) acquisition module 2 is used to acquire the user's EMG signals and send them to the fusion decoding module 3. The EMG acquisition module 2 can be an electrode-lead type, such as a wristband-type device. After the user wears the wristband-type device, the EMG acquisition electrodes are attached to the user's arm to acquire the user's EMG signals.

[0084] The fusion decoding module 3 is used to execute the rehabilitation control and assessment method based on EEG and EMG fusion in the first embodiment.

[0085] Furthermore, the rehabilitation control and assessment system based on the fusion of EEG and EMG may also include: a synchronous preprocessing module, a health prior training module, a patient health prior electromyography generation module, a multimodal fusion decoding module, a rehabilitation control output module, and a data storage module.

[0086] In addition, this system may also include: a patient data quality assessment module, a confidence gating module, a cross-subject / cross-center adaptive module, a lightweight model deployment module, an electromyography visualization feedback module, and a pathological interference early warning module, among other extended units.

[0087] Since the first embodiment corresponds to this embodiment, this embodiment can be implemented in conjunction with the first embodiment. The relevant technical details mentioned in the first embodiment remain valid in this embodiment, and the technical effects achievable in the first embodiment can also be achieved in this embodiment. To reduce repetition, they will not be repeated here. Correspondingly, the relevant technical details mentioned in this embodiment can also be applied to the first embodiment.

[0088] The third embodiment of the present invention relates to a computer-readable storage medium, which is a non-volatile or non-transient storage medium, on which a computer program is stored. When the computer program is run by a processor, it executes the steps of the rehabilitation control and assessment method based on EEG and EMG fusion as described in the first embodiment.

[0089] The preferred embodiments of the present invention have been described in detail above, but it should be understood that, if necessary, aspects of the embodiments can be modified to utilize aspects, features, and concepts from various patents, applications, and publications to provide other embodiments.

[0090] In light of the detailed description above, these and other changes can be made to the embodiments. Generally, the terminology used in the claims should not be considered limited to the specific embodiments disclosed in the specification and claims, but should be understood to include all possible embodiments together with the full scope of equivalents enjoyed by these claims.

Claims

1. A rehabilitation control and assessment method based on the fusion of EEG and EMG, characterized in that, include: Acquire the first EEG and first EMG signals of the currently collected user; Based on the first EEG signal, the first EMG signal, and the pre-trained generator, a reference EMG signal output by the generator is obtained. The first EEG signal is fused with the reference EMG signal to obtain a target biosignal for action classification, and the user's current movement intention is determined based on the target biosignal. During training, the generator takes EEG and EMG signals projected into the same shared semantic space as input and healthy EMG signals as output. The generator includes a channel-sharing module and a channel-independent module. Based on the first EEG signal, the first EMG signal, and the pre-trained generator, a reference EMG signal output by the generator is obtained, including: The first EEG signal and the first EMG signal are projected onto a shared semantic space to obtain shared semantic features; The channel sharing module extracts action-related features that are action-related but unrelated to electromyography channels from the shared semantic features. The waveforms of each electromyographic channel are reconstructed based on the action-related features using the independent channel module to obtain reference electromyographic signals for multiple electromyographic channels.

2. The rehabilitation control and assessment method based on the fusion of EEG and EMG according to claim 1, characterized in that, The generator is trained as follows: Acquire multiple second EEG signals and multiple second EMG signals, along with corresponding action tags; The second EEG signal and the second EMG signal corresponding to the same time window and the same action label are projected into the same shared semantic space to obtain positive training sample pairs; the second EEG signal and the second EMG signal corresponding to different action labels are projected into the same shared semantic space to obtain negative training sample pairs. Construct a generator that takes EEG and EMG signals projected into the same shared semantic space as input and healthy EMG signals as output; The generator is trained using the positive training sample pairs and the negative training sample pairs to obtain the final generator, which uses the sum of the time-domain error term and the frequency-domain error term as the training loss term during the training process.

3. The rehabilitation control and assessment method based on the fusion of EEG and EMG according to claim 1, characterized in that, The first electroencephalogram (EEG) signal is fused with the reference electromyogram (EMG) signal to obtain a target biosignal for action classification, including: The corresponding feature information is extracted from the first electroencephalogram (EEG) signal and the reference electromyogram (EMG) signal, respectively. The feature information includes: temporal features and spatial features. The temporal and spatial features of the first EEG signal and the reference EMG signal are fused to obtain the target biosignal for action classification.

4. The rehabilitation control and assessment method based on the fusion of EEG and EMG according to claim 3, characterized in that, The temporal and spatial features of the first EEG signal and the reference EMG signal are fused to obtain the target biosignal for action classification, including: Based on the cross-attention mechanism, the temporal features of the first EEG signal and the temporal features of the reference EMG signal are fused to obtain temporal cross-attention features. Based on the cross-attention mechanism, the spatial features of the first EEG signal and the spatial features of the reference EMG signal are fused to obtain spatial cross-attention features. The target biological signal is obtained by combining the temporal cross-attention features and the spatial cross-attention features.

5. The rehabilitation control and assessment method based on the fusion of EEG and EMG according to claim 4, characterized in that, Based on the cross-attention mechanism, the temporal features of the first EEG signal and the temporal features of the reference EMG signal are fused to obtain temporal cross-attention features, including: The temporal cross-attention feature is obtained by fusing the temporal features of the first EEG signal as the Q vector and the temporal features of the reference EMG signal as the K and V vectors; or... The temporal cross-attention feature is obtained by fusing the temporal features of the reference electromyography signal as the Q vector and the temporal features of the first electroencephalogram (EEG) signal as the K and V vectors.

6. The rehabilitation control and assessment method based on the fusion of EEG and EMG according to claim 4, characterized in that, Based on the cross-attention mechanism, the spatial features of the first EEG signal and the spatial features of the reference EMG signal are fused to obtain spatial cross-attention features, including: The spatial cross-attention feature is obtained by fusing the spatial features of the first EEG signal as the Q vector and the spatial features of the reference EMG signal as the K and V vectors; or... The spatial cross-attention features are obtained by fusing the spatial features of the reference electromyography signal as the Q vector and the spatial features of the first electroencephalogram (EEG) signal as the K and V vectors.

7. The rehabilitation control and assessment method based on the fusion of EEG and EMG according to claim 3, characterized in that, The method further includes: Calculate the temporal attention score between the temporal features of the first EEG signal and the temporal features of the reference EMG signal, and the spatial attention score between the spatial features of the first EEG signal and the spatial features of the reference EMG signal.

8. The rehabilitation control and assessment method based on the fusion of EEG and EMG according to claim 1, characterized in that, After acquiring the first EEG and first EMG signals of the currently collected user, the process also includes: The first EEG signal and the first EMG signal are preprocessed respectively, and the preprocessing includes at least one of the following: bandpass filtering, notch filtering to remove power frequency, artifact suppression, time synchronization, segmentation, standardization, and data augmentation.

9. A rehabilitation control and assessment system based on the fusion of EEG and EMG, characterized in that, include: EEG acquisition module, EMG acquisition module, and fusion decoding module; The EEG acquisition module is used to acquire the user's EEG signals and send them to the fusion decoding module; The electromyography acquisition module is used to acquire the user's first electromyography signal and send it to the fusion decoding module; The fusion decoding module is used to execute the rehabilitation control and assessment method based on the fusion of EEG and EMG as described in any one of claims 1 to 8.

Images