Cross-subject domain adaptive motor imagery decoding method fusing EEG-fNIRS and application

Abstract

The invention discloses a cross-subject-domain adaptive motor imagery decoding method fusing EEG-fNIRS, and the method comprises the steps: analyzing collected decomposition actions of a clinical stroke patient during the execution of sitting-standing conversion rehabilitation training, and designing a 2D dynamic sitting-standing motor imagery stimulation experiment normal form based on E-Prime software; original EEG and fNIRS data under a subject motor imagery task are obtained and preprocessed; obtaining EEG-fNIRS data of a new subject, and segmenting a task state and a resting state according to a preset mark point to construct a target domain data set; loading the pre-trained cross-subject domain adaptive model, and performing fine tuning on the pre-trained model by using the target domain data set to realize classification of new subject motor imagery intentions; and a classification result output according to the optimized cross-subject domain adaptive model is converted into a control instruction. The method is applied to a rehabilitation training system. According to the invention, the accuracy and robustness of motor imagery intention recognition are improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of rehabilitation engineering and robot technology, and particularly relates to a cross-subject domain adaptive motor imagery decoding method fusing EEG-fNIRS and application thereof, which is applied to a rehabilitation training system, in particular, a hybrid brain-computer interface neural injury rehabilitation training system. BACKGROUND

[0002] Motor dysfunction is a major health problem that seriously affects the quality of life of patients, and is commonly seen in sequelae of stroke, spinal cord injury and neurodegenerative diseases. At present, the motor rehabilitation training in the clinic mainly relies on one-on-one physical assistance by a rehabilitation therapist. This mode generally has problems of high labor cost, passive training, low efficiency and high intensity. Therefore, developing a hybrid brain-computer interface neural injury rehabilitation training system that can stimulate the initiative of patients to actively participate and promote the remodeling of neural function is a key technical problem to be solved in the field.

[0003] Brain-computer interface (BCI) technology does not rely on peripheral nerves and muscle tissue, and directly establishes a communication path between the brain and external rehabilitation equipment, providing a new way for active rehabilitation. According to different stimulation induction paradigms, BCI paradigms are mainly divided into three categories: event-related potential (P300), steady-state visual evoked potential (SSVEP) and motor imagery (MI). Among them, the MI paradigm does not rely on external visual or auditory stimuli compared to the former two, and only through guiding patients to actively imagine limb movement, it can activate brain areas similar to motor execution, effectively promote brain neural plasticity, stimulate the initiative of patients to actively participate in rehabilitation training, and thus accelerate the recovery of motor function of patients.

[0004] Currently, electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) are two mainstream portable brain imaging technologies. EEG signals measure brain electrical activity through scalp electrodes, have high temporal resolution, and are suitable for capturing rapid neural electrical activity, but have low spatial resolution and are susceptible to noise interference; fNIRS signals indirectly reflect neural electrical activity by measuring changes in brain blood oxygen, have good spatial resolution, but have low temporal resolution. In view of the limitations of single modality EEG signals and fNIRS signals, hybrid brain-computer interface combines the complementary characteristics of neural electrical activity and hemodynamic response, utilizes the neural vascular coupling mechanism to represent the same brain function activity from different perspectives, fuses temporal and spatial resolution, automatically extracts more rich feature information based on deep learning algorithm, detects and decodes the brain signals of human motion intention and converts them into output control instructions, and finally realizes human-computer interaction, which plays a crucial role in the fields of motion control, neural rehabilitation training, etc.

[0005] In the prior art, for example, Chinese Patent No. CN113398422B discloses a "rehabilitation training system and method based on motor imagery-brain-computer interface and virtual reality", first, the subject wears VR glasses to perform motor imagery according to different training motion scene prompts; then the collected EEG signals are preprocessed and features are extracted; then a PSO-SVM model is established to identify left and right hand grasping tasks; finally, the recognition result is output to the corresponding limb in the virtual scene to complete the corresponding action.

[0006] Chinese Patent No. CN115969389B discloses a "motor imagery intention recognition method based on individual EEG signal migration", first, the existing individual EEG signal is obtained as the source signal, and the new individual EEG signal is obtained as the target signal; second, the source signal is corrected using the CSP spatial feature alignment and ERD / ERS intensity compensation method; then a separate LDA recognition model is constructed for each source individual and evaluated; finally, a multi-LDA integrated recognition model is established to identify the left and right hand motor imagery intention of the new individual.

[0007] Chinese invention patent application publication number CN120372438A discloses "a method for post-fusion decoding of motion imagery based on EEG-fNIRS data". First, EEG and fNIRS signals are preprocessed separately. Second, EEG is extracted based on a depthwise separable convolutional module of the Inception architecture, and fNIRS feature information is extracted based on a multi-scale convolutional network and a Transformer module. Then, the dimensions of EEG and fNIRS data are aligned and concatenated. Finally, the hybrid features are input into a multilayer perceptron for task classification.

[0008] However, the aforementioned existing technologies indicate that current motor imagery-hybrid brain-computer interface rehabilitation training systems still face two major problems that urgently need to be addressed: 1) Insufficient cross-modal fusion: How to efficiently fuse heterogeneous EEG-fNIRS signals and deeply mine complementary feature information between modalities to further improve the decoding accuracy of motor intentions; 2) Poor cross-subject generalization ability: Significant differences exist between individual signals, making it difficult to directly apply existing classification models to new subjects. Typically, a long period of calibration is required for each new subject, which greatly limits the clinical applicability of rehabilitation training systems. Summary of the Invention

[0009] To overcome the shortcomings of existing motor imagery-brain-computer interfaces, such as low decoding accuracy and poor model generalization, which limit their application in the rehabilitation field, this invention provides a cross-subject domain adaptive motor imagery decoding method and application that integrates EEG-fNIRS, thereby improving the accuracy and robustness of motor imagery intention recognition.

[0010] The technical solution adopted by this invention to solve its technical problem is:

[0011] A cross-subject domain adaptive motion imagery decoding method incorporating EEG-fNIRS includes the following steps:

[0012] Step 1: Analyze the decomposed movements of clinical stroke patients during sit-to-stand transition rehabilitation training, and design a 2D dynamic sit-to-stand motor imagery stimulation experiment paradigm based on E-Prime software.

[0013] Step 2: Build a dual-modal EEG-fNIRS signal synchronous acquisition system, locate the brain motor-sensory cortex area, determine the placement of light poles and electrodes, and synchronously acquire the raw EEG and fNIRS data of the subject under the motor imagery task.

[0014] Step 3: fNIRS data preprocessing. The original optical amplitude is converted into oxyhemoglobin (HbO) concentration changes using a modified Beer-Lambert law. A first-order polynomial fitting algorithm is used for linear detrending, and Temporal Derivative Distribution Repair (TDDR) is used for motion artifact correction. A third-order Infinite Impulse Response (IIR) filter is used to bandpass filter the data in the 0.01Hz and 0.04Hz range.

[0015] Step 4: EEG data preprocessing. A Butterworth bandpass filter (4-40Hz), a notch filter (50Hz), and a Chebyshev Type II filter are used for filtering; Independent Component Analysis (ICA) is employed to remove physiological artifacts such as those related to electrooculography.

[0016] Step 5: Obtain EEG-fNIRS data of new subjects and segment task state and resting state according to preset markers to construct target domain dataset; load pre-trained cross-subject domain adaptive model and fine-tune the pre-trained model using target domain dataset to classify the motor imagery intentions of new subjects.

[0017] Step 6: Convert the classification results output by the optimized cross-subject domain adaptive model into control commands.

[0018] Furthermore, in step 1, the decomposed movements of clinical stroke patients during sit-to-stand transition rehabilitation training are analyzed, and a 2D dynamic sit-to-stand motor imagery stimulation experiment paradigm is designed based on E-Prime software:

[0019] Clinically, stroke patients initially stand in a sitting position with their feet flat on the ground, hands clasped, and arms fully extended forward. They lean forward, and as their shoulders pass their knees, they immediately lift their hips, raise their head forward and upward, extend their hip and knee joints, and straighten their trunk to achieve independent standing. To more accurately simulate the actual sitting-standing rehabilitation effects of stroke patients in clinical practice, breaking down the movements helps to guide patients to complete the sitting-standing transition motor imagery task. The following is a 2D dynamic sitting-standing motor imagery stimulation experiment paradigm designed based on E-Prime software:

[0020] First, in the environmental preparation phase, subjects were placed in a quiet, semi-dark, and temperature-controlled room, 80cm away from the screen, seated in a relaxed posture, and watched a 2D sit-to-stand transition video played on the stimulus-induced screen to complete the sit-to-stand motor imagery experiment. Throughout the experiment, subjects were required to maintain complete stillness and minimal limb movement. Second, in the baseline data acquisition phase, a red "+" symbol was displayed on the screen as a cue to focus attention. Subjects kept their eyes fixed on the target point, maintaining focus, and baseline data was collected for 30 seconds. Next, the screen prompted "action execution," and subjects slowly stood up within 6 seconds, following the cue, then sat down for a 10-second rest to prepare for the motor imagery task. Then, in the motor imagery phase, a "bi" beep sound signaled the start of the experiment. The screen displayed a pseudo-random sequence of scenes from sitting to standing or sitting still. Subjects were required to simultaneously perform motor imagery for 10 seconds while watching the video, minimizing blinking. Finally, a 10-second rest was required to complete one trial. By repeating these steps, each experiment consisted of 30 trials, with a 15-minute interval between each trial, resulting in the collection of data from 5 sets of experiments.

[0021] Furthermore, in step 2, a dual-modal EEG-fNIRS signal synchronous acquisition system is constructed. This system comprises a main control computer, a multi-channel EEG acquisition device, a multi-channel functional near-infrared spectroscopy acquisition device, an integrated headgear, and supporting host computer software. The main control computer is split into two via a parallel port, simultaneously sending synchronous trigger signals to both the EEG acquisition device and the fNIRS analysis device. E-prime software is used to generate time stamps to align the timestamps of subsequently acquired EEG and fNIRS data, ensuring consistency of the dual-modal data in the temporal dimension. The subject wears the integrated EEG-fNIRS headgear, and based on the international 10-20 lead system, the crown center (Cz) and other key sites are located. These key sites include a ground (GND) electrode, a reference (REF) electrode, a rereferenced mastoid electrode, and 12 EEG signal acquisition electrodes covering the region from FC5 to CP6. By applying a conductive medium and pre-treating the scalp, the contact impedance between all EEG electrodes and the scalp is made lower than a preset threshold to ensure the quality of the EEG signal. At the same time, the hair is parted with a luminous ear pick, and 8 pairs of light electrodes (light source-detector pairs) are inserted in sequence. The motor imagery experiment is started according to the stimulation paradigm designed in step 1. Finally, the EEG and fNIRS data are saved simultaneously.

[0022] Furthermore, in step 3, the raw fNIRS data collected in step 2 is preprocessed. The method first converts the raw optical density signal into a value representing the change in oxyhemoglobin concentration based on the modified Beer-Lambert law (MBLL). Then, a first-order polynomial fitting algorithm is applied to linearly detrend the concentration signal to eliminate baseline drift caused by equipment or physiological processes. Next, to suppress artifacts introduced by subject motion, a time derivative distribution repair algorithm is used. This algorithm iteratively weights and corrects the signal's time derivative and integrates the correction results to effectively filter out motion artifacts. Finally, a third-order infinite impulse response bandpass filter is used to process the signal, limiting the signal bandwidth to between 0.01 Hz and 0.04 Hz, thereby separating specific frequency band signals related to hemodynamic response and obtaining the final preprocessed result.

[0023] In step 4, the raw EEG data acquired in step 2 is preprocessed. The method first uses a Butterworth bandpass filter (4-40Hz), a notch filter (50Hz), and a Chebyshev Type II filter for filtering. Then, independent component analysis (ICA) is used to decompose the filtered signal into multiple statistically independent components. By identifying and removing independent components corresponding to physiological artifacts such as electrooculography (EOG), the remaining pure EEG components are reconstructed to obtain a high-quality EEG signal free of specific noise and physiological artifacts.

[0024] In step 5, the cross-subject domain adaptive model is trained: First, the deep temporal features of each modality are extracted using an EEG encoder and an fNIRS encoder, respectively; second, a domain adaptation module is introduced, and through an adversarial training mechanism, the encoder is prompted to learn domain-invariant features that are irrelevant to the subject; then, through an attention-based cross-modal feature fusion module, the two features are dynamically weighted and fused to generate a unified feature representation; finally, the classification result is output through a fully connected classifier. In clinical application, first, the EEG-fNIRS synchronous data of new subjects is acquired and preprocessed, and the task state and resting state are segmented according to preset markers to construct the target domain dataset; second, the pre-trained cross-subject domain adaptive model is loaded, and the pre-trained model is fine-tuned using the target domain dataset to finally achieve the classification of the new subject's motor imagination intention.

[0025] In step 6, the new subject's intention to switch between sitting and standing is identified based on the cross-subject domain adaptive model in step 5. The classification result is converted into actual motion control commands for the sitting-standing rehabilitation robot and sent to the sitting-standing rehabilitation robot system. The motion execution drive system then drives the sitting-standing rehabilitation robot system to assist the patient in completing the sitting-standing rehabilitation training.

[0026] An application of a cross-subject domain adaptive motor imagery decoding method that integrates EEG-fNIRS is proposed. This method is applied to a rehabilitation training system to drive a sit-to-stand rehabilitation robot system to assist patients in completing sit-to-stand transition rehabilitation training.

[0027] Preferably, the rehabilitation training system is a hybrid brain-computer interface neurological injury rehabilitation training system.

[0028] The beneficial effects of this invention are mainly reflected in:

[0029] (1) Construct an EEG-fNIRS hybrid brain-computer interface, which integrates the high temporal resolution of EEG signals and the high spatial resolution of fNIRS signals, and combines the complementary characteristics of neural electrical activity and hemodynamic response to extract richer and deeper feature information, thereby improving the accuracy and robustness of motor imagery intention recognition and overcoming the technical bottleneck of insufficient feature information in a single modality.

[0030] (2) A cross-subject domain adaptive classification algorithm is proposed. By learning from the source domain of known labeled data and performing selective knowledge transfer, it learns features with greater robustness and generalization ability, thereby adapting the model to the target domain with unknown labels and achieving higher classification accuracy on new target subjects. This eliminates the need to collect large amounts of calibration data for new users, significantly reducing the time and economic costs of application deployment and enhancing the system's practicality.

[0031] (3) By integrating a cross-subject domain adaptive decoding algorithm with motor imagery-hybrid brain-computer interface technology, a "thought-driven" active closed-loop sit-to-stand rehabilitation training system was constructed. The recognized motor intentions were converted into control commands to drive the sit-to-stand rehabilitation system to assist patients in actively completing sit-to-stand transition rehabilitation training, which changed the passive mode of traditional rehabilitation that relied on external assistance from physiotherapists or preset programs on equipment. Attached Figure Description

[0032] Figure 1 This is a schematic diagram of a sitting-standing rehabilitation training system driven by a motor imagery-hybrid brain-computer interface.

[0033] Figure 2 A time series diagram of a single trial for a motion imagination paradigm;

[0034] Figure 3 This is a schematic diagram showing the distribution of EEG signal acquisition electrodes and fNIRS signal acquisition optical electrodes;

[0035] Figure 4 A schematic diagram of the cross-subject domain adaptive model architecture;

[0036] Figure 5This is a schematic diagram of the sitting-to-standing rehabilitation robot system, which includes: 1. aluminum profile support, 2. suspension protection device, 3. electrical control cabinet, 4. cable reel, 5. coaxial double pulleys, 6. traction rope, and 7. handrail. Detailed Implementation

[0037] The present invention will now be further described with reference to the accompanying drawings.

[0038] Reference Figures 1-5 This invention discloses a cross-subject domain adaptive motor imagery decoding method integrating EEG-fNIRS. As a general motor imagery-brain-computer interface decoding technology, its application is not limited to sitting-standing rehabilitation. This method can also be applied to other motor rehabilitation scenarios, such as, but not limited to, supine rehabilitation robots, lower limb gait rehabilitation robots, and upper limb rehabilitation robots. Applying the decoding algorithm of this invention to these devices falls within the scope of protection of this invention. This invention describes the decoding method and the hybrid brain-computer interface neural injury rehabilitation training system using a sitting-standing rehabilitation training system as an example.

[0039] A cross-subject domain adaptive motion imagery decoding method incorporating EEG-fNIRS includes the following steps:

[0040] Step 1: Analyze the decomposed movements of clinical stroke patients during sit-to-stand transition rehabilitation training, and design a 2D dynamic sit-to-stand motor imagery stimulation experiment paradigm based on E-Prime software.

[0041] To more accurately simulate the actual sitting-standing rehabilitation effect of stroke patients in clinical practice, this experiment adopted a customized autonomous standing stimulation paradigm. By breaking down the movements, it is easier to induce patients to complete the sitting-standing transition motor imagery task. The specific sitting-standing transition process is as follows: The subject's initial state is a sitting posture with both feet flat on the ground, hands interlaced, both upper limbs fully extended forward, and the body leaning forward. When the shoulders are forward past the knees, the subject immediately lifts the hips, raises the head forward and upward, extends the hip and knee joints, and stands up with the trunk straight.

[0042] Reference Figure 2 Preparation phase (-30s-0s): Subjects are in a quiet, semi-dark, and temperature-controlled room, 80cm away from the screen, sitting in a relaxed posture in a chair. They watch a 2D sit-to-stand transition video played on the stimulus-induced screen to complete the sit-to-stand motion imagery experiment. Throughout the experiment, no significant limb movement is required. When a red "+" symbol appears on the screen (as a cue for focused attention), the subject's eyes remain fixed on the target point, maintaining focus. Baseline data is collected for 30 seconds.

[0043] Motor execution phase (0s-6s): When the word "motion execution" appears on the screen, the subject slowly stands up for 6 seconds following the prompts. This action is a breakdown of the movements performed by clinical stroke patients during sit-to-stand rehabilitation training.

[0044] Rest phase (6s-16s): Sit down and rest for 10 seconds to prepare for the motor imagery task, with the aim of returning to baseline level;

[0045] Motor imagery phase (16s-26s): The motor imagery experiment officially begins after hearing the "bi" beep. The screen presents a pseudo-random sequence of action scenes from sitting to standing or static sitting scenes. Subjects are asked to think while watching, minimize blinking, and continue to imagine for 10 seconds.

[0046] Rest phase (26s-36s): Rest for 10 seconds to complete one trial;

[0047] Each subject participated in 5 sets of experiments (with a 15-minute interval between each set), and each set of experiments contained 30 trials.

[0048] Step 2: Build a dual-modal EEG-fNIRS signal synchronous acquisition system, locate the brain motor-sensory cortex area, determine the placement of light poles and electrodes, and synchronously acquire the raw EEG and fNIRS data of the subject under the motor imagery task.

[0049] The architecture of the dual-modal EEG-fNIRS signal synchronous acquisition system mainly consists of a PC host, a multi-channel EEG acquisition subsystem, a multi-channel functional near-infrared spectroscopy acquisition subsystem, an integrated headgear, and supporting host computer software. In this embodiment, the Okti 128-channel portable EEG system manufactured by Compumedics, Australia, is used. This subsystem includes EEG electrodes, signal amplifiers, and supporting data acquisition host computer software (e.g., CURRY data acquisition software); a portable near-infrared optical signal analysis system is also used, such as the LIGHTNIRS system manufactured by Shimadzu Corporation, Japan. This subsystem includes a light source emitter, a photosensitive detector, and supporting data acquisition host computer software (e.g., kNIRS data acquisition software).

[0050] Before signal acquisition, the following EEG signal acquisition electrodes and fNIRS signal acquisition optical electrodes need to be deployed, referring to... Figure 3The subject was fitted with an integrated EEG-fNIRS headgear, and the Cz position was positioned at the center of the top of the head according to the international 10-20 system. Conductive adhesive was applied to the 12 electrode positions in sequence, including ground (GND), reference electrode (REF), left and right mastoids (as secondary references), and FC5 to CP6. The scalp was rubbed from side to side with a flat-tipped needle to remove dead skin cells and oil, further improving electrode contact. The impedance of all electrodes was checked in real time to ensure that it was below the 20kΩ threshold and to guarantee the quality of the EEG signal. The hair was parted with a light-emitting ear pick, and 8 pairs of light electrodes (light source-detector pairs) were inserted in sequence.

[0051] To ensure precise temporal alignment of EEG and fNIRS data, a hardware-triggered connection is established between the main control computer and the EEG and fNIRS acquisition subsystems via a parallel port. In the software, E-prime stimulation software is used to write inline code. At the start of the motor imagery task in the experimental paradigm, the main control computer simultaneously sends synchronization markers to both subsystems via the parallel port. These markers are used to align the timestamps of the EEG and fNIRS signals, ensuring temporal consistency between the two modalities.

[0052] While the subjects performed the motor imagery task paradigm designed in step 1, two subsystems simultaneously acquired data: the fNIRS light source emitter emitted near-infrared light, penetrating the scalp and skull to a depth of 1-2.5 cm in the cerebral cortex, and the receiving probe received the reflected near-infrared light. The fNIRS data acquisition host computer software calculated and recorded the concentration changes of oxyhemoglobin, deoxyhemoglobin, and total hemoglobin in real time based on the attenuation of light intensity (following Beer-Lambert's law). These changes reflected the local hemodynamic response induced by neural activity. The EEG system captured the potential signals on the scalp through electrodes, amplified and converted the raw EEG signals to digital, and then transmitted them wirelessly to the PC for recording by the EEG data acquisition host computer software. Finally, the EEG and fNIRS data were saved simultaneously.

[0053] Step 3: fNIRS data preprocessing. The original optical amplitude is converted into oxyhemoglobin concentration change values ​​using a modified Beer-Lambert law; a first-order polynomial fitting algorithm is used for linear detrending, and a time derivative distribution repair method is used for motion artifact correction; a third-order infinite impulse response filter is used to bandpass filter the data in the range of 0.01Hz and 0.04Hz.

[0054] First, the acquired raw optical amplitude signal is converted into concentration changes of oxyhemoglobin, deoxyhemoglobin, and total hemoglobin according to the modified Beer-Lambert law:

[0055]

[0056] Where Δ represents the relative change at a given moment relative to the initial moment; α represents the molar extinction coefficient of a certain hemoglobin (oxy or deoxy) for a certain wavelength of light (λ1 or λ2); A represents the light density (λ1 or λ2) at a certain wavelength detected by the fNIRS system; and B represents the optical path length, which is related to the distance between the emitter and receiver (generally 3 cm apart, a pre-determined constant).

[0057] Secondly, a first-order polynomial fitting algorithm is used for linear detrending. For an original signal y... original (t), fitting a first-order polynomial y trend (t) represents the trend of the signal:

[0058] y trend (t) = at + b;

[0059] Where a is the slope of the line, b is the intercept, and t is time. Parameters a and b are estimated using the least squares method to minimize the sum of squared errors between the original signal and the fitted line.

[0060] Detrended signal y trend (t) is obtained by subtracting the fitted trend line from the original signal:

[0061] y detrend (t)=y original (t)-y trend (t)=y original (t)-(at+b);

[0062] Next, motion artifact correction is performed using the time derivative distribution repair method, the process of which is as follows:

[0063] Calculate the time derivative y(t) of the signal x(t):

[0064] y(t) = x(t) - x(t-1);

[0065] The weights of each time derivative are calculated through an iterative process. In each iteration, the weighted mean μ and the robust standard deviation σ of the residuals are calculated. The scaling bias d is then calculated based on the residual r(t) = |y(t) - μ| and the robust standard deviation σ at each point. t :

[0066]

[0067] Where c is the tuning constant.

[0068] Update the weights ω using Tukey's biweight function. t :

[0069]

[0070] After iterative convergence, the final weight ω is used. t Corrected time derivative:

[0071] y' corrected (t)=ω t ·(y(t)-μ);

[0072] By integrating the corrected time derivative, the motion artifact-free signal x' can be reconstructed. corrected (t);

[0073] Bandpass filtering of data in the range of 0.01Hz and 0.04Hz is performed using a third-order infinite impulse response filter:

[0074]

[0075] When expanded, it appears as follows:

[0076] y[n]=b0x[n]+b1x[n-1]+b2x[n-2]+b3x[n-3]-(a1y[n-1]+a2y[n-2]+a3y[n-3]);

[0077] Where x[n] is the fNIRS signal after the aforementioned steps; y[n] is the filtered output signal; b k and a k These are the coefficients of the filter.

[0078] Step 4: EEG data preprocessing. A Butterworth bandpass filter (4-40Hz), a notch filter (50Hz), and a Chebyshev Type II filter are used for filtering; independent component analysis is employed to remove physiological artifacts such as those related to electrooculography.

[0079] This method extracts effective neural activity signals from raw multichannel EEG data while maximally suppressing noise and physiological artifacts. Specifically, it includes the following steps:

[0080] First, a Butterworth bandpass filter is used to extract the target signal in the 4-40Hz frequency band. The squared gain of an nth-order Butterworth low-pass filter is |G(ω)|. 2 Defined as:

[0081]

[0082] Where ω is the signal angular frequency, ω cLet ω be the cutoff angular frequency and n be the filter order. Through frequency transformation, the low-pass filter prototype is converted into a band-pass filter with a center frequency of ω0 and a bandwidth of BW. Its transfer function preserves signals in the 4-40Hz range while attenuating low-frequency and high-frequency noise outside this range.

[0083] Secondly, a 50Hz notch filter is used to eliminate 50Hz power frequency interference. Its transfer function H(z) is expressed in the digital domain as:

[0084]

[0085] in, It is the normalized angular frequency corresponding to the notch filter frequency, f notch =50Hz, f s The sampling frequency is r, which determines the bandwidth of the notch filter.

[0086] Next, a Chebyshev Type II filter is used to further sharpen the transition between the passband and stopband, while maintaining ripple-free passband. Its gain square |G(ω)| 2 Defined as:

[0087]

[0088] Among them, T n (x) is an nth-order Chebyshev polynomial of the first kind, ω s denoted as the stopband start angular frequency, and ∈ represents a parameter related to stopband attenuation.

[0089] Finally, independent component analysis was used to separate and remove physiological artifacts such as electrooculography.

[0090] Suppose that the observed multichannel EEG signal x is a linear mixture of multiple statistically independent source signals s (including real EEG signals and noise sources such as electrooculography) through an unknown mixing matrix A. Its mathematical model is as follows:

[0091] x = As;

[0092] Where x = [x1(t), x2(t), ..., x m (t)] T It is the vector of observed signals recorded by m electrode channels, s=[s1(t),s2(t),...,s n (t)] T There are n independent source signal vectors. The goal of ICA is to find a separation matrix W such that the output signal u:

[0093] u = Wx;

[0094] Here, u is the estimate of the source signal s, and W is solved by maximizing the non-Gaussianity of the output component. After obtaining the separation matrix W, the independent components representing EOG interference can be identified and removed from the mixed signal. Then, the remaining pure EEG signal is reconstructed, thereby achieving the purpose of removing EOG interference.

[0095] Step 5: Obtain EEG-fNIRS data of new subjects and segment task state and resting state according to preset markers to construct target domain dataset; load pre-trained cross-subject domain adaptive model and fine-tune the pre-trained model using target domain dataset to classify the motor imagery intentions of new subjects.

[0096] Reference Figure 4 The cross-subject domain adaptive model proposed in this invention mainly comprises four parts: an encoder, a domain adaptation module, a cross-modal fusion module, and a classifier. The encoder module employs a dual-branch structure, with separate EEG encoders and fNIRS encoders for parallel extraction of feature information from both modalities. The processing flow within each encoder is as follows: First, the original input features are projected to a high-dimensional feature space through a linear projection layer to capture more complex feature associations and unify the feature dimensions of different modalities. The calculation formula is as follows:

[0097] Y = XW T +b;

[0098] Where X is the input feature matrix, W is the weight matrix, b is the bias vector, and Y is the projected high-dimensional feature matrix.

[0099] Secondly, high-dimensional features are processed using grouped convolutions with a kernel size of k=3 and a stride of 1. Assume the number of channels in the input feature map is C. in The number of output channels is C out The number of groups is g, and the number of parameters for grouped convolution is The output of the convolutional layer is normalized and a non-linear ReLU activation function is introduced to accelerate model convergence, prevent gradient vanishing or exploding, and enhance model stability. Next, pointwise convolutions with a kernel size of 1 and a stride of 1 are used to fuse features extracted independently from previous group convolutions across channels, generating new feature representations. Then, during training, some elements of the current layer's output are randomly set to zero with a probability of p=0.1 as a regularization measure to prevent overfitting. Finally, an adaptive average pooling layer is used to convert feature sequences of arbitrary length into fixed-dimensional feature vectors, calculated as follows:

[0100]

[0101] Among them, z cIt is the c-th element of the output vector, x c,i,j It is the value of the c-th input channel feature map at position (i,j).

[0102] Domain Adaptation Module: First, the domain classifier G d A two-layer fully connected neural network is used to distinguish the input feature vector f = G. f (x) Whether it originates from the EEG domain or the fNIRS domain, to achieve the learning of domain-invariant features, a gradient reversal layer (GRL) is introduced. This layer connects the feature encoder and the domain classifier G. d During the forward propagation of the network, GRL inverts the gradient from the loss of the domain classifier, and its calculation formula is expressed as:

[0103] GRL(f) = f.

[0104] During backpropagation, this layer multiplies the gradient generated by the subsequent loss by a negative constant. λ Then pass it to the preceding network layer:

[0105]

[0106] in, For the loss of the domain classifier, θ f For feature extractor G f The parameters are set. While minimizing the neighborhood classifier loss, the feature extractor G is updated. f The parameters are set to maximize them. The direction optimization, after multiple rounds of adversarial training, forces the encoder to learn domain-independent general features shared by the EEG and fNIRS domains.

[0107] Secondly, given the significant differences in statistical distribution between EEG and fNIRS data, an independent batch normalization layer is configured for each domain, for each batch from the EEG domain. and batches from the fNIRS domain Their normalization operations are defined as follows:

[0108]

[0109] Where μ and σ 2 These are the mean and variance of the batch data, respectively, and γ. S ,β S and γ T ,β T These are the scaling and translation parameters for learning in the EEG and fNIRS domains, respectively.

[0110] Cross-modal fusion module: First, an attention-weighted feature fusion strategy is used to fuse feature vectors (f) from different modalities. eeg and f fnirs ) concatenate them into a combined feature vector (f) concat =[f eeg ;f fnirs The input is fed into a fully connected layer to learn the complex interaction between features of two modalities, generating an original "importance" score s for each modality. Next, these scores are normalized to a set of attention weights α that sum to 1 using a Softmax function. Finally, the original modal features are weighted and summed using these attention weights to dynamically generate the fused feature representation f. attention_fused The modalities that contribute more will be given higher weights.

[0111] s = tanh(W att ·f concat +b att );

[0112] Among them, W att and b att Here are the weights and biases of the fully connected layer, and tanh is a commonly used activation function.

[0113] f attention_fused =α a ·f a +α b ·f b ;

[0114] Based on this, a feature transformation-based fusion strategy is adopted to combine feature vectors (f) from different modalities. eeg and f fnirs ) directly concatenate them into a longer combined vector (f) concat =[f eeg ;f fnirs The input is fed into a neural network consisting of fully connected layers, the nonlinear activation function ReLU, and Dropout layers. The transformation process can be represented as follows:

[0115] f transform_fused =Dropout(ReLU(W trans ·f concat +b trans ));

[0116] Among them, W trans and b trans These are the weight matrix and bias term of the fully connected layer, respectively. By learning the complex interactions and correlations between modalities, high-dimensional combined information is non-linearly mapped and refined into a new, lower-dimensional unified feature space, ultimately generating a new, more information-dense fusion feature representation f.transform_fused .

[0117] The classifier part works as follows: First, a fully connected layer compresses the high-dimensional fused features into a more compact representation space; second, a ReLU activation function is used to introduce non-linearity; then, a Dropout layer is used to randomly deactivate the features to prevent overfitting; finally, the output fully connected layer maps the intermediate features to two original scores representing different categories, thus completing the binary classification task.

[0118] This invention employs the PyTorch deep learning framework to construct and train an adaptive model across subject domains. During training, the AdamW optimizer is used to iteratively optimize the model's network parameters, and the overall loss function, a weighted combination of classification loss, domain adversarial loss, and feature consistency loss, is jointly optimized to improve the model's generalization ability in the target domain.

[0119] In clinical applications, firstly, based on the EEG-fNIRS signal synchronous acquisition system described in step 2, EEG-fNIRS data of new subjects performing motor imagery tasks are acquired. Secondly, according to the methods described in steps 3 and 4, the original fNIRS and EEG signals are preprocessed. Based on the event tags output by the E-prime software, the preprocessed EEG and fNIRS data are segmented into task-state and resting-state states. For fNIRS data, the time period from 2 seconds before the stimulus begins to 15 seconds after the stimulus ends in each trial is extracted to form single-trial fNIRS data. For EEG data, the time period from 200 milliseconds before the stimulus begins to 800 milliseconds after the stimulus ends in each trial is extracted to form single-trial EEG data. Baseline correction is performed on all extracted single-trial data to form target domain data, which is then input into a pre-trained cross-subject domain adaptive model. Finally, the pre-trained model is fine-tuned to classify the motor imagery intentions of new subjects.

[0120] Step 6: Convert the classification results output by the optimized cross-subject domain adaptive model into control commands to drive the sitting-to-standing rehabilitation training robot system to assist the patient in completing sitting-to-standing rehabilitation training.

[0121] The optimized cross-subject domain adaptive model from step 5 is used to identify the sitting-to-standing transition intention of new subjects. The classification results are then converted into actual motion control commands for the sitting-to-standing rehabilitation robot and sent to the robot system for reference. Figure 5 The motor in the electrical control cabinet 3 drives the movement of the two ropes 6, which in turn drives the movement of one rope through the take-up reel 4. This causes the two ropes to move through the coaxial double pulleys 5. The subject holds onto the handle 7, and with the support of the suspension protection device 2, the subject is pulled to complete the sitting-to-standing rehabilitation training.

[0122] An application of a cross-subject domain adaptive motor imagery decoding method that integrates EEG-fNIRS is proposed. This method is applied to a rehabilitation training system to drive a sit-to-stand rehabilitation robot system to assist patients in completing sit-to-stand transition rehabilitation training.

[0123] Reference Figure 1 A hybrid brain-computer interface rehabilitation training system for neural injuries, integrating EEG-fNIRS with a cross-subject domain adaptive motor imagery decoding method, is described. The system includes an aluminum profile support 1, a suspension protection device 2, an electrical control cabinet 3, a cable reel 4, coaxial double pulleys 5, traction ropes 6, and handrails 7. During the sitting-to-standing transition, the decoded motor intention is converted into a motor control signal. The motor in the electrical control cabinet 3 drives the movement of the two ropes 6, which in turn drives the movement of one rope via the cable reel 4. This movement then drives the two ropes passing through the coaxial double pulleys 5. The subject holds onto the handrails 7, and the system, based on the suspension protection device 2, guides the subject to complete the target movement.

[0124] The embodiments described in this specification are merely examples of implementations of the inventive concept and are for illustrative purposes only. The scope of protection of this invention should not be considered limited to the specific forms described in these embodiments; rather, it extends to equivalent technical means conceived by those skilled in the art based on the inventive concept.

Claims

1. A cross-subject domain adaptive motion imagery decoding method integrating EEG-fNIRS, characterized in that, The method includes the following steps: Step 1: Analyze the decomposed movements of clinical stroke patients during sit-to-stand transition rehabilitation training, and design a 2D dynamic sit-to-stand motor imagery stimulation experiment paradigm based on E-Prime software. Step 2: Build a dual-modal EEG-fNIRS signal synchronous acquisition system, locate the brain motor-sensory cortex area, determine the placement of light poles and electrodes, and synchronously acquire the raw EEG and fNIRS data of the subject under the motor imagery task. Step 3, fNIRS data preprocessing: The original optical amplitude is converted into the oxyhemoglobin concentration change value using the modified Beer-Lambert law; a first-order polynomial fitting algorithm is used for linear detrending, and a time derivative distribution repair method is used for motion artifact correction; a third-order infinite impulse response filter is used to bandpass filter the data in the range of 0.01Hz and 0.04Hz. Step 4, EEG data preprocessing: Filtering is performed using a Butterworth bandpass filter (4-40Hz), a notch filter (50Hz), and a Chebyshev Type II filter; Independent component analysis is used to remove electrooculogenic artifacts. Step 5: Obtain EEG-fNIRS data of new subjects and segment task state and resting state according to preset markers to construct target domain dataset; load pre-trained cross-subject domain adaptive model and fine-tune the pre-trained model using target domain dataset to classify the motor imagery intentions of new subjects. Step 6: Convert the classification results output by the optimized cross-subject domain adaptive model into control commands to drive the sit-to-stand rehabilitation robot system to assist the patient in completing sit-to-stand transition rehabilitation training.

2. The cross-subject domain adaptive motion imagery decoding method integrating EEG-fNIRS as described in claim 1, characterized in that, In step 1, the initial position for stroke patients in clinical practice is a sitting posture with both feet flat on the ground, hands clasped together, arms fully extended forward, and body leaning forward. When the shoulders are forward past the knees, the patient immediately lifts their hips, raises their head forward and upward, extends their hip and knee joints, and straightens their trunk to complete the independent standing. To more accurately simulate the actual sitting-standing rehabilitation effect of stroke patients in clinical practice, breaking down the movements helps to induce patients to complete the sitting-standing transition motor imagery task. The 2D dynamic sitting-standing motor imagery stimulation experiment paradigm designed based on E-Prime software is as follows: First, in the environmental preparation phase, subjects were placed in a quiet, semi-dark, and temperature-controlled room, 80cm away from the screen, seated in a relaxed posture, and watched a 2D sit-to-stand transition video played on the stimulus-induced screen to complete the sit-to-stand motor imagery experiment. Throughout the experiment, subjects were required to maintain complete stillness and minimal limb movement. Second, in the baseline data acquisition phase, a red "+" symbol was displayed on the screen as a cue to focus attention. Subjects kept their eyes fixed on the target point, maintaining focus, and baseline data was collected for 30 seconds. Next, the screen prompted "action execution," and subjects slowly stood up within 6 seconds, following the cue, then sat down for a 10-second rest to prepare for the motor imagery task. Then, in the motor imagery phase, a "bi" beep sound signaled the start of the experiment. The screen displayed a pseudo-random sequence of scenes from sitting to standing or sitting still. Subjects were required to simultaneously perform motor imagery for 10 seconds while watching, minimizing blinking. Finally, a 10-second rest was required to complete one trial. By repeating these steps, each experiment consisted of 30 trials, with a 15-minute interval between each trial, resulting in the collection of data from 5 sets of experiments.

3. The cross-subject domain adaptive motion imagery decoding method integrating EEG-fNIRS as described in claim 1 or 2, characterized in that, In step 2, a dual-modal EEG-fNIRS signal synchronous acquisition system is built. The system consists of a main control computer, a multi-channel EEG acquisition device, a multi-channel functional near-infrared spectroscopy acquisition device, an integrated headgear, and supporting host computer software. The main control computer is split into two via a parallel port, and synchronous trigger signals are sent to the EEG acquisition device and the fNIRS analysis device simultaneously. E-prime software was used to generate time stamps to align the timestamps of subsequently acquired EEG and fNIRS data, ensuring consistency of the bimodal data in the temporal dimension. Subjects wore an integrated EEG-fNIRS headgear, and the center Cz of the top of the head and other key sites were located according to the international 10-20 lead system. These key sites included a grounding GND electrode, a reference REF electrode, a secondary reference mastoid electrode, and 12 EEG signal acquisition electrodes covering the area from FC5 to CP6. A conductive medium was applied and the scalp was pretreated to ensure that the contact impedance between all EEG electrodes and the scalp was below a preset threshold, guaranteeing the quality of the EEG signal. Simultaneously, a luminous ear pick was used to part the hair, and eight pairs of light electrodes were inserted sequentially. A motor imagery experiment was then conducted according to the stimulation paradigm designed in step 1. Finally, the EEG and fNIRS data were simultaneously saved.

4. The cross-subject domain adaptive motion imagery decoding method integrating EEG-fNIRS as described in claim 1 or 2, characterized in that, In step 3, the raw fNIRS data acquired in step 2 is preprocessed. First, based on a modified Beer-Lambert law, the raw optical density signal is converted into a value representing the change in oxyhemoglobin concentration. Then, a first-order polynomial fitting algorithm is applied to linearly detrend the concentration signal to eliminate baseline drift caused by equipment or physiological processes. Next, to suppress artifacts introduced by subject motion, a time derivative distribution repair algorithm is used. This algorithm iteratively weights and corrects the signal's time derivative and integrates the correction results to effectively filter out motion artifacts. Finally, a third-order infinite impulse response bandpass filter is used to process the signal, limiting the signal bandwidth to between 0.01 Hz and 0.04 Hz, thereby separating specific frequency band signals related to hemodynamic response and obtaining the final preprocessed result.

5. The cross-subject domain adaptive motion imagery decoding method integrating EEG-fNIRS as described in claim 1 or 2, characterized in that, In step 4, the raw EEG data collected in step 2 is preprocessed. The method first uses a Butterworth bandpass filter of 4-40Hz, a notch filter of 50Hz, and a Chebyshev Type II filter for filtering. Subsequently, independent component analysis was used to decompose the filtered signal into multiple statistically independent components. By identifying and removing independent components corresponding to physiological artifacts such as electrooculography (EOG), the remaining pure EEG components were reconstructed to obtain a high-quality EEG signal free of specific noise and physiological artifacts.

6. The cross-subject domain adaptive motion imagery decoding method integrating EEG-fNIRS as described in claim 1 or 2, characterized in that, In step 5, the cross-subject domain adaptive model is trained: First, the deep temporal features of each modality are extracted using an EEG encoder and an fNIRS encoder, respectively; second, a domain adaptation module is introduced, and through an adversarial training mechanism, the encoder is prompted to learn domain-invariant features that are irrelevant to the subject; then, through an attention-based cross-modal feature fusion module, the two features are dynamically weighted and fused to generate a unified feature representation; finally, the classification result is output through a fully connected classifier. In clinical application, first, the EEG-fNIRS synchronous data of new subjects is acquired and preprocessed, and the task state and resting state are segmented according to preset markers to construct the target domain dataset; second, the pre-trained cross-subject domain adaptive model is loaded, and the pre-trained model is fine-tuned using the target domain dataset to finally achieve the classification of the new subject's motor imagination intention.

7. The cross-subject domain adaptive motion imagery decoding method integrating EEG-fNIRS as described in claim 1 or 2, characterized in that, In step 6, the new subject's intention to switch between sitting and standing is identified based on the cross-subject domain adaptive model in step 5. The classification result is converted into actual motion control commands for the sitting-standing rehabilitation robot and sent to the sitting-standing rehabilitation robot system. The motion execution drive system then drives the sitting-standing rehabilitation robot system to assist the patient in completing the sitting-standing rehabilitation training.

8. An application of the cross-subject domain adaptive motion imagery decoding method fused with EEG-fNIRS as described in claim 1, characterized in that, This method is applied to a rehabilitation training system, driving a sitting-to-standing rehabilitation robot system to assist patients in completing sitting-to-standing rehabilitation training.

9. The application as described in claim 8, characterized in that, The rehabilitation training system is a hybrid brain-computer interface neurological injury rehabilitation training system.

Images