A method for motor image recognition combining EEG and functional near-infrared spectroscopy

Abstract

This invention relates to a method for motor imagery recognition that combines electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS). It simultaneously acquires EEG and fNIRS data during a motor imagery task using a multi-channel acquisition system, and preprocesses and augments the acquired data from both modalities. Based on the characteristics of the two signals, time-frequency and spatial features of EEG and spatial features of fNIRS are extracted respectively. A masked autoencoder is designed for each modality feature and pre-trained. Finally, the parameters obtained from the pre-training are combined to perform feature fusion through a cross-modal cross-attention mechanism, and training is performed using cross-entropy loss. This invention improves the accuracy of motor imagery recognition by effectively combining the independent and complementary information from EEG and fNIRS.

Description

Technical Field

[0001] This invention relates to a method for recognizing motor imagery by combining electroencephalography (EEG) and functional near-infrared spectroscopy, belonging to the field of signal processing technology. Background Technology

[0002] Motor imagery is a paradigm based on decoding brain neural activity, originating from the exploration of the neural mechanisms of the motor cortex. When an individual imagines performing a specific limb movement, even without actual action, neuronal groups in the brain's sensorimotor area are still activated. Brain-computer interface (BCI) systems capture and classify these characteristic signals, translating them into control commands for external devices. The advantage of motor imagery is that it allows for human-computer interaction directly through psychological training without external stimulation or actual movement, thus attracting significant attention in fields such as neurorehabilitation and assistive technology.

[0003] Translating the laboratory-validated motor imagery-based brainstem in-person imaging (BCI) paradigm into clinical practice remains a critical challenge, fraught with challenges such as neural signal stability, individual adaptability, and environmental resistance. To address these issues, the selection of real-time neuroimaging technologies requires finding an optimal balance between temporal resolution, spatial accuracy, portability, and operational cost. Compared to technologies like functional magnetic resonance imaging (fMRI) and magnetoencephalography (MEG), which are limited by high equipment costs and bulky size, and positron emission tomography (PET), which involves invasive procedures and low temporal resolution, electroencephalography (EEG) and functional near-infrared spectroscopy (fNTIS) stand out due to their unique advantages. EEG captures rhythmic oscillations induced by motor imagery with millisecond-level temporal resolution, while fNTIS provides centimeter-level spatial localization through hemodynamic responses; both can be acquired synchronously using wearable devices. This complementary combination of spatiotemporal characteristics not only meets the clinical demands for real-time performance, portability, and cost control but also significantly enhances the robustness and reliability of motion decoding through deep fusion of multimodal data.

[0004] Although EEG-fNIRS multimodal fusion has improved the performance of motion imagery BCI, existing algorithms still have many shortcomings in handling heterogeneous data fusion and individual differences. Therefore, how to more efficiently mine and utilize the heterogeneous information of EEG and fNIRS signals remains a key issue. Summary of the Invention

[0005] To overcome the shortcomings of existing technologies, this invention provides a method for motor image recognition that combines EEG and functional near-infrared spectroscopy, making full use of the multi-domain features of EEG and fNIRS to improve the accuracy of motor image recognition and effectively combine independent and complementary information from EEG and fNIRS.

[0006] A method for recognizing motor imagery combining electroencephalography (EEG) and functional near-infrared spectroscopy includes the following steps:

[0007] Step 1: Set up the experiment according to the motor imagery experiment paradigm, and simultaneously collect the EEG data and functional near-infrared spectroscopy (fNIRS) data of the subjects under the motor imagery task. Preprocess the collected synchronous data and perform data augmentation.

[0008] The experiment's specific procedure included: a 2-second instruction phase, a 10-second task execution phase, and a 15-17 second rest phase, with a 60-second buffer phase between each experiment. During the task execution phase, participants completed a motor imagery task involving the imagination of their "left hand" and "right hand." The collected data will be precisely labeled according to the timeline of the experimental paradigm, and will undergo preprocessing and data augmentation.

[0009] Step 2: Extract the time-frequency and spatial features of EEG and the spatial features of fNIRS. Pre-train the above features using a deep mask autoencoder to extract the low-dimensional representation of the features and obtain the training weights of the encoder.

[0010] Step 3: Using the pre-trained encoder as the initial weights, the time-frequency and spatial features of EEG and the spatial features of fNIRS are fused through a cross-modal cross-attention mechanism, and the fusion result is classified using cross-entropy loss.

[0011] Step one specifically includes:

[0012] S1.1: For EEG data, first extract the resting state data for the first 3 seconds of the task to perform baseline correction and eliminate individual resting potential differences; then suppress low-frequency drift and high-frequency electromyography interference through a bandpass filter, and combine independent component analysis to separate and remove physiological artifacts such as blinking and head movement.

[0013] For fNIRS data, given the 6-8 second delay in its blood oxygen dynamics response, a sliding time window of 10 seconds with a step size of 1 second, matching the task execution time, was used for analysis. The modified Beer-Lambert law was used to convert the raw optical density data into a quantitative index reflecting the changes in oxyhemoglobin concentration and deoxyhemoglobin concentration, followed by baseline correction and bandpass filtering.

[0014] S1.2: Take the data preprocessed in S1.1 and use a sliding time window of size 3s and step size 1s to align the preprocessed EEG and fNIRS data. Implement a data augmentation strategy based on Gaussian noise injection for the segmented data and apply normal distribution noise to expand the training samples.

[0015] Step two specifically includes:

[0016] S2.1: For EEG signals, Morlet wavelet transform is used to extract time-frequency energy features, and a time-frequency energy spectrum matrix is ​​constructed using continuous wavelet coefficients; cross-channel phase synchronization is calculated based on the weighted phase lag exponent, and the information coupling characteristics of functional brain regions such as the motor cortex are quantified. For fNIRS signals, the high spatial resolution advantage is focused on, and an inter-channel functional connectivity matrix is ​​constructed using the Pearson correlation coefficient.

[0017] S2.2: For the above three features, reconstruction is performed using a sub-mask autoencoder. The calculation process is as follows:

[0018] First, the EEG time-frequency features, EEG spatial features, and fNIRS spatial features are divided into blocks, and the feature matrix of each mode is further divided into... Where N i The total number of blocks for the i-th mode; then a dynamic random mask matrix is ​​constructed. Partial feature blocks are randomly masked with probability p = 0.4 to form partial observation input X. i ⊙M i In the encoding stage, a self-attention-enhanced feedforward network is used to extract the latent representation Z. i The system captures global dependencies across feature blocks through a multi-head attention mechanism; the decoding blocks then reconstruct the original features through inverse block operations and the multi-head attention mechanism. Loss function used:

[0019]

[0020] Where K i =∑ j M ij , where m is the number of input modes.

[0021] Step three specifically includes:

[0022] S3.1: Load the submask autoencoder parameters trained in step two, initialize the encoder module of the fusion classification model, and retain the noise-resistant feature extraction capability learned in the pre-training stage; transfer the EEG time-frequency features... EEG airspace characteristics fNIRS Spatial Features Each input is fed into the corresponding encoder to generate the corresponding low-dimensional representation Z. i ;

[0023] S3.2: Multimodal features are interactively fused through a cross-modal cross-attention mechanism for heterogeneous data. This includes intra-modal cross-attention and cross-modal cross-attention mechanisms. In the intra-modal attention mechanism for EEG data, time-frequency features are integrated... As a query, spatial features As a key / value pair, the process can be represented as follows:

[0024]

[0025] Among them W q t W k ts and W v ts The learnable parameter matrix is ​​σ, which represents layer normalization. EEG time-frequency features are fused with the functional connectivity network via a cross-attention mechanism. The calculation process is as follows:

[0026]

[0027] Where f(n) and g(n) represent the mapping functions used for dimension mapping, respectively, in the cross-modal cross-attention mechanism, using EEG spatial features. For the query, fNIRS spatial features For key / value pairs, compute cross-modal attention dominated by EEG spatial features; using fNIRS Spatial domain features are Query and EEG spatial domain features. For Key / Value pairs, compute cross-modal attention dominated by fNIRS spatial features.

[0028] Based on the fused features obtained in step three, the final predicted probabilities are output through a fully connected layer and a softmax layer. The expression for the loss function derived from the predicted probabilities is as follows:

[0029]

[0030] Where L is the loss function and C is the number of label categories. This represents the probability value predicted by the network that the sample belongs to the i-th category label.

[0031] Compared with existing EEG-fNIRS multimodal fusion methods, this invention has the following advantages:

[0032] To address the issue of insufficient feature fusion in existing EEG-fNIRS integration methods, this invention develops a deep fusion framework based on EEG-fNIRS. This framework fully leverages the high temporal resolution of EEG and the high spatial resolution of fNIRS. Through a cross-modal cross-attention mechanism, the fusion of EEG's temporal and spatial features yields a more comprehensive multi-domain representation. Furthermore, the interaction and fusion of EEG and fNIRS spatial features overcomes the limitation of EEG's spatial resolution, while simultaneously capturing the bidirectional coupling mechanism between EEG and fNIRS, achieving efficient fusion of the two modalities. Additionally, pre-training with a masked autoencoder accelerates model training, significantly improving the model's generalization performance and achieving more accurate classification and recognition. Attached Figure Description

[0033] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0034] Figure 1 This is a flowchart of the motion image recognition method combining EEG and functional near-infrared spectroscopy according to the present invention.

[0035] Figure 2 This is a flowchart of the data augmentation process of the present invention.

[0036] Figure 3 This is a diagram of the deep fusion network structure combining EEG and functional near-infrared spectroscopy proposed in this invention.

[0037] Figure 4 This refers to the feature fusion mode involved in this invention.

[0038] Figure 5 This is a structural diagram of the cross-modal cross-attention mechanism involved in this invention. Detailed Implementation

[0039] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0040] This embodiment, in conjunction with the accompanying drawings, provides a detailed description of a method for recognizing motor imagery by combining electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS). Based on the complementary characteristics of electrophysiological signals EEG and hemodynamic responses (fNIRS) in terms of signal modalities, and combined with the inherent advantage of no signal crosstalk during synchronous acquisition, EEG-fNIRS multimodal data fusion can significantly improve the accuracy of motor imagery task recognition. The invention will now be described in detail with reference to the accompanying drawings.

[0041] See Figure 1 The present invention provides a method for recognizing motor imagery by combining electroencephalography (EEG) and functional near-infrared spectroscopy, specifically comprising the following steps:

[0042] Step 1: EEG-fNIRS Synchronization Data Preprocessing and Data Augmentation

[0043] The experiment was designed according to the paradigm of motor imagery experiments, simultaneously collecting EEG and fNIRS data from subjects during the motor imagery task. The specific experimental procedure included: a 2-second instruction phase, a 10-second task execution phase, and a 15-17 second rest phase, with a 60-second buffer phase between each experiment. During the task execution phase, subjects completed a motor imagery task involving the imagination of the "left hand" and "right hand." The collected data will be precisely labeled according to the timeline of the experimental paradigm, and will undergo preprocessing and data augmentation.

[0044] Specifically, during the motor imagery task implementation phase, participants used a tactile motor imagery paradigm to simulate hand grasping movements, maintaining neural focus through a non-visual cognitive mode. A visual cueing system indicated the left and right hand task directions with a continuous 2-second black arrow, and presented a fixation cross as the visual focus throughout the task period. Participants performed hand grasping imagery based on a 1 Hz rhythm mastered through pre-trained physical movements. Task termination was triggered simultaneously by a 1-second cue sound and a stop marker. The experimental design included three independent subgroups, each containing 20 randomly ordered unilateral hand trials, totaling 60 complete trial sequences. During the rest intervals, participants were required to maintain fixation on the same point to reduce eye movement artifacts.

[0045] During EEG signal preprocessing, baseline correction was first performed using data collected 3 seconds prior to the experiment. Artifacts such as blinking and electromyography (EMG) were then removed using bandpass filtering and independent component analysis. For functional near-infrared spectroscopy (fNIRS) signals, given the significant time lag effect of hemoglobin concentration changes, a sliding time window of 10 seconds with 1-second increments, matching the task execution time, was used for analysis. Using the modified Beer-Lambert law, the raw optical density data was converted into quantitative indicators reflecting changes in oxyhemoglobin and deoxyhemoglobin concentrations. The conversion process is described below:

[0046]

[0047] Where, ε HbO (λ1), ε HbO (λ2), ε HbR (λ1) and ε HbR (λ²) represents the absorption coefficients of oxyhemoglobin and deoxyhemoglobin at different wavelengths, respectively, and d represents the distance between the signal source and receiver. Baseline correction and bandpass filtering from 0.01 to 0.2 Hz are then performed. The preprocessed data is then time-segmented using a sliding time window of 3 s with a step size of 1 s. See [link to relevant documentation]. Figure 2 For segmented data, a data augmentation strategy based on Gaussian noise injection was implemented to augment the training samples with normally distributed noise. The final dimensions of the EEG data were (number of subjects, number of trials × number of segments, number of channels, 3-second time window × sampling frequency), and the dimensions of the functional near-infrared spectroscopy data were (number of subjects, 10-second time window × sampling frequency, number of trials × number of segments, number of channels, 3-second time window × sampling frequency).

[0048] Step 2: Pre-training of the masked autoencoder

[0049] Extract the time-frequency and spatial features of EEG and the spatial features of fNIRS. Pre-train these features using a deep mask autoencoder to extract low-dimensional representations of each feature and obtain the encoder pre-training weights in the deep mask autoencoder.

[0050] Specifically, considering the characteristics of EEG and fNIRS signals, the focus is on extracting the time-frequency and spatial features of EEG and the spatial features of fNIRS. For EEG signals, Morlet wavelet transform is used to extract time-frequency energy features. The time-frequency energy spectrum matrix is ​​constructed using continuous wavelet coefficients and can be calculated as follows:

[0051]

[0052] Where a is the scale parameter and b is the translation parameter. For complex conjugate, ω0 is the center frequency. The constant component; simultaneously, based on the weighted phase lag exponent, cross-channel phase synchronicity is calculated to quantify the information coupling characteristics of functional brain regions such as the motor cortex, which can be calculated as follows:

[0053]

[0054] Δφ(t)=φ x (t)-φ y (t)

[0055] in The imaginary part of the cross spectrum. The phase difference is given. For the fNIRS signal, leveraging its high spatial resolution advantage, the inter-channel functional connectivity matrix can be constructed using the Pearson correlation coefficient, and can be calculated as follows:

[0056]

[0057] Where x i and y i The fNIRS time series are for different cortical regions. Based on the above multi-domain features, reconstruction is performed using a sub-mask autoencoder. The calculation process is as follows: First, the EEG time-frequency features, EEG spatial features, and fNIRS spatial features are divided into blocks, and the feature matrix of each modality is segmented into... Where N i The total number of blocks for the i-th mode; then a dynamic random mask matrix is ​​constructed. Partial feature blocks are randomly masked with probability p = 0.4 to form partial observation input X. i ⊙M i In the encoding stage, a self-attention-enhanced feedforward network is used to extract the latent representation Z. i This involves capturing global dependencies across feature blocks using a multi-head attention mechanism. The multi-head attention mechanism can be computed as follows:

[0058]

[0059] The decoder block reconstructs the original features through inverse block operations and multi-head attention mechanisms. Among them, the loss function is adopted.

[0060]

[0061] Where K i =∑ j M ij , where m is the number of input modalities. This loss function calculates the mean squared error only for the occluded region, forcing the model to infer global context information from visible feature blocks, thereby enhancing its robustness to noise interference.

[0062] Step 3: Cross-modal feature fusion and classification

[0063] The pre-trained encoder weights are used for weight initialization. The time-frequency and spatial features of EEG and the spatial features of fNIRS are extracted by the encoder to extract their latent representations. The features are then fused through a cross-modal cross-attention mechanism, and the result of the feature fusion is classified using the cross-entropy loss function.

[0064] The proposed deep fusion network architecture diagram is as follows: Figure 3As shown. The pre-trained sub-mask autoencoder parameters are loaded to initialize the encoder module of the fusion classification model, preserving the noise-resistant feature extraction capability learned during the pre-training stage; the EEG time-frequency features are then... EEG airspace characteristics fNIRS Spatial Features Each input is fed into the corresponding encoder to generate the corresponding low-dimensional representation Z. i .like Figure 4 As shown, a multi-layer cross-modal cross-attention mechanism for heterogeneous data is used to interactively fuse multimodal features. Specifically, this includes intra-modal cross-attention and cross-modal cross-attention mechanisms, such as... Figure 5 As shown. In the EEG intramodal attention mechanism, time-frequency features are... As a query, spatial features As a key / value pair, the process can be represented as follows:

[0065]

[0066] Among them W q t W k ts and W v ts For learnable parameter matrix, EEG time-frequency characteristics token, EEG spatial characteristics The patch, σ, represents layer normalization. EEG time-frequency features are fused with the functional connectivity network using a cross-attention mechanism; the calculation process is as follows:

[0067]

[0068] Where f(n) and g(n) represent the mapping functions used for dimension mapping. In the cross-modal cross-attention mechanism, EEG spatial features are used respectively. For the query, fNIRS spatial features For key / value pairs, compute cross-modal attention dominated by EEG spatial features; using fNIRS Spatial domain features are Query and EEG spatial domain features. For the key / value pair, calculate cross-modal attention dominated by fNIRS spatial features. Finally, output the final predicted probability through a fully connected layer and a softmax layer. The expression for the loss function derived from the predicted probability is:

[0069]

[0070] Where L is the loss function and C is the number of label categories. This represents the probability value predicted by the network that the sample belongs to the i-th category label.

[0071] The embodiments of the present invention have been described in detail above with reference to the accompanying drawings, but the present invention is not limited to the described embodiments. For those skilled in the art, various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principles and spirit of the present invention, and these variations still fall within the protection scope of the present invention.

Claims

1. A motor imagery recognition method combining electroencephalogram and functional near-infrared spectroscopy, characterized in that: The method comprises the following steps: Step one: set up an experiment according to the motor imagery experiment paradigm, synchronously collect the electroencephalogram (EEG) data and functional near-infrared spectroscopy (fNIRS) data of a subject under the motor imagery task, pre-process the collected synchronous data and perform data enhancement; Step two: extract the time-frequency features and spatial features of the EEG and the spatial features of the fNIRS, pre-train the features through a deep mask autoencoder to extract low-dimensional representations of the features and obtain the training weights of the encoder; Step three: take the pre-trained encoder as the initial weights, fuse the time-frequency features and spatial features of the EEG and the spatial features of the fNIRS through a cross-modal cross-attention mechanism, and classify the fusion results through a cross-entropy loss; The step three specifically comprises: S3.1: Load the parameters of the sub-mask code completed in step two training, initialize the parameters of the encoder module of the fusion classification model, and retain the anti-noise feature extraction ability learned in the pre-training stage; input the EEG time-frequency features EEG spatial features fNIRS spatial features respectively into the corresponding encoder to generate the corresponding low-dimensional representation Z i ; S3.2: Interactively fuse the multi-modal features through the cross-modal cross-attention mechanism for heterogeneous data, including intra-modal cross-attention mechanism and cross-modal cross-attention mechanism, in the EEG modal attention mechanism, the time-frequency features As Query, spatial features As Key / Value, the process can be represented as follows: where W q t , W k ts and W v ts are learnable parameter matrices, σ denotes layer normalization, and the fusion of EEG time-frequency features and functional connectivity networks is achieved through cross-attention mechanisms, and the calculation process is as follows: where f(n) and g(n) represent mapping functions for dimension mapping, in the cross-modal cross-attention mechanism, EEG spatial features are Query, fNIRS spatial features are Key / Value, to calculate the cross-modal attention dominated by EEG spatial features; fNIRS spatial features are Query, EEG spatial features are Key / Value, to calculate the cross-modal attention dominated by fNIRS spatial features.

2. The method according to claim 1, wherein the method is characterized by: The step one specifically comprises: S1.1: for the EEG data, first intercept 3 seconds of resting state data before the task to implement baseline correction and eliminate individual resting potential differences; through a band-pass filter to suppress low-frequency drift and high-frequency electromyographic interference, and combine independent component analysis to separate and remove blinking and head movement physiological artifacts; For the fNIRS data, a sliding time window with a size of 10 seconds and a step of 1 second matching the task execution time is used for analysis, the original optical density data is converted into a quantitative indicator reflecting the concentration changes of oxyhemoglobin and deoxyhemoglobin by using the modified Beer-Lambert law, and baseline correction and band-pass filtering are performed; S1.2: take the pre-processed data in S1.1, uniformly align the pre-processed EEG and fNIRS data using a sliding time window; for the segmented data, implement a data enhancement strategy based on Gaussian noise injection to apply normally distributed noise to expand the training samples.

3. The method according to claim 1, wherein the method is characterized by: The step two specifically comprises: S2.1: for the EEG signal, use Morlet wavelet transform to extract time-frequency energy features, and construct a time-frequency energy spectrum matrix through continuous wavelet coefficients; calculate the cross-channel phase synchronization based on the weighted phase lag index to quantify the information coupling characteristics between functional brain regions; for the fNIRS signal, focus on its high spatial resolution advantage, and construct a functional connection matrix between channels through the Pearson correlation coefficient; S2.2: for the above three features, reconstruct them through sub-mask autoencoders respectively.

4. The method of claim 1, wherein the method is characterized in that: On the fusion features obtained in step three, the final prediction probability is output through a fully connected layer and a Softmax layer, and the expression of the loss function according to the prediction probability is: where L is a loss function, C is the number of label classes, represents the probability value of the network predicting that the sample belongs to the i-th class label.

Images