Motor imagery recognition method combining electroencephalogram and functional near infrared spectrum

Abstract

The invention relates to a motor imagery recognition method combining EEG (electroencephalogram) and functional near infrared spectroscopy, which comprises the following steps: synchronously acquiring EEG data and functional near infrared spectroscopy signal fNIRS data under a motor imagery task through a multi-channel acquisition system, and preprocessing and enhancing the acquired two modal data. Aiming at the characteristics of the two signals, respectively extracting the time-frequency characteristics and the space-domain characteristics of the EEG and the space-domain characteristics of the fNIRS; designing a mask auto-encoder for each modal feature to carry out pre-training; and finally, carrying out feature fusion through a cross-modal cross attention mechanism in combination with parameters obtained by pre-training, and carrying out training through cross entropy loss. According to the method, the accuracy of motor imagery recognition is improved, and independent information and complementary information of EEG and fNIRS are effectively combined.

Description

Technical Field

[0001] The present invention relates to a method for recognizing motor imagery by combining electroencephalogram (EEG) and functional near-infrared spectroscopy (fNIRS), belonging to the technical field of signal processing. Background Art

[0002] Motor imagery is a paradigm based on decoding brain neural activities, and its background stems from the exploration of the neural mechanisms of the motor cortex. When an individual imagines performing a specific limb movement, even if no actual movement is generated, the neuron groups in the sensorimotor area of the brain will still be activated. A brain-computer interface (BCI) system captures and classifies these characteristic signals and converts them into control instructions for external devices. The advantage of motor imagery is that it can directly achieve human-computer interaction through mental training without external stimuli or actual movements, so it has attracted much attention in the fields of neurorehabilitation and assistive technologies.

[0003] In the process of transforming the laboratory-verified motor imagery BCI paradigm into clinical practice, there are still key obstacles such as the stability of neural signals, individual adaptability, and environmental anti-interference. To solve these problems, the selection of real-time neuroimaging techniques needs to seek an optimal balance among temporal resolution, spatial accuracy, portability, and operation cost. Compared with techniques such as functional magnetic resonance imaging (fMRI) and magnetoencephalography (MEG) that are limited by high equipment costs and bulky volumes, as well as the invasive operations and low temporal resolution defects involved in positron emission tomography (PET), electroencephalogram (EEG) and functional near-infrared spectroscopy (fNIRS) stand out due to their unique advantages. EEG captures the rhythm oscillations induced by motor imagery with millisecond-level temporal resolution, while fNIRS provides centimeter-level spatial localization through the hemodynamic response. The two can be synchronously collected through wearable devices. This combination with complementary spatio-temporal characteristics not only meets the requirements of clinical scenarios for real-time performance, portability, and cost control, but also can significantly enhance the robustness and reliability of motor decoding through the deep fusion of multi-modal data.

[0004] Although the multi-modal fusion of EEG-fNIRS improves the performance of motor imagery BCI, however, there are still many deficiencies in the processing of existing algorithms in aspects such as heterogeneous data fusion and individual differences. Therefore, how to more efficiently mine and utilize the heterogeneous information of EEG and fNIRS signals is still a key issue. Summary of the Invention

[0005] In order to overcome the deficiencies of the existing technology, the present invention provides a method for recognizing motor imagery by combining electroencephalogram and functional near-infrared spectroscopy, which makes full use of the multi-domain features of EEG and fNIRS to improve the accuracy of motor imagery recognition and effectively combines the independent information and complementary information of EEG and fNIRS.

[0006] A method for motor imagery recognition combining electroencephalogram and functional near-infrared spectroscopy, comprising the following steps:

[0007] Step 1: Set up an experiment according to the motor imagery experimental paradigm, synchronously collect electroencephalogram (EEG) data and functional near-infrared spectroscopy (fNIRS) data of the subject under the motor imagery task, preprocess the collected synchronous data and perform data augmentation;

[0008] The specific process of the experiment includes: a 2-second instruction phase, a 10-second task execution phase, a 15-second to 17-second rest phase, and a 60-second buffer phase between each experiment. During the task execution phase, the subject needs to complete the motor imagery task, and the task content involves the imagination of "left hand" and "right hand". For the collected data, it will be accurately labeled according to the time flow nodes of the experimental paradigm, and the data will be preprocessed and data augmented.

[0009] Step 2: Extract the time-frequency features and spatial domain features of EEG, and the spatial domain features of fNIRS, pre-train the above features through a deep masked autoencoder, extract the low-dimensional representation of the features, and obtain the training weights of the encoder;

[0010] Step 3: Use the encoder obtained by pre-training as the initial weight, fuse the time-frequency features and spatial domain features of EEG, and the spatial domain features of fNIRS through a cross-modal cross-attention mechanism, and classify the fusion result through cross-entropy loss.

[0011] The specific content of Step 1 includes:

[0012] S1.1: For EEG data, first intercept the 3-second resting state data before the task to perform baseline correction to eliminate individual resting potential differences; then suppress low-frequency drift and high-frequency myoelectric interference through a band-pass filter, and combine independent component analysis to separate and remove physiological artifacts such as blinks and head movements;

[0013] For fNIRS data, considering its 6-8 second delay characteristic of the blood oxygen dynamics response, a sliding time window with a size of 10 seconds and a step of 1 second that matches its task execution time is used to analyze it. Using the modified Beer-Lambert law, the original optical density data is converted into a quantitative index reflecting the changes in oxyhemoglobin concentration and deoxyhemoglobin concentration, and baseline correction and band-pass filtering are performed.

[0014] S1.2: Take the data preprocessed in S1.1, and use a sliding time window with a size of 3s and a step of 1s to uniformly align the preprocessed EEG and fNIRS data; implement a data augmentation strategy based on Gaussian noise injection for the segmented data, and apply normal distribution noise to expand the training samples.

[0015] Step 2 specifically includes the following:

[0016] S2.1: For EEG signals, Morlet wavelet transform is used to extract time-frequency energy features, and a time-frequency energy spectrum matrix is constructed through continuous wavelet coefficients; based on the weighted phase lag index, cross-channel phase synchrony is calculated to quantify the information coupling features between functional brain regions such as the motor cortex. For fNIRS signals, focusing on its high spatial resolution advantage, a functional connectivity matrix between channels is constructed through the Pearson correlation coefficient.

[0017] S2.2: For the above three types of features, reconstruction is performed through a sub-mask autoencoder respectively, and the calculation process is as follows:

[0018] First, the EEG time-frequency features, EEG spatial features, and fNIRS spatial features are blocked, and the feature matrices of each modality are divided into where N i is the total number of blocks for the i-th modality; subsequently, a dynamic random mask matrix is used to randomly mask some feature blocks with a probability p = 0.4 to form a partially observed input X i ⊙M i . In the encoding stage, a feed-forward network enhanced by self-attention is used to extract the latent representation Z i , and the global dependence relationship across feature blocks is captured through the multi-head attention mechanism; in the decoding stage, the original features are reconstructed through inverse blocking operation and the multi-head attention mechanism The loss function is used:

[0019]

[0020] where K i = ∑ j M ij , and m is the number of input modalities.

[0021] Step 3 specifically includes the following:

[0022] S3.1: Load the parameters of the sub-mask autoencoder trained in Step 2 to initialize the parameters of the encoder module of the fusion classification model, and retain the anti-noise feature extraction ability learned in the pre-training stage; input the EEG time-frequency features EEG spatial features fNIRS spatial features into the corresponding encoders respectively to generate the corresponding low-dimensional representations Z i ;

[0023] S3.2: Cross-modal cross-attention mechanism for heterogeneous data is used to interact and fuse multi-modal features, specifically including intra-modal cross-attention mechanism and cross-modal cross-attention mechanism. In the EEG intra-modal attention mechanism, the time-frequency features As the Query, the spatial domain feature As the Key / Value, the process can be expressed as follows:

[0024]

[0025] Where W q t 、W k ts And W v ts Are learnable parameter matrices, σ represents layer normalization, and the fusion of EEG time-frequency features and the functional connectivity network is achieved through the cross-attention mechanism. The calculation process is as follows:

[0026]

[0027] Where f(n) and g(n) represent mapping functions for dimension mapping. In the cross-modal cross-attention mechanism, taking the EEG spatial domain feature As the Query, and the fNIRS spatial domain feature As the Key / Value, calculate the cross-modal attention dominated by the EEG spatial domain feature; taking the fNIRS Spatial domain feature as the Query, and the EEG spatial domain feature As the Key / Value, calculate the cross-modal attention dominated by the fNIRS spatial domain feature.

[0028] On the fusion feature obtained in Step 3, the final prediction probability is output through the fully connected layer and the Softmax layer. The expression of the loss function according to the prediction probability is:

[0029]

[0030] Where L is the loss function, C is the number of label categories, Represents the probability value that the network predicts this sample belongs to the i-th class label.

[0031] Compared with the existing EEG-fNIRS multi-modal fusion methods, the present invention has the following advantages:

[0032] To solve the problem of insufficient feature fusion in existing EEG-fNIRS integration work, the present invention develops a deep fusion framework based on EEG-fNIRS. This framework makes full use of the high temporal resolution of EEG and the high spatial resolution of fNIRS. Through the cross-modal cross-attention mechanism, the fusion of the time-frequency features of EEG and the spatial features of EEG can obtain a more comprehensive multi-domain representation; while the interactive fusion of the spatial features of EEG and the spatial features of fNIRS compensates for the lack of EEG spatial resolution, captures the bidirectional coupling mechanism of EEG and fNIRS at the same time, and realizes the efficient fusion of the two modalities. At the same time, pre-training with a masked autoencoder accelerates model training, significantly improves the generalization performance of the model, and achieves more accurate classification and recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] To more clearly illustrate the technical solutions in the embodiments of the present invention or in the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0034] Figure 1 It is a flowchart of the method for motion imagination recognition combining electroencephalogram and functional near-infrared spectroscopy of the present invention.

[0035] Figure 2 It is a flowchart of the data augmentation process of the present invention.

[0036] Figure 3 It is a structural diagram of the deep fusion network combining electroencephalogram and functional near-infrared spectroscopy proposed by the present invention.

[0037] Figure 4 It is the feature fusion modality involved in the present invention.

[0038] Figure 5 It is a structural diagram of the cross-modal cross-attention mechanism involved in the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0039] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the drawings in the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, rather than all embodiments. All other embodiments obtained by those of ordinary skill in the art without creative efforts based on the embodiments of the present invention belong to the scope of protection of the present invention.

[0040] This embodiment will describe in detail a motor imagery recognition method combining electroencephalogram (EEG) and functional near-infrared spectroscopy (fNIRS) with reference to the accompanying drawings. Based on the complementary characteristics of electrophysiological signals EEG and hemodynamic responses fNIRS in signal modalities, and the natural advantage of no signal crosstalk during synchronous acquisition, the EEG-fNIRS multimodal data fusion can significantly improve the accuracy of motor imagery task recognition. The present invention will be described in detail below with reference to the accompanying drawings.

[0041] See Figure 1 , a motor imagery recognition method combining electroencephalogram and functional near-infrared spectroscopy of the present invention specifically includes the following steps:

[0042] Step 1: EEG-fNIRS synchronous data preprocessing and data augmentation

[0043] Set up the experiment according to the motor imagery experimental paradigm, and synchronously collect the electroencephalogram signal EEG data and the functional near-infrared spectroscopy signal fNIRS data of the subject under the motor imagery task. The specific process of the experiment includes: a 2-second instruction stage, a 10-second task execution stage, a 15-second to 17-second rest stage, and a 60-second buffer stage between each experiment. During the task execution stage, the subject needs to complete the motor imagery task, and the task content involves the imagination of "left hand" and "right hand". The collected data will be accurately labeled according to the time flow nodes of the experimental paradigm, and the data will be preprocessed and data augmented.

[0044] Specifically, during the implementation stage of the motor imagery task, the subject uses a tactile motor imagery paradigm to simulate the hand grasping movement and maintains neural concentration through a non-visual cognitive mode. The visual cue system indicates the task direction of the left and right hands through a 2-second black arrow, and a fixation cross is presented as the visual focus during the task cycle. The subject imagines hand grasping according to the 1-hertz rhythm mastered through prior physical action training, and the task termination is triggered synchronously by a 1-second prompt sound effect and a stop sign. The experimental design includes three independent groups, each group contains 20 unilateral hand trials in random order, and a total of 60 complete trial sequences are formed. During the intermittent stage, the subject is required to maintain fixation on the fixation point to reduce the interference of eye movement artifacts.

[0045] During the preprocessing of the electroencephalogram signal, first, baseline correction is performed by taking the data of the first 3 seconds before the experiment, and artifacts such as blinks and electromyograms are removed through band-pass filtering and independent component analysis. For the functional near-infrared spectroscopy signal, considering the obvious time lag effect of the hemoglobin concentration change of the fNIRS signal, a sliding time window with a size of 10 seconds and a step of 1 second that matches the task execution time is used to analyze it. Using the modified Beer-Lambert law, the original optical density data is converted into a quantitative index reflecting the changes in oxyhemoglobin concentration and deoxyhemoglobin concentration. The conversion process is described as follows:

[0046]

[0047] where ε HbO (λ1), ε HbO (λ2), ε HbR (λ1) and ε HbR (λ2) represent the absorption coefficients of oxyhemoglobin and deoxyhemoglobin at different wavelengths respectively, and d represents the distance between the signal source and the receiver. Subsequently, baseline correction and band-pass filtering from 0.01 to 0.2 Hz are performed. Taking the preprocessed data, the preprocessed data is segmented in time series using a sliding time window with a size of 3 s and a step size of 1 s. See Figure 2 , and a data augmentation strategy based on Gaussian noise injection is implemented for the segmented data, and normal distribution noise is applied to expand the training samples. The final dimension of the EEG data is (number of subjects, number of trials × number of segments, number of channels, 3 s time window × sampling frequency), and the dimension of the functional near-infrared spectroscopy data is (number of subjects, 10 s time window × sampling frequency, number of trials × number of segments, number of channels, 3 s time window × sampling frequency).

[0048] Step 2: Pretraining of the masked autoencoder

[0049] Extract the time-frequency features and spatial features of EEG and the spatial features of fNIRS, and pre-train the above features through a deep masked autoencoder to extract the low-dimensional representations of the above respective features, and obtain the pre-trained weights of the encoder in the deep masked autoencoder.

[0050] Specifically, according to the respective characteristics of the EEG signal and the fNIRS signal, focus on extracting the time-frequency features and spatial features of EEG and the spatial features of fNIRS. Among them, for the EEG signal, Morlet wavelet transform is used to extract the time-frequency energy features, and the time-frequency energy spectrum matrix is constructed through continuous wavelet coefficients, which can be calculated as follows:

[0051]

[0052] where a is the scale parameter, b is the translation parameter, is the complex conjugate, ω0 is the central frequency, is the constant component; at the same time, based on the weighted phase lag index, the cross-channel phase synchrony is calculated to quantify the information coupling characteristics between functional brain regions such as the motor cortex, which can be calculated as follows:

[0053]

[0054] Δφ(t) = φ x (t) - φ y (t)

[0055] where is the imaginary part of the cross-spectrum, and is the phase difference. For fNIRS signals, focusing on its high spatial resolution advantage, a functional connectivity matrix between channels can be constructed through the Pearson correlation coefficient, and the calculation is as follows:

[0056]

[0057] where x i and y i are the fNIRS time series of different cortical regions. Based on the above multi-domain features, reconstruction is performed through a sub-mask autoencoder respectively, and the calculation process is as follows. First, the EEG time-frequency features, EEG spatial features, and fNIRS spatial features are divided into blocks, and the feature matrices of each modality are segmented into where N i is the total number of blocks for the i-th modality; subsequently, a dynamic random mask matrix is constructed to randomly mask some feature blocks with a probability p = 0.4 to form a partially observed input X i ⊙M i . In the encoding stage, a feed-forward network enhanced by self-attention is used to extract the latent representation Z i , and the global dependence relationship across feature blocks is captured through the multi-head attention mechanism. The multi-head attention mechanism can be calculated as follows

[0058]

[0059] The decoder block then reconstructs the original features through inverse block operation and the multi-head attention mechanism where the loss function

[0060]

[0061] where K i = ∑ j M ij , and m is the number of input modalities. This loss function only calculates the mean square error for the masked area, forcing the model to infer the global context information through the visible feature blocks, thereby enhancing the robustness to noise interference.

[0062] Step 3: Cross-modal feature fusion and classification

[0063] The weights obtained from pre-training are used for weight initialization. The time-frequency features and spatial features of EEG, and the spatial features of fNIRS are used to extract their latent representations through the encoder, and feature fusion is performed through the cross-modal cross-attention mechanism, and the result of feature fusion is classified through the cross-entropy loss function.

[0064] The structure diagram of the proposed deep fusion network is as shown in Figure 3As shown, load the parameters of the pre-trained sub-mask autoencoder to initialize the parameters of the encoder module of the fusion classification model, retaining the anti-noise feature extraction ability learned in the pre-training stage; input the EEG time-frequency features EEG spatial domain features fNIRS spatial domain features into the corresponding encoders respectively to generate the corresponding low-dimensional representations Z i . As Figure 4 shown, interactively fuse the multi-modal features through a multi-layer cross-modal cross-attention mechanism for heterogeneous data, specifically including the intra-modal cross-attention mechanism and the cross-modal cross-attention mechanism, as Figure 5 shown. In the EEG intra-modal attention mechanism, take the time-frequency features as Query, and the spatial domain features as Key / Value. The process can be expressed as follows:

[0065]

[0066] where W q t , W k ts and W v ts are learnable parameter matrices, is the token of the EEG time-frequency feature , is the patch of the EEG spatial domain feature , and σ represents layer normalization. The fusion of the EEG time-frequency feature and the functional connectivity network is achieved through the cross-attention mechanism. The calculation process is as follows:

[0067]

[0068] where f(n) and g(n) represent the mapping functions for dimension mapping. In the cross-modal cross-attention mechanism, take the EEG spatial domain feature as Query, and the fNIRS spatial domain feature as Key / Value to calculate the cross-modal attention dominated by the EEG spatial domain feature; take the fNIRS spatial domain feature as Query, and the EEG spatial domain feature as Key / Value to calculate the cross-modal attention dominated by the fNIRS spatial domain feature. Finally, output the final prediction probability through the fully connected layer and the Softmax layer. The expression of the loss function according to the prediction probability is:

[0069]

[0070] where L is the loss function and C is the number of label categories, represents the probability value that the network predicts this sample belongs to the i-th category label.

[0071] The above has described the embodiments of the present invention in detail in conjunction with the accompanying drawings, but the present invention is not limited to the described embodiments. For those skilled in the art, without departing from the principles and spirit of the present invention, various changes, modifications, substitutions, and variations made to these embodiments still fall within the protection scope of the present invention.

Claims

1. A method for motor imagery recognition combining electroencephalogram and functional near-infrared spectroscopy, characterized in that: It includes the following steps: Step 1: Set up an experiment according to the motor imagery experimental paradigm, synchronously collect electroencephalogram (EEG) data and functional near-infrared spectroscopy (fNIRS) data of the subject under the motor imagery task, preprocess the collected synchronous data, and perform data augmentation; Step 2: Extract the time-frequency features and spatial domain features of EEG, and the spatial domain features of fNIRS. Pre-train the above features through a deep masked autoencoder, extract the low-dimensional representation of the features, and obtain the training weights of the encoder; Step 3: Use the encoder obtained from pre-training as the initial weight, fuse the time-frequency features and spatial domain features of EEG, and the spatial domain features of fNIRS through a cross-modal cross-attention mechanism, and classify the fusion result through cross-entropy loss.

2. The method for motor imagery recognition combining electroencephalogram and functional near-infrared spectroscopy according to claim 1, characterized in that: The specific content of Step 1 includes: S1.1: For EEG data, first intercept the 3-second resting state data before the task to perform baseline correction to eliminate individual resting potential differences; suppress low-frequency drift and high-frequency myoelectric interference through a band-pass filter, and combine independent component analysis to separate and remove physiological artifacts such as blinking and head movement; For fNIRS data, analyze it using a sliding time window with a size of 10 seconds and a step of 1 second that matches its task execution time. Using the modified Beer-Lambert law, convert the original optical density data into a quantitative index reflecting the changes in oxyhemoglobin concentration and deoxyhemoglobin concentration, and perform baseline correction and band-pass filtering; S1.2: Take the data after preprocessing in S1.1, and uniformly align the preprocessed EEG and fNIRS data using a sliding time window; implement a data augmentation strategy based on Gaussian noise injection for the segmented data, and apply normal distribution noise to expand the training samples.

3. The method for motor imagery recognition combining electroencephalogram and functional near-infrared spectroscopy according to claim 1, wherein: The specific content of Step 2 includes: S2.1: For EEG signals, use Morlet wavelet transform to extract time-frequency energy features, and construct a time-frequency energy spectrum matrix through continuous wavelet coefficients; calculate the cross-channel phase synchrony based on the weighted phase lag index to quantify the information coupling characteristics between functional brain regions such as the motor cortex; for fNIRS signals, focus on its high spatial resolution advantage and construct an inter-channel functional connectivity matrix through the Pearson correlation coefficient; S2.2: Reconstruct the above three types of features respectively through a sub-masked autoencoder.

4. A motor imagery recognition method combining electroencephalogram and functional near-infrared spectroscopy according to claim 1, characterized in that: The specific content of Step 3 includes: S3.1: Load the parameters of the sub-mask autoencoder trained in Step 2 to initialize the parameters of the encoder module of the fusion classification model, and retain the anti-noise feature extraction ability learned in the pre-training stage; input the EEG time-frequency features EEG spatial domain features fNIRS spatial domain features into the corresponding encoders respectively to generate the corresponding low-dimensional representations Z i ; S3.2: Interactively fuse multimodal features through a cross-modal cross-attention mechanism for heterogeneous data, including an intra-modal cross-attention mechanism and a cross-modal cross-attention mechanism. In the EEG intra-modal attention mechanism, the time-frequency features are used as the Query, and the spatial domain features are used as the Key / Value. The process can be expressed as follows: Among which W q t 、W k ts and W v ts are learnable parameter matrices, σ represents layer normalization, and the fusion of EEG time-frequency features and the functional connectivity network is achieved through the cross-attention mechanism. The calculation process is as follows: Among them, f(n) and g(n) represent the mapping functions for dimensionality mapping. In the cross-modal cross-attention mechanism, the EEG spatial domain features are used as the Query, and the fNIRS spatial domain features are used as the Key / Value to calculate the cross-modal attention dominated by the EEG spatial domain features; the fNIRS spatial domain features are used as the Query, and the EEG spatial domain features are used as the Key / Value to calculate the cross-modal attention dominated by the fNIRS spatial domain features.

5. A motor imagery recognition method combining electroencephalogram and functional near-infrared spectroscopy according to claim 1, characterized in that: On the fusion features obtained in Step 3, output the final prediction probability through a fully connected layer and a Softmax layer. The expression of the loss function according to the prediction probability is: where L is the loss function and C is the number of label categories, indicating the probability value that the network predicts this sample belongs to the i-th class label.

Images