An infrared brain-computer interface method with multi-modal signals
Patent Information
- Application Number
- CN202610378333.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-03-26
- Publication Date
- 2026-06-26
Smart Images
Figure CN122284819A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of brain-computer interface technology, and in particular to an infrared brain-computer interface method with multimodal signals. Background Technology
[0002] Brain-computer interface is a technology that enables direct communication between the brain and external devices without relying on peripheral nerves and muscle tissue. It allows brain signals to be converted into control commands that can be recognized by the device, showing great application potential in many fields such as medical rehabilitation, smart homes, and entertainment.
[0003] Traditional brain-computer interfaces (BCIs) are primarily based on single signal modalities, such as electroencephalograms (EEGs), functional magnetic resonance imaging (fMRI), or near-infrared spectroscopy. However, single-modal signals have inherent limitations. For example, while EEGs directly reflect brain electrical activity, they are susceptible to individual differences, environmental noise, and their own characteristics, making it difficult to achieve ideal accuracy in decoding brain intentions using a single signal. Secondly, traditional BCI signal acquisition is highly susceptible to motion artifacts and environmental noise. Slight head movements and muscle contractions during BCI use can generate motion artifacts that are superimposed on EEG or infrared spectral signals, leading to signal distortion. This results in deviations in the decoding of brain intentions using contaminated signals, reducing system performance. Furthermore, commonly used brain intention decoding models, due to their limited model complexity, struggle to learn the complex nonlinear mapping relationship between brain signals and intentions when processing complex brain signals. Limitations in training methods and network structures also make them prone to overfitting or underfitting when processing multimodal fusion features, resulting in low decoding accuracy and failing to meet the high-precision decoding requirements of practical applications. Summary of the Invention
[0004] The purpose of this invention is to provide an infrared brain-computer interface method with multimodal signals, thereby solving the technical problems existing in the prior art.
[0005] To achieve the above-mentioned objectives, the technical solution adopted by this invention is as follows:
[0006] An infrared brain-computer interface method with multimodal signals includes the following steps: S1: Simultaneously acquiring the user's infrared spectral signals, EEG signals, and eye-tracking signals using a multimodal sensing system; S2: Preprocessing the acquired multimodal signals and removing motion artifacts and environmental noise using an adaptive filtering algorithm; S3: Constructing a multimodal feature fusion network based on an attention mechanism to perform hierarchical fusion of the infrared spectral features, EEG features, and eye-tracking features processed in step S2, generating a joint feature vector; S4: Inputting the joint feature vector into a deep belief network for brain intention decoding and outputting corresponding control commands; S5: Based on the deviation between the decoding result and the actual control effect, dynamically adjusting the feature fusion weights and network parameters using a reinforcement learning algorithm to achieve closed-loop adaptive optimization.
[0007] Furthermore, the infrared spectral signal in step S1 is acquired by a near-infrared spectral module. The near-infrared spectral module uses a four-wavelength laser light source with wavelengths of 780nm, 850nm, 940nm and 1064nm, and has ≥32 detection channels. The detection channels are distributed in the prefrontal cortex, motor cortex and parietal lobe regions according to the 10-20 international EEG system standard.
[0008] Furthermore, the EEG signals in step S1 are acquired using a high-density EEG electrode array, which comprises 64-128 electrodes. Electrodes, sampling frequency of 1000-2000Hz, input impedance > Common-mode rejection ratio > 120dB.
[0009] Furthermore, the eye movement signal in step S1 is acquired using the corneal reflection method, with a sampling frequency of 250-450Hz.
[0010] Furthermore, the multimodal signals in step S2 include: near-infrared signals, EEG signals, and EOG signals; the preprocessing step for the near-infrared signals is: R1: removing bad pixels from the original light intensity signal using... The criteria identify defects caused by sudden changes in light intensity and replace them using linear interpolation; R2: calculated using differential spectroscopy for optical density signals in the 760nm and 850nm bands. , , The concentration change time series; R3: Using sliding window polynomial fitting to remove low-frequency baseline drift of the signal; The preprocessing steps of the EEG signal are as follows: T1: Using the Butterworth function of 0.5-80Hz to remove DC drift and high-frequency noise; Using an adaptive notch filter of 50Hz and its harmonics to remove power frequency interference; T2: Using full-channel average reference to eliminate common-mode interference; T3: Using correlation coefficient method and standard deviation method to identify bad channels; T4: According to the event marker of the paradigm, the continuous EEG signal is segmented into signals of individual trials; The time window of each trial is from 100ms before the event to 4000ms after the event, and the signal of 150ms before the event is used as the baseline for baseline correction; The preprocessing steps of the EOG signal are as follows: Q1: Using the Butterworth function of 0.1-30Hz to remove baseline drift and high-frequency noise; Q2: Based on the threshold method to identify blink and saccade events, extract the number of blinks, blink duration, saccade amplitude, saccade frequency, and fixation time ratio for processing.
[0011] Furthermore, the adaptive filtering algorithm in step S2 includes: a dynamic noise suppression module based on Kalman filtering and an artifact separation module based on independent component analysis; the dynamic noise suppression module based on Kalman filtering employs a state equation and an observation equation, wherein the state equation is... The observation equation is ,in, Here is the state transition matrix. For the observation matrix, and These are process noise and observation noise, respectively.
[0012] Furthermore, the attention-based multimodal feature fusion network includes:
[0013] The feature extraction layer extracts features from the infrared spectral signal, electroencephalogram signal, and eye movement signal.
[0014] The self-attention layer performs self-attention calculation on the features of each modality to obtain key features within the modality. The attention weights are calculated using the Softmax function.
[0015] A cross-attention layer is used to establish an attention interaction mechanism between modalities, and to achieve the association and fusion of features from different modalities through the attention weight matrix between modalities.
[0016] The fusion output layer maps the multimodal features processed by self-attention and cross-attention into a joint feature vector through a concatenation operation and a fully connected layer.
[0017] Furthermore, the deep belief network in step S4 includes a pre-training stage and a fine-tuning stage; the pre-training stage uses a contrastive divergence algorithm; the fine-tuning stage uses a backpropagation algorithm combined with a stochastic gradient descent algorithm to perform supervised parameter optimization.
[0018] Furthermore, the reinforcement learning algorithm in step S5 is a deep deterministic policy gradient algorithm, which uses feature fusion weights and network parameters as the action space, and uses decoding accuracy, response time and user fatigue as the reward function.
[0019] Compared with the prior art, the present invention has the following beneficial effects:
[0020] (i) This invention improves the accuracy of brain intention decoding by fusing multimodal signals, which provides more comprehensive and richer features; and by using an adaptive filtering algorithm to remove motion artifacts and environmental noise, it improves signal quality and thus ensures signal reliability.
[0021] (ii) This invention adaptively captures key features within a modality and intermodal correlation information through self-attention and cross-attention mechanisms, highlighting key information in signals from different modalities, better mining the potential connections between modalities, making the generated joint feature vector more representative, and helping to improve the performance of brain intention decoding;
[0022] (iii) The present invention inputs the joint feature vector into the deep belief network for brain intention decoding, which enables the deep belief network to better learn the mapping relationship between the joint feature vector and brain intention, and has a strong decoding capability, thereby accurately outputting the corresponding control command and improving the practicality and reliability of the system. Attached Figure Description
[0023] Figure 1 This is a flowchart of an infrared brain-computer interface method with multimodal signals disclosed in this invention. Detailed Implementation
[0024] To make the content of this invention easier to understand, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Identical components are represented by the same reference numerals. It should be noted that the terms "front," "rear," "left," "right," "up," and "down" used in the following description refer to directions in the accompanying drawings, while the terms "inner" and "outer" refer to directions toward or away from the geometric center of a specific component, respectively.
[0025] like Figure 1 As shown, this embodiment provides an infrared brain-computer interface method with multimodal signals, including the following steps:
[0026] S1: A multimodal sensing system is used to simultaneously acquire the user's infrared spectral signals, electroencephalogram (EEG) signals, and eye movement signals. The infrared spectral signals are acquired through a near-infrared spectral module, which uses a four-wavelength laser light source with wavelengths of 780nm, 850nm, 940nm, and 1064nm, and the power density of the light source is controlled at [value missing]. The sampling frequency is 100-500Hz, and the number of detection channels is ≥32. These detection channels are distributed in the prefrontal cortex, motor cortex, and parietal lobe regions according to the 10-20 international EEG system standard. This arrangement allows the near-infrared spectroscopy module to comprehensively and meticulously detect changes in blood oxygenation in key brain regions, providing strong data support for interpreting brain neural activity. The EEG signals are acquired through a high-density EEG electrode array, which includes 64-128 electrodes. Electrodes, sampling frequency of 1000-2000Hz, input impedance > The common-mode inhibition ratio is >120dB, which greatly suppresses common-mode interference, highlights the effective components of the EEG signal, and provides high-quality data for subsequent EEG signal analysis. The eye movement signal is acquired using the corneal reflectance method, with a sampling frequency of 250-450Hz and a spatial resolution better than 0.5°, which can accurately capture the minute rotations of the eyes. The tracking range covers horizontal ±60° and vertical ±40°, which can comprehensively track the movement trajectory of the eyes in all directions.
[0027] S2: The acquired multimodal signals are preprocessed, and motion artifacts and environmental noise are removed using an adaptive filtering algorithm; the multimodal signals include: near-infrared signals, EEG signals, and EOG signals;
[0028] The preprocessing steps for the near-infrared signal are as follows:
[0029] R1: Removes bad pixels from the original light intensity signal. The criteria identify bad pixels with abrupt changes in light intensity and replace them using linear interpolation. Specifically, once the location of the bad pixel is determined, linear interpolation is immediately applied for replacement. This linear interpolation is based on the changing trend of normal light intensity data around the bad pixel, reasonably estimating the light intensity value that the bad pixel should have, thereby ensuring the continuity and accuracy of the light intensity signal and avoiding data deviations caused by the presence of bad pixels.
[0030] R2: Calculated using differential spectroscopy for optical density signals in the 760nm and 850nm bands. , , The time series of concentration changes; this step is of key significance for understanding the blood oxygen supply and metabolic status of local brain regions;
[0031] R3: A sliding window polynomial fitting method is used to remove low-frequency baseline drift of the signal. The sliding window moves point by point on the signal sequence with a fixed step size. Each time, the data in the window is fitted by a polynomial to generate a smooth curve, which represents the baseline trend of the signal in the window. Subtracting the fitted baseline curve from the actual signal in the window can eliminate the influence of low-frequency baseline drift on the signal, making the signal more clearly present the high-frequency feature changes related to brain activity, and improving the usability and analytical value of the signal.
[0032] The preprocessing steps for the EEG signal are as follows:
[0033] T1: A Butterworth function with a frequency range of 0.5–80 Hz is used to remove DC drift and high-frequency noise. The Butterworth function, with its excellent passband flatness, can effectively remove DC drift components from the signal, causing the signal to fluctuate around zero level, while suppressing high-frequency noise and retaining key frequency components related to brain neural activity. An adaptive notch filter with a frequency of 50 Hz and its harmonics is used to remove power frequency interference, ensuring that the quality of the EEG signal is not affected.
[0034] T2: Employs full-channel average reference to eliminate common-mode interference; averages the signals acquired by all electrodes to obtain a reference signal representing common-mode interference; then subtracts the reference signal from the original signal of each electrode, thereby effectively eliminating common interference components in the signals of each electrode and highlighting differential signals related to local neuronal activity in the brain.
[0035] T3: The correlation coefficient method and standard deviation method are used to identify bad channels. The correlation coefficient method evaluates the consistency of the channel signal with the overall signal by calculating the correlation between each channel and the signals of other channels.
[0036] T4: Based on the event markers of the paradigm, the continuous EEG signal is segmented into signals of individual trials; the time window of each trial is from 100ms before the event to 4000ms after the event, which can not only fully capture the brain neural activity response triggered by the event, but also include the baseline state information before the event; the signal 150ms before the event is used as the baseline for baseline correction, and the signal within each trial is subtracted from the baseline signal to eliminate the signal offset between different trials caused by individual physiological state or collection environment, so that the signals of different trials can be compared and analyzed on the same benchmark.
[0037] The preprocessing steps for the EOG signal are as follows:
[0038] Q1: Using the Butterworth function of 0.1-30Hz, baseline drift and high-frequency noise are removed, making the signal more stable, while suppressing high-frequency noise and highlighting the effective signal components related to eye movement;
[0039] Q2: Accurate identification of blinking and saccade events based on threshold method. The threshold method can accurately distinguish between normal eye movements and special eye movement events such as blinking and saccade by setting reasonable signal amplitude and time thresholds. Then, by extracting the number of blinks, blink duration, saccade amplitude, saccade frequency, and fixation time ratio, the above feature parameters comprehensively reflect the eye movement state and visual attention allocation, providing rich eye movement information dimensions for subsequent comprehensive analysis combined with other multimodal signals.
[0040] The adaptive filtering algorithm includes: a dynamic noise suppression module based on Kalman filtering and an artifact separation module based on independent component analysis; the dynamic noise suppression module based on Kalman filtering employs a state equation and an observation equation, wherein the state equation is... The observation equation is ,in, Here is the state transition matrix. For the observation matrix, and These represent process noise and observation noise, respectively; the covariance matrix is updated in real time using an adaptive recursive least squares method; independent component analysis is performed using... The algorithm separates artifact components such as electromyography and electrooculography while preserving brain-derived signal components.
[0041] S3: Construct a multimodal feature fusion network based on the attention mechanism, and perform hierarchical fusion of infrared spectral features, EEG features and eye movement features processed in step S2 to generate a highly representative joint feature vector;
[0042] The attention-based multimodal feature fusion network includes:
[0043] The feature extraction layer extracts features from the infrared spectral signals, electroencephalogram (EEG) signals, and eye movement signals. Specifically, for the infrared spectral signals, the feature extraction layer can uncover feature information related to brain blood oxygen metabolism and neural activity areas, revealing the physiological mysteries of the brain. For the EEG signals, the feature extraction layer focuses on extracting features reflecting the patterns and rhythmic changes of neuronal electrical activity in the brain; these features are important clues for interpreting brain thought processes and cognitive processes. For the eye movement signals, the feature extraction layer is adept at capturing features such as changes in fixation point, saccade amplitude, and frequency; these features reflect the user's visual attention allocation and cognitive focus shifts.
[0044] The self-attention layer performs self-attention calculation on the features of each modality to obtain key features within the modality. The attention weights are calculated using the Softmax function. The Softmax function presents the relative importance of each feature within the modality in the form of a probability distribution, and gives higher weights to features that are more critical to the data of that modality.
[0045] The cross-attention layer establishes an attention interaction mechanism between modalities, and realizes the association and fusion of features of different modalities through the attention weight matrix between modalities, thereby organically integrating the unique information of each modality;
[0046] The fusion output layer maps the multimodal features processed by self-attention and cross-attention into a joint feature vector through a concatenation operation and a fully connected layer. Through concatenation, features from different modalities are integrated on the same dimension, preserving their respective characteristics while achieving information convergence. Then, the fully connected layer is used to map the concatenated features. The joint feature vector integrates the essential information from infrared spectroscopy, EEG, and eye-tracking signals, providing a comprehensive, accurate, and efficient feature representation for subsequent brain intent decoding, thereby improving the accuracy and reliability of interpreting the user's brain intent.
[0047] S4: The joint feature vector is input into a deep belief network for brain intention decoding, and the corresponding control command is output. The deep belief network contains multiple restricted Boltzmann machine layers and a Softmax classification layer. The deep belief network includes two stages: a pre-training stage and a fine-tuning stage. The pre-training stage uses a contrastive divergence algorithm, which adjusts the data distribution during training by quickly approximating the data distribution. The parameters; specifically, the contrastive divergence algorithm starts from a given data sample and utilizes... Sampling at A small number of alternating sampling steps are performed between the visible and hidden layers; during each sampling process, the energy difference between the samples in the visible and hidden layers is calculated to update the layer. The weights and bias parameters are determined; the fine-tuning stage employs a backpropagation algorithm combined with a stochastic gradient descent algorithm with momentum for supervised parameter optimization; it calculates the error between the predicted output and the true label, and propagates this error back along the network's hierarchical structure to obtain the contribution of each parameter to the error; in the fine-tuning stage, the joint feature vector and the corresponding brain intention label are used as input, the error is calculated through the backpropagation algorithm, and then the parameters of the deep belief network are adjusted using the stochastic gradient descent algorithm with momentum.
[0048] S5: Based on the deviation between the decoding result and the actual control effect, a reinforcement learning algorithm is used to dynamically adjust the feature fusion weights and network parameters to achieve closed-loop adaptive optimization. This allows the system to continuously adjust its state according to the actual operating conditions to achieve optimal performance. The reinforcement learning algorithm is a deep deterministic policy gradient algorithm, which uses the feature fusion weights and network parameters as the action space and the decoding accuracy, response time, and user fatigue as the reward function. The decoding accuracy directly reflects the accuracy of the system in decoding brain intentions; higher accuracy means the system can more accurately convert brain signals into control commands. The response time is related to the real-time performance of the system; a fast response ensures timely execution of control commands, improving the system's usability and user experience. User fatigue reflects the attention paid to the user's state in the actual application scenario. Since prolonged use of the brain-computer interface system may lead to user fatigue, which in turn affects signal acquisition and system performance, it is included in the reward function to help optimize the system, minimizing user burden while ensuring performance.
[0049] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. A method for an infrared brain-computer interface with multimodal signals, characterized in that: Includes the following steps: S1: Employs a multimodal sensing system to simultaneously acquire the user's infrared spectral signals, electroencephalogram (EEG) signals, and eye movement signals; S2: Preprocess the acquired multimodal signals and remove motion artifacts and environmental noise using an adaptive filtering algorithm; S3: Construct a multimodal feature fusion network based on an attention mechanism to perform hierarchical fusion of infrared spectral features, EEG features, and eye-tracking features processed in step S2 to generate a joint feature vector; S4: Input the joint feature vector into a deep belief network to decode brain intentions and output the corresponding control command; S5: Based on the deviation between the decoding result and the actual control effect, the feature fusion weights and network parameters are dynamically adjusted through reinforcement learning algorithm to achieve closed-loop adaptive optimization.
2. The infrared brain-computer interface method with multimodal signals according to claim 1, characterized in that: The infrared spectral signal in step S1 is acquired by a near-infrared spectral module. The near-infrared spectral module uses a four-wavelength laser light source with wavelengths of 780nm, 850nm, 940nm and 1064nm, and has ≥32 detection channels. The detection channels are distributed in the prefrontal cortex, motor cortex and parietal lobe regions according to the 10-20 international EEG system standard.
3. The infrared brain-computer interface method with multimodal signals according to claim 1 or 2, characterized in that: The EEG signals in step S1 are acquired through a high-density EEG electrode array, which comprises 64-128 electrodes. Electrodes, sampling frequency of 1000-2000Hz, input impedance > Common-mode rejection ratio > 120dB.
4. The infrared brain-computer interface method with multimodal signals according to claim 3, characterized in that: The eye movement signal in step S1 is acquired using the corneal reflection method, with a sampling frequency of 250-450Hz.
5. The infrared brain-computer interface method with multimodal signals according to claim 1, characterized in that: The multimodal signals in step S2 include: near-infrared signals, EEG signals, and EOG signals; The preprocessing steps for the near-infrared signal are as follows: R1: Removes bad pixels from the original light intensity signal. The criteria identify defects caused by sudden changes in light intensity and replace them using linear interpolation; R2: Calculated using differential spectroscopy for optical density signals in the 760nm and 850nm bands. , , Concentration change time series; R3: Low-frequency baseline drift of the signal is removed by using sliding window polynomial fitting; The preprocessing steps for the EEG signal are as follows: T1: Uses a Butterworth function of 0.5-80Hz to remove DC drift and high-frequency noise; uses an adaptive notch filter of 50Hz and its harmonics to remove power frequency interference; T2: Employs full-channel average reference to eliminate common-mode interference; T3: Identify bad channels using the correlation coefficient method and standard deviation method; T4: Based on the event markers of the paradigm, the continuous EEG signal is segmented into signals of a single trial; the time window of each trial is from 100ms before the event to 4000ms after the event, and the signal 150ms before the event is used as the baseline for baseline correction; The preprocessing steps for the EOG signal are as follows: Q1: Use the Butterworth function of 0.1-30Hz to remove baseline drift and high-frequency noise; Q2: Based on the threshold method, identify blink and saccade events, and extract the number of blinks, blink duration, saccade amplitude, saccade frequency, and fixation time ratio for processing.
6. The infrared brain-computer interface method with multimodal signals according to claim 1, characterized in that: The adaptive filtering algorithm in step S2 includes: a dynamic noise suppression module based on Kalman filtering and an artifact separation module based on independent component analysis; the dynamic noise suppression module based on Kalman filtering employs a state equation and an observation equation, wherein the state equation is... The observation equation is ,in, Here is the state transition matrix. For the observation matrix, and These are process noise and observation noise, respectively.
7. The infrared brain-computer interface method with multimodal signals according to claim 1, characterized in that: The attention-based multimodal feature fusion network includes: The feature extraction layer extracts features from the infrared spectral signal, electroencephalogram signal, and eye movement signal. The self-attention layer performs self-attention calculation on the features of each modality to obtain key features within the modality. The attention weights are calculated using the Softmax function. A cross-attention layer is used to establish an attention interaction mechanism between modalities, and to achieve the association and fusion of features from different modalities through the attention weight matrix between modalities. The fusion output layer maps the multimodal features processed by self-attention and cross-attention into a joint feature vector through a concatenation operation and a fully connected layer.
8. The infrared brain-computer interface method with multimodal signals according to claim 1, characterized in that: The deep belief network in step S4 includes a pre-training stage and a fine-tuning stage; the pre-training stage uses the contrastive divergence algorithm; the fine-tuning stage uses the backpropagation algorithm combined with the stochastic gradient descent algorithm with momentum for supervised parameter optimization.
9. The infrared brain-computer interface method with multimodal signals according to claim 1, characterized in that: The reinforcement learning algorithm in step S5 is a deep deterministic policy gradient algorithm, which uses feature fusion weights and network parameters as the action space, and uses decoding accuracy, response time and user fatigue as the reward function.