Multi-modal brain-computer interface signal processing method and system based on timing alignment and feature decoupling

Abstract

The application discloses a brain-computer interface signal processing method and system based on time sequence alignment and feature decoupling. In view of the problems of time asynchronization of electroencephalogram and near-infrared signal and feature entanglement caused by neurovascular coupling in the existing hybrid brain-computer interface, the application firstly extracts multi-scale features from the electroencephalogram signal and extracts multi-delay candidate feature sequences from the near-infrared signal; secondly, uses the electroencephalogram features as a query to guide a time delay attention mechanism to adaptively align the hemodynamic response features; then, projects the aligned features into shared and private subspaces through a multiple-constrained neural feature subspace decoupling module to separate the cross-modal neural intention and private physiological noise; finally, outputs a classification result through adaptive gating and multi-head decision network fusion. The application effectively overcomes the individual differences of neurovascular delay, strips the modality-specific noise, and improves the decoding accuracy and robustness.

Description

Technical Field

[0001] This invention belongs to the field of biomedical signal processing and brain-computer interface technology, specifically relating to a brain-computer interface signal processing method and system based on multimodal temporal alignment and feature decoupling. Background Technology

[0002] Brain-computer interfaces (BCIs) bypass traditional peripheral neuromuscular pathways, enabling direct communication between the brain and external devices. However, single-modal BCI systems struggle to simultaneously capture transient electrophysiological oscillations and detailed hemodynamic distributions, resulting in inherent spatiotemporal information loss when reconstructing complex cognitive processes.

[0003] To overcome the limitations of single-sensor technology, a hybrid BCI architecture that integrates non-invasive electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) has become a highly promising solution. EEG provides millisecond-level temporal resolution, enabling precise targeting of neural oscillation patterns for specific tasks; while fNIRS provides high spatial accuracy by monitoring changes in the concentrations of oxyhemoglobin and deoxyhemoglobin.

[0004] Although EEG and fNIRS are theoretically highly complementary, existing fusion paradigms face two major technical bottlenecks in practical decoding tasks:

[0005] First, there is the issue of temporal asynchrony. Due to the neurovascular coupling (NVC) mechanism, there is an inherent time delay between rapid EEG neural activity and the slow fNIRS hemodynamic response. Early fusion frameworks typically used a fixed time offset to compensate for this delay. However, the efficiency of neurovascular coupling varies significantly among different subjects and in different cortical regions (such as the primary motor cortex and the prefrontal cortex). Fixed time compensation strategies cannot adapt to these complex dynamic changes, leading to cross-modal temporal misalignment.

[0006] Secondly, there is the problem of feature representation entanglement. Temporal asynchrony further exacerbates the entanglement of feature representations. Existing deep learning fusion methods typically employ simple feature concatenation strategies, lacking effective decoupling mechanisms. This results in shared neural intents being mixed with modality-specific physiological artifacts and noise in the latent feature space, making it impossible for the system to effectively distinguish between task-related signals and interference noise, severely limiting the complementary advantages of multimodal information. Summary of the Invention

[0007] To address the shortcomings of existing technologies, this invention provides a brain-computer interface signal processing method and system based on temporal alignment and feature decoupling, in order to solve the temporal misalignment problem caused by neurovascular coupling delay in hybrid brain-computer interfaces, as well as the noise entanglement and information redundancy problems caused by direct feature fusion.

[0008] This invention provides a multimodal brain-computer interface signal processing method based on temporal alignment and feature decoupling, comprising the following steps:

[0009] Step 1: Acquire multi-channel EEG signals and functional near-infrared spectral signals of the subject; extract instantaneous EEG features by multi-scale temporal feature extraction of the EEG signals; divide the functional near-infrared spectral signals into candidate windows according to a preset time delay step and extract features to generate a near-infrared candidate feature sequence containing multiple time delay stages.

[0010] Step 2: Using the instantaneous EEG features as the query benchmark and the near-infrared candidate feature sequence as the key-value pair, input the time delay attention module to perform cross-attention calculation, so as to achieve time adaptive alignment between hemodynamic response features and EEG neural activity, and obtain the aligned near-infrared features.

[0011] Step 3: Project the aligned near-infrared features and the transient EEG features to the shared feature subspace and the private feature subspace respectively, and separate the common neural intention features and the unique private physiological features of the two modalities through a multi-constraint mechanism to obtain a decoupled feature set;

[0012] Step 4: After adjusting the decoupled feature set through an adaptive gating mechanism, input it into a multi-head decision fusion network for dynamic quality-aware weighted fusion, and output the brain-computer interface intent classification result.

[0013] Furthermore, the multi-scale temporal feature extraction of the EEG signal in step one specifically includes: using parallel multi-scale deep separable convolutions with different receptive field sizes to capture EEG rhythm features of different frequency bands related to a specific cognitive task, and then introducing learnable positional coding after splicing to obtain the instantaneous EEG features.

[0014] The extraction of features from the functional near-infrared spectral signal specifically includes: generating multiple time-shifted versions of near-infrared signal segments according to a fixed time delay step, extracting blood oxygen concentration change features through one-dimensional convolution, and generating the near-infrared candidate feature sequence.

[0015] Furthermore, the cross-attention calculation in step two specifically includes: mapping the instantaneous EEG features to a query matrix, mapping the near-infrared candidate feature sequence to a key matrix and a value matrix; generating a delayed attention map by calculating the dot product similarity between the query matrix and the key matrix, and performing a weighted summation on the value matrix based on the delayed attention map to obtain the aligned near-infrared features that best match the instantaneous EEG dynamics.

[0016] Furthermore, during the model training phase, Pearson correlation coefficient loss is introduced as an auxiliary supervision for the time-delay attention module to constrain the neurovascular coupling consistency between the aligned near-infrared features and EEG features.

[0017] Furthermore, in step three, four independent multilayer perceptron encoders are used to map the instantaneous EEG features into shared EEG features and private EEG features, respectively, and to map the aligned near-infrared features into shared near-infrared features and private near-infrared features.

[0018] Furthermore, the aforementioned multiple constraint mechanisms include neural consensus constraints, modality-specific constraints, and functional separation constraints:

[0019] The neural consensus constraint is used to align the shared EEG features and the shared near-infrared features from the same batch of samples to extract cross-modal consistent semantic information;

[0020] The modality-specific constraint suppresses environmental fluctuation interference by injecting Gaussian noise perturbations into the EEG private features and near-infrared private features respectively and maximizing the feature similarity before and after the perturbation.

[0021] The functional separation constraint forces the shared feature subspace and the private feature subspace to be orthogonally separated by minimizing the cross-correlation norm between shared features and private features.

[0022] Furthermore, the adaptive gating mechanism in step four specifically includes: inputting each feature in the decoupled feature set into the residual gating unit, and independently adjusting the response intensity of each feature dimension through learnable gating weights to strengthen discriminative features and suppress irrelevant dimensions;

[0023] The multi-head decision fusion network specifically includes: inputting the adjusted features into independent classification prediction heads, dynamically generating quality-aware weights for each classification prediction head through a lightweight network, and weighting and summing the output probabilities of each classification prediction head to obtain the final brain-computer interface intent classification result.

[0024] This invention also provides a multimodal brain-computer interface signal processing system based on temporal alignment and feature decoupling, comprising:

[0025] The multimodal feature encoding module is used to acquire multi-channel EEG signals and functional near-infrared spectral signals of the subjects, and to perform multi-scale temporal feature extraction and multi-delay candidate feature extraction.

[0026] A time-delayed attention alignment module, connected to the multimodal feature encoding module, is used to achieve temporal adaptive alignment of near-infrared hemodynamic response features and brain electrical neural activity through a cross-attention mechanism;

[0027] The neural feature subspace decoupling module, connected to the time-delay attention alignment module, is used to project the aligned features to the shared and private subspaces, and separate neural intent features from private physiological features through multiple constraints.

[0028] An adaptive multi-head decision fusion module, connected to the decoupled module, is used to output brain-computer interface intent classification results through adaptive gating and dynamic quality perception weighted fusion.

[0029] The present invention also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method steps provided by the present invention.

[0030] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method steps provided by the present invention.

[0031] Compared with existing hybrid brain-computer interface fusion methods, the present invention has the following advantages:

[0032] First, this invention overcomes the limitations of neurovascular delay to achieve data-driven dynamic temporal alignment. It utilizes fast EEG dynamics as an "anchor point" to adaptively synchronize the slow fNIRS response, effectively compensating for the varying neurovascular coupling delays between different subjects and tasks. Furthermore, the learned attention weights highly match the actual hemodynamic response function (HRF), demonstrating strong neurophysiological interpretability.

[0033] Secondly, a triple-constrained subspace decoupling is constructed to effectively remove mode-specific noise. This invention, through orthogonal decomposition of shared and private subspaces, effectively eliminates physiological fluctuations unrelated to intent, removes information redundancy caused by simple splicing, and improves the discriminative power of features.

[0034] Finally, the robustness of the system under complex environments is improved. The adaptive gating mechanism simulates the gain control of the biological cortex, dynamically allocating decision weights based on the signal quality of each branch, and exhibits extremely high stability when the signal quality fluctuates.

[0035] In summary, this invention constructs an end-to-end brain-computer interface signal processing framework consisting of "neurophysiological feature encoding - temporal adaptive alignment - subspace feature decoupling - dynamic quality fusion". Starting from prior knowledge of neurophysiology, this framework effectively overcomes the two core challenges inherent in bimodal brain-computer interfaces: the difference in neurovascular coupling delay and modal feature entanglement. It achieves high-precision decoding of the subject's true neural intent even in complex interference environments. This invention not only significantly improves the model's generalization ability and system robustness across different subject scenarios but also provides stable, reliable, and refined technical support for practical brain-computer interaction applications such as neurorehabilitation training, barrier-free assisted communication, and neurofeedback therapy.

[0036] To verify the effectiveness of this invention, it was systematically and scientifically validated on the publicly available hybrid EEG-fNIRS bimodal dataset. Experiments show that this invention significantly outperforms several existing mainstream multimodal fusion baseline models in intent classification for multiple cognitive tasks, including motor imagery (MI), mental arithmetic (MA), and word generation (WG). Furthermore, the temporal alignment weights learned by the model highly match the real hemodynamic response function (HRF), endowing this deep learning framework with neurophysiological interpretability. This framework not only demonstrates strong potential in the field of brain-computer interface intent parsing, but its core ideas of adaptive temporal synchronization and multi-feature decoupling can also be generalized and extended to other multimodal physiological signal processing scenarios with temporal asynchrony and noise redundancy, showing broad prospects for clinical translation and engineering applications. Attached Figure Description

[0037] Figure 1 This is a schematic diagram of the overall framework structure of the method of the present invention;

[0038] Figure 2 This is a schematic diagram of the time-delay attention module structure;

[0039] Figure 3 A schematic diagram of the neural feature subspace decoupling module structure;

[0040] Figure 4 This is a comparison chart of the temporal adaptive alignment weight distribution of the present invention under three brain-computer interface tasks. Detailed Implementation

[0041] The embodiments of the present invention will be described in detail below with reference to the figures and tables: This embodiment is implemented under the premise of the technical solution of the present invention, and provides detailed implementation methods and specific operation processes.

[0042] like Figure 1As shown, this application provides a brain-computer interface signal processing method based on temporal alignment and feature decoupling. Its overall architecture encompasses multimodal feature encoding, temporal delay attention (TDAM) alignment, neural feature subspace decoupling (NFSD), and a fusion classifier, specifically including the following steps:

[0043] Step 1: Multimodal feature encoding.

[0044] Combination Figure 1 As shown in the encoding module on the left, for EEG data preprocessing, this application selects to use effective EEG signals after 0.5-50Hz bandpass filtering and amplitude normalization to retain task-related neural rhythms and eliminate high-frequency electromyographic noise and low-frequency drift. The preprocessed EEG signal is input into a multi-scale rhythm-aware encoder. First, spatial features are mixed using 1×1 convolution, and then temporal features are extracted using parallel multi-scale depthwise separable convolution. Furthermore, the multi-scale rhythm-aware encoder sets up parallel branches with convolution kernel sizes of 13, 25, 51, and 101, corresponding to temporal receptive fields of 65ms, 125ms, 255ms, and 505ms, respectively, to capture rhythmic features related to cognitive tasks. Finally, adaptive average pooling and positional encoding are performed to obtain instantaneous EEG features. .

[0045] For fNIRS signals, the Beer-Lambert law was first corrected to convert changes in oxygenated (HbO) and deoxygenated (HbR) hemoglobin concentrations, and 0.01–0.1 Hz bandpass filtering and baseline correction were applied to suppress physiological noise. To explicitly model the neurovascular coupling delay, a time delay step size was used. Multiple time-shifted sequences are generated in 0.5s.

[0046] No. Each candidate delay window is defined as:

[0047] (1)

[0048] in, The window length is defined. Subsequently, one-dimensional convolution and positional encoding are used to generate near-infrared candidate feature sequences containing multiple temporal delay stages. .

[0049] Furthermore, to address the differences in cortical hemodynamic response delays for different brain-computer interface tasks, the time delay span and number of candidate windows are adaptively configured. Specifically, for motor imagery tasks, a first number of candidate windows is configured to cover shorter response delay periods; for mental arithmetic or word generation tasks, a second number of candidate windows is configured to cover longer response delay periods, and the second number is greater than the first number.

[0050] The purpose of this step is to construct a high-quality basic feature space for cross-modal fusion. The principle behind this is to respect the heterogeneous physical priors of bimodal signals. Specifically, for the high-frequency dynamic characteristics of EEG signals, multi-scale receptive fields are used to efficiently capture brain rhythms at different frequency bands; for the slow metabolic characteristics of near-infrared signals, implicit time-shift candidate windows are explicitly generated, visualizing the invisible time shift as a spatial sequence, thus providing a complete retrieval and matching space for subsequent modules.

[0051] Step 2: Temporal alignment based on time-delay attention.

[0052] like Figure 2 The diagram shows the structure of the time-delay attention module. This module uses instantaneous EEG features as the time query anchor point to adaptively retrieve the most relevant fNIRS blood flow response. An asymmetric cross-modal attention mechanism maps the two modalities to a common latent space.

[0053] (2)

[0054] In the formula, This is a query matrix generated from transient EEG feature mapping. and These are the key matrix and value matrix generated by mapping near-infrared candidate feature sequences, respectively. , , These are the corresponding learnable projective weight matrices.

[0055] Furthermore, multi-head scaled dot product attention is used to calculate cross-modal correlations and perform weighted summation. The calculation method is as follows:

[0056] (3)

[0057] In the formula, As a feature dimension, a delayed attention map is generated by calculating the similarity between the query and the key, and then the values ​​are weighted and summed.

[0058] To guide the model in learning time-aligned representations that conform to physiological patterns, Pearson correlation coefficient loss is introduced. As an auxiliary constraint:

[0059]

[0060] In the formula, This indicates the transient EEG characteristics. This represents the aligned near-infrared feature; This represents a function for calculating the covariance between and . and These represent transient EEG characteristics. and aligned near-infrared features Standard deviation; This indicates the absolute value operation, to accommodate the bidirectional changing characteristics of neurovascular coupling.

[0061] The core function of this step is to achieve high-precision cross-modal time synchronization. The principle is to utilize an asymmetric cross-attention mechanism, using EEG features representing "instantaneous neural states" as time anchors, to perform a data-driven soft search of the hemodynamic response fragments with the strongest correlation in near-infrared multi-delay candidate sequences. This mechanism cleverly transforms temporal delay alignment into an attention weight allocation process.

[0062] This step explicitly addresses the technical problem pointed out in the background section: "fixed time compensation strategies cannot adapt to such complex dynamic changes, leading to cross-modal time misalignment." By dynamically allocating attention weights, the traditional static fixed delay assumption is abandoned, achieving adaptive dynamic temporal alignment between the task and the individual.

[0063] Step 3: Decoupling of the neural feature subspace.

[0064] like Figure 3 The diagram shows the structure of the neural feature subspace decoupling module. The aligned global features are input into four independent multilayer perceptron (MLP) encoders, mapped to a dimension of... By dividing the shared and private subspaces, shared EEG features and near-infrared shared features were obtained. , With their respective private characteristics , .

[0065] Combination Figure 3 The internal logic of the neural feature subspace in the middle is used to construct a triple-constraint optimization objective to achieve strict decoupling:

[0066] (1) Neural consensus constraints ): Used to enforce cross-modal semantic consistency alignment, its calculation formula is:

[0067]

[0068] In the formula, For cosine similarity, For temperature parameters, For batch size, and For sample indexes within a batch; Indicates the first Shared EEG feature vectors for each sample; and They represent the first The and the first Near-infrared shared feature vectors of each sample; and These represent the exponential function and the logarithmic function, respectively.

[0069] (2) Modal-specific constraints By injecting Gaussian noise, the similarity of private features before and after the perturbation is maximized to suppress fluctuation-sensitive noise.

[0070]

[0071] in, and They represent the first Initial EEG and near-infrared proprietary features of each sample; and These represent the EEG private features and near-infrared private features corresponding to the injection of Gaussian noise perturbation, respectively.

[0072] (3) Functional separation constraints Minimize the cross-correlation norm of the shared and private feature matrices to force subspace orthogonality:

[0073] (7)

[0074] In the formula, and These represent the shared EEG feature matrix and the private EEG feature matrix for all samples within the batch, respectively. and These represent the shared near-infrared feature matrix and the private near-infrared feature matrix of all samples within the batch, respectively. This represents the matrix transpose operation; The square of the Frobenius norm of the matrix is ​​given by . and These represent the dimensionality of the shared feature subspace and the private feature subspace, respectively.

[0075] The purpose of this step is to refine discriminative features and eliminate feature entanglement and information redundancy between modalities. The principle is to combine contrastive learning with soft orthogonal constraints to decouple entangled high-dimensional features: neural consensus constraints allow the shared subspace to focus on extracting consistent neural intent across modalities; modality-specific constraints filter out fluctuation artifacts; and functional separation constraints force subspaces to be orthogonal, allowing private subspaces to focus on preserving modality-specific physiological markers.

[0076] This step clearly addresses the technical problem of "lack of an effective decoupling mechanism" pointed out in the background section. It successfully cuts off the cross-modal propagation path of single-modal physiological artifacts, avoiding the mixed entanglement of shared neural intent and modality-specific noise in the latent feature space.

[0077] Step 4: Adaptive multi-head decision quality perception fusion.

[0078] Recombined Figure 1 As shown in the fusion decision section on the right, the decoupled features are input into the residual gating unit to dynamically adjust the response intensity of each dimension:

[0079]

[0080] in For the gated weight vector, For MLP transformation, For Hadama accumulation.

[0081] The adjusted enhanced features are input into independent classification prediction heads, and fusion weights are dynamically generated based on feature quality through a lightweight network:

[0082]

[0083] In the formula, Indicates the first Normalized fusion weights for each classification prediction head; and They represent the first One and The first is a lightweight weighted network consisting of a multilayer perceptron and activation functions; and These represent the inputs to the first... The and the first Gated enhancement features for each classification prediction head; The denominator represents the exponential function; the denominator represents the Softmax normalization of the confidence scores of the three independent prediction branches.

[0084] By using this weight to sum the output probabilities of each predictor, dynamic decision-making at the sample level is achieved, resulting in the final brain-computer interface intent classification result.

[0085] The purpose of this step is to improve the system's robustness against interference and decision confidence in complex interference environments. Its principle is to simulate the gain control mechanism of the cerebral cortex, preserving discriminative dimensions and suppressing irrelevant dimensions at the feature level; and dynamically allocating weights based on the quality of real-time signals at the decision level, ensuring that the system can still operate smoothly even when signal attenuation occurs in a single mode.

[0086] By combining the above steps in a coordinated manner, this step finally overcame the decoding bottleneck in complex mixed signal environments and successfully achieved the core technical goal of "high-precision decoding" of the subject's true neural intentions.

[0087] Based on the same concept as the above method, this application also provides a brain-computer interface signal processing system based on temporal alignment and feature decoupling, comprising:

[0088] The multimodal feature encoding module is used to acquire EEG signals and near-infrared signals and extract multi-scale instantaneous features and multi-delay candidate feature sequences, respectively.

[0089] The time-delay attention alignment module is used to use EEG features as a query benchmark and achieve time-adaptive alignment of near-infrared blood flow response features through cross-attention calculation.

[0090] The neural feature subspace decoupling module is used to project aligned features to shared and private subspaces, and separates neural intent from private physiological features through triple constraints;

[0091] The adaptive multi-head decision fusion module is used to dynamically quality-aware weighted fusion of decoupled features after residual gating and output classification results.

[0092] Based on the same concept as the above method, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method steps of this application.

[0093] Based on the same concept as the above method, this application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the method steps of this application.

[0094] In summary, one of the core innovations of this application is the adoption of an architecture that deeply integrates Temporal Delayed Attention (TDAM) and Neural Feature Decoupling (NFSD), which can accurately overcome the neurovascular coupling delay and feature entanglement between EEG and near-infrared signals. Through the scheme of "EEG-guided alignment + triple constraint decoupling", the model can dynamically adapt to the differences in hemodynamic delay among different individuals, and effectively separate cross-modal intent and specific noise using orthogonal subspaces; at the same time, the introduction of a residual gating mechanism improves the robustness of the model in environments with fluctuating signal quality.

[0095] This embodiment:

[0096] The specific implementation process of this embodiment is as follows: Figure 1 The diagram shown is the overall framework and data flow diagram of the signal processing model based on temporal alignment and feature decoupling described in this application. The diagram fully presents the end-to-end core workflow from dual-modal signal input to prediction and classification.

[0097] In the specific dataset processing implementation, a subject-independent five-fold cross-validation strategy was adopted, covering motor imagery (MI), mental arithmetic (MA), and word generation (WG) tasks. The number of candidate delay windows was adaptively configured to address the differences in response latency across different cortical regions: for the MI task activating the primary motor cortex, the configuration was as follows: =13; For MA and WG task configurations that activate the prefrontal cortex =21.

[0098] Combination Figure 1 As shown in the multimodal feature encoding stage on the left and in Table 1, the preprocessed EEG and near-infrared signals are input into parallel feature extraction branches. Table 1 details the evolution of the core modules, convolution kernel size, and output dimensions of each branch of the feature extractor, ensuring strict alignment of feature dimensions.

[0099] Table 1. Details of Feature Extractor Network Parameters

[0100]

[0101] After feature extraction is complete, the data flows to the cross-modal time synchronization stage. For example... Figure 2 The diagram shows the internal tensor computation of the Time Delayed Attention Module (TDAM). This module receives the features output from Table 1, uses multi-scale EEG features as the query variable (Q), and triggers cross-attention computation on near-infrared candidate sequences (K, V). Figure 2 The matrix heatmap on the right visually illustrates the dynamically generated "delayed attention map" used by the model to guide the network in adaptively retrieving the most relevant blood flow responses and outputting aligned features. Meanwhile, the dashed lines in the graph indicate... The introduced Pearson correlation coefficient loss is used to constrain the alignment features to conform to the bidirectional variation characteristics of neurovascular coupling.

[0102] Subsequently, to eliminate information redundancy caused by cross-modal feature concatenation, the aligned features enter the decoupling stage. For example... Figure 3 The diagram shows the specific computational architecture of the Neural Feature Subspace Decoupling Module (NFSD). The left and right sides connect to the feature encoding and the final fusion layer, respectively. The large dashed box in the middle details the mechanism of the triple constraint: the private feature flows on the top and bottom sides are introduced with Gaussian noise. Modal specific constraints The shared features in the middle flow through neural consensus constraints. The orthogonal symbols running throughout symbolize functional separation constraints. Forced subspace separation is used to completely remove physiological noise artifacts.

[0103] go through Figure 3After decoupling and splicing, the features are finally input. Figure 1 The right side represents the adaptive multi-head decision-making quality-aware fusion stage. Decoupled features are first processed by a residual gating unit to adjust their response intensity, then input into multiple independent classification prediction heads. A lightweight network dynamically generates normalized fusion weights, such as... Figure 1 In , , As shown, high-precision intent classification is achieved after weighted summation.

[0104] During model training, the overall optimization objective combines labeled smooth cross-entropy classification loss with auxiliary alignment and decoupling losses, with weight parameters set as follows: , , .

[0105] Table 2 shows a comparison of the classification accuracy of the embodiments in this application with baseline models such as FGANet, fNIRSNet, and M2NN across trials:

[0106] Table 2. Comparison of classification accuracy with mainstream baseline models

[0107]

[0108] To further verify the necessity of the core technical features in this embodiment, such as the time-delay attention alignment mechanism, the neural feature subspace decoupling mechanism, and the adaptive decision fusion mechanism, and their beneficial effects on the overall decoding performance, this embodiment designed a multi-dimensional ablation experiment. The specific results are shown in Table 3.

[0109] Table 3. Comparison of classification performance of the embodiments in this application with various ablation variants.

[0110]

[0111] In the table, all the metrics results of this application embodiment are optimal. To intuitively reveal the physical meaning of the model's temporal alignment mechanism, this embodiment visualizes the time-delay attention weight distribution automatically learned by the model in the test set, such as... Figure 4 As shown. By Figure 4 A comparison of the three subgraphs shows that the delayed attention weight distribution learned by the model closely matches the true hemodynamic response (HRF) pattern of the cerebral cortex: such as Figure 4 As shown in the MI task subplot above, due to the high vascular density in the primary motor cortex activated by the motor imagery task, its weight exhibits an early single peak of 4.5-5.5 seconds; Figure 4As shown in the middle MA task subplot and the lower WG task subplot, the mental arithmetic and vocabulary generation tasks primarily activate the prefrontal cortex, with their peak weights delayed to the right to 6.0-9.0 seconds and 6.0-8.0 seconds, respectively. This phenomenon fully confirms the neurophysiological rationality of the model's temporal alignment mechanism.

[0112] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A brain-computer interface multimodal signal processing method based on temporal alignment and feature decoupling, characterized in that, Includes the following steps: Step 1: Acquire multi-channel EEG signals and functional near-infrared spectral signals of the subject; extract instantaneous EEG features by multi-scale temporal feature extraction of the EEG signals; divide the functional near-infrared spectral signals into candidate windows according to a preset time delay step and extract features to generate a near-infrared candidate feature sequence containing multiple time delay stages. Step 2: Using the instantaneous EEG features as the query benchmark and the near-infrared candidate feature sequence as the key-value pair, input the time delay attention module to perform cross-attention calculation, so as to achieve time adaptive alignment between hemodynamic response features and EEG neural activity, and obtain the aligned near-infrared features. Step 3: Project the aligned near-infrared features and the transient EEG features to the shared feature subspace and the private feature subspace respectively, and separate the common neural intention features and the unique private physiological features of the two modalities through a multi-constraint mechanism to obtain a decoupled feature set; Step 4: After adjusting the decoupled feature set through an adaptive gating mechanism, input it into a multi-head decision fusion network for dynamic quality-aware weighted fusion, and output the brain-computer interface intent classification result.

2. The brain-computer interface signal processing method based on temporal alignment and feature decoupling according to claim 1, characterized in that, Step one involves extracting multi-scale temporal features from the EEG signal, specifically including: using parallel multi-scale deep separable convolutions with different receptive field sizes to capture EEG rhythm features of different frequency bands related to a specific cognitive task, and then introducing learnable positional coding after splicing to obtain the instantaneous EEG features. The extraction of features from the functional near-infrared spectral signal specifically includes: generating multiple time-shifted versions of near-infrared signal segments according to a fixed time delay step, extracting blood oxygen concentration change features through one-dimensional convolution, and generating the near-infrared candidate feature sequence.

3. The brain-computer interface signal processing method based on temporal alignment and feature decoupling according to claim 1 or 2, characterized in that, The cross-attention calculation in step two specifically includes: mapping the instantaneous EEG features to a query matrix, mapping the near-infrared candidate feature sequence to a key matrix and a value matrix; generating a delayed attention map by calculating the dot product similarity between the query matrix and the key matrix; and performing a weighted summation on the value matrix based on the delayed attention map to obtain the aligned near-infrared features that best match the instantaneous EEG dynamics.

4. The brain-computer interface signal processing method based on temporal alignment and feature decoupling according to claim 3, characterized in that, During the model training phase, Pearson correlation coefficient loss is introduced as an auxiliary supervision for the time-delay attention module to constrain the neurovascular coupling consistency between the aligned near-infrared features and EEG features.

5. The brain-computer interface signal processing method based on temporal alignment and feature decoupling according to claim 1, characterized in that, In step three, four independent multilayer perceptron encoders are used to map the instantaneous EEG features into shared EEG features and private EEG features, respectively, and to map the aligned near-infrared features into shared near-infrared features and private near-infrared features.

6. The brain-computer interface signal processing method based on temporal alignment and feature decoupling according to claim 5, characterized in that, The multiple constraint mechanisms include neural consensus constraints, modality-specific constraints, and functional separation constraints: The neural consensus constraint is used to align the shared EEG features and the shared near-infrared features from the same batch of samples to extract cross-modal consistent semantic information; The modality-specific constraint suppresses environmental fluctuation interference by injecting Gaussian noise perturbations into the EEG private features and near-infrared private features respectively and maximizing the feature similarity before and after the perturbation. The functional separation constraint forces the shared feature subspace and the private feature subspace to be orthogonally separated by minimizing the cross-correlation norm between shared features and private features.

7. The brain-computer interface signal processing method based on temporal alignment and feature decoupling according to claim 1, characterized in that, The adaptive gating mechanism in step four specifically includes: inputting each feature in the decoupled feature set into the residual gating unit, and independently adjusting the response intensity of each feature dimension through learnable gating weights to strengthen discriminative features and suppress irrelevant dimensions; The multi-head decision fusion network specifically includes: inputting the adjusted features into independent classification prediction heads, dynamically generating quality-aware weights for each classification prediction head through a lightweight network, and weighting and summing the output probabilities of each classification prediction head to obtain the final brain-computer interface intent classification result.

8. A brain-computer interface signal processing system based on temporal alignment and feature decoupling, characterized in that, include: The multimodal feature encoding module is used to acquire multi-channel EEG signals and functional near-infrared spectral signals of the subject, extract instantaneous EEG features by multi-scale temporal feature extraction of the EEG signals, divide the functional near-infrared spectral signals into candidate windows according to a preset time delay step and extract features to generate a near-infrared candidate feature sequence containing multiple time delay stages. The time-delay attention alignment module is connected to the multimodal feature encoding module. It is used to take the instantaneous EEG features as the query benchmark and the near-infrared candidate feature sequence as the key-value pair, and input them into the time-delay attention module for cross-attention calculation to realize the time-adaptive alignment of hemodynamic response features and EEG neural activity to obtain aligned near-infrared features. The neural feature subspace decoupling module, connected to the time-delay attention alignment module, is used to project the aligned near-infrared features and the instantaneous EEG features to the shared feature subspace and the private feature subspace, respectively, and to separate the neural intention features common to the two modalities and the unique private physiological features through a multi-constraint mechanism to obtain a decoupled feature set. An adaptive multi-head decision fusion module, connected to the neural feature subspace decoupling module, is used to adjust the decoupled feature set through an adaptive gating mechanism, input it into the multi-head decision fusion network for dynamic quality-perception weighted fusion, and output the brain-computer interface intent classification result.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 7.

Images