A method for non-invasive brain-computer interface fine motor imagery decoding

Abstract

The system and method for non-invasive brain-computer interface fine motor imagination decoding belong to the field of deep learning. First, a data enhancement method based on a generative adversarial network is used to generate enhanced samples that maintain the original electroencephalogram signal and functional near-infrared spectroscopy signal joint feature distribution. Second, according to the data characteristics of the two modal signals, a representation learning method based on a convolutional neural network is used, and the latent feature distributions of the two are constrained to be close by minimizing the KL divergence between the two modal feature distributions. Third, a cross-modal multi-head attention fusion module guided by electroencephalogram data features is used to dynamically select and fuse the complementary features of the modes. Finally, the fused features pass through a classifier to predict the fine motor intention contained in the physiological signal data of the current trial.

Description

Technical Field

[0001] This invention, based on deep learning technology, studies a non-invasive brain-computer interface (BCI) system for decoding fine motor imagery using dual-modal signals from electroencephalography (EEG) and functional near-infrared spectroscopy (FIR). First, a data augmentation method based on generative adversarial networks (GANs) is used to generate augmented samples that maintain the joint feature distribution of the original EEG and FIR signals. Second, based on the data characteristics of each modality, a representation learning method based on convolutional neural networks is employed, and the potential feature distributions of the two modalities are constrained to be similar by minimizing the KL divergence between them. Third, a cross-modal multi-head attention fusion module guided by EEG data features is used to dynamically select and fuse complementary features from both modalities. Finally, the fused features are processed by a classifier to predict the fine motor intention implied by the physiological signal data in the current trial. This invention belongs to the field of non-invasive brain-computer interface signal processing, specifically involving deep learning, data augmentation, and modality fusion technologies. Background Technology

[0002] Brain-computer interfaces (BCIs) establish a direct connection between neural activity and external devices by measuring and interpreting physiological signals generated by the brain. Among the various applications of BCI, motor imagery (MI) is widely used because it can measure and estimate voluntary motor intentions by decoding motor imagery electroencephalography (MI-EEG) signals. MI-EEG signals are typically divided into five frequency bands: delta (0.5–4 Hz), theta (4–7 Hz), alpha (8–13 Hz), beta (14–30 Hz), and gamma (31–100 Hz). During MI tasks, event-related desynchronization (ERD) and event-related synchronization (ERS) phenomena are commonly observed in the alpha and beta bands; these are important indicators reflecting neural activity related to motor intentions. Currently, much of the EEG-based MI-BCI research focuses on pattern measurement and differentiation between simple limb motor imagery (SLMI) tasks, such as left-hand, right-hand, and foot movements. However, some motor imagery tasks appear unnatural in practical applications when the imagined limb movements are spatially inconsistent with the movements of the external output device. This mismatch reduces the reliability and effectiveness of motor intention measurement, leading to a discrepancy between user intent and device response.

[0003] To alleviate this mismatch problem, recent research has increasingly shifted towards decoding Fine Joint Motor Imagery (FJMI), which involves visualizing the movements of different joints within the same limb. Unlike traditional SLMI tasks, FJMI tasks involve activation patterns in more similar brain regions, making them more difficult to distinguish. For example, Ofner et al. extracted low-frequency time-domain EEG signals for decoding upper limb movements, achieving a best-in-class accuracy of only 27% in a six-category task (elbow flexion / extension, forearm pronation / supination, hand clenching / opening). However, the limited spatial resolution of EEG restricts decoding performance in FJMI tasks, prompting researchers to introduce multimodal signals to improve the ability to distinguish spatial patterns.

[0004] To overcome the inherent limitations of EEG, an increasing number of researchers are introducing functional near-infrared spectroscopy (fNIRS) signals as supplementary information. fNIRS detects changes in brain blood flow using near-infrared light, thereby monitoring hemodynamic responses. Compared to EEG, it provides higher spatial accuracy for deeper brain regions and has stronger resistance to motion artifacts. Therefore, utilizing the complementary characteristics of EEG and fNIRS signals can effectively compensate for the spatial resolution deficiencies of EEG. Given the core role of multimodal fusion in hybrid BCI systems, existing fusion strategies are generally divided into decision-level fusion and feature-level fusion. Decision-level fusion classifies each modality independently and then combines their predictions to obtain the final output. For example, Wang et al. proposed a decision-level fusion network that fuses EEG and fNIRS outputs based on the classification accuracy of each modality as weights, making decision integration more reliable and thus improving overall performance. However, this method ignores the dependencies between modalities, limiting the potential for performance improvement. In contrast, feature-level fusion integrates information after feature extraction. Cooney et al. proposed an EEG–fNIRS hybrid BCI system for speech decoding, in which two independent convolutional networks extract modal features respectively, and then the concatenated representation is input into a gated recurrent unit (GRU) layer for processing. However, simple feature concatenation limits cross-modal interaction because it simply merges the EEG and fNIRS representations directly without considering their dependencies. This may introduce irrelevant information and lead to unreasonable allocation of modal contribution weights, thereby weakening the utilization of their complementary properties.

[0005] From an instrumentation and measurement perspective, FJMI decoding is not merely a pattern recognition problem, but also a problem of measuring and estimating subtle physiological states based on multimodal biosignals. In this context, the goal is to reliably measure neural activity related to motor intention under realistic perceptual conditions (such as low signal-to-noise ratio, limited spatial resolution, and cross-modal temporal mismatch). EEG measurements are limited by volume conduction effects and insufficient spatial specificity, while fNIRS measurements suffer from hemodynamic delays and limited temporal resolution. These factors introduce significant uncertainty and instability into the estimation of fine motor intentions. Therefore, an effective multimodal measurement strategy should explicitly address measurement uncertainties and cross-modal alignment issues while fNIRS complementary information is fused. Within this framework, this paper aims to improve the accuracy, stability, and consistency of multimodal motor imagery estimation, thereby supporting the fusion of multimodal physiological signals in brain-computer interface systems.

[0006] Furthermore, due to the high time and manpower costs required to acquire large amounts of MI-EEG data, the limited data scale remains one of the main reasons for overfitting in deep learning models. To address this issue, researchers have introduced data augmentation (DA) methods in MI decoding to expand the training data scale. Traditional MI-BCI data augmentation methods include techniques such as adding noise and fragment reconstruction. However, because EEG signals are highly sensitive to external interference such as eye movements and electromyography, the original data usually already contains inherent noise. Therefore, even adding additional noise such as white noise may lead to signal distortion, thereby reducing the model training effect. At the same time, there are significant differences in the distribution between EEG and fNIRS signals, and introducing additional noise may further disrupt the alignment between modalities. In contrast, Generative Adversarial Networks (GANs) have been widely used in the BCI field due to their efficiency in data generation and feature learning. However, to our knowledge, there is still a lack of an effective data augmentation method based on GANs that can simultaneously achieve multimodal alignment of EEG and fNIRS.

[0007] To address the aforementioned problems, this invention proposes an EEG-fNIRS bimodal fine motion intention recognition system based on data augmentation and cross-modal fusion techniques. First, a data augmentation model based on a generative adversarial network is trained using the original bimodal data. The trained model generates augmented samples, which, along with the original data, serve as training data for representation learning and modality fusion methods. Second, considering the data characteristics of the two modalities, a multi-temporal parallel convolutional neural network feature extraction module is constructed for the EEG data, and a single-domain aggregated convolutional neural network feature extraction module is designed for the fNIRS data. The potential feature distributions of the two modalities are constrained to be similar by minimizing the KL divergence between their feature distributions. Third, a cross-modal multi-head attention fusion module guided by EEG features dynamically selects and fuses complementary features from both modalities, achieving information complementarity between EEG and fNIRS. Finally, the fused features are processed by a classifier to predict the fine motion intention implied in the physiological signal data of the current trial. Summary of the Invention

[0008] This invention proposes a bimodal decoding method for fine joint motion imagery based on generative adversarial network enhancement and cross-modal attention fusion using electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS). The method first performs data augmentation on the acquired EEG and fNIRS bimodal signals, introducing a multimodal interactive generation mechanism to generate synthetic samples with cross-modal consistency, thereby expanding the training data scale and alleviating the data shortage problem. Subsequently, the enhanced bimodal data are input into a two-branch feature learning network to extract multi-scale temporal features from the EEG signals and aggregate features from the fNIRS signals, aligning the feature distributions of the two modalities by introducing KL divergence constraints. Based on this, an EEG-guided cross-modal attention fusion mechanism is constructed to dynamically weight and interactively model information from different modalities, thereby achieving adaptive fusion of multimodal features. Finally, a classifier is used to discriminate the fused features, achieving accurate decoding of the fine joint motion imagery task. The main process of this method is shown in the appendix. Figure 1 As shown, it can be divided into the following three steps: a bimodal data augmentation method based on generative adversarial networks, a bi-branch representation learning and KL divergence feature aggregation method for bimodal data, and an EEG-guided modality fusion strategy based on cross-modal attention mechanism.

[0009] (1) A dual-modal data augmentation method based on generative adversarial networks

[0010] This invention proposes a generative adversarial network (GAN) enhancement method for EEG and fNIRS bimodal data. By constructing a multimodal interactive conditional GAN, it achieves joint modeling and generation of bimodal data. The method takes random noise and category information as input, and uses a multimodal generator (as shown in the attached diagram) to generate the data. Figure 2 (As shown) Simultaneously generate EEG and fNIRS signals with cross-modal correlation characteristics, and pass them through a multimodal discriminator (as shown in the attached diagram). Figure 3 (As shown) This invention distinguishes and classifies generated data from real data. During training, a multimodal interaction mechanism is introduced, enabling both the generator and discriminator to perceive and model the correlations between different modalities, thereby improving the realism and consistency of the generated data. Simultaneously, to enhance the distribution consistency between generated and real data, a mixed sample construction strategy is employed during training, combining generated and real samples and inputting them into the multimodal discriminator to enhance the stability of adversarial training and reduce mode shift issues. Furthermore, a regularization mechanism based on distribution difference constraints is introduced to ensure that the generated data maintains statistical characteristics consistent with the original data, thus guaranteeing that the generated samples are not only realistic in a single modality but also consistent in cross-modal relationships. Through these methods, the scale of training data can be effectively expanded, the risk of overfitting can be reduced, and a more sufficient and consistent data foundation can be provided for subsequent multimodal feature learning.

[0011] (2) Multi-scale feature aggregation method based on KL alignment

[0012] This invention addresses the differences in temporal resolution and physiological characteristics between EEG and fNIRS signals by designing a dual-branch representation learning structure to achieve efficient extraction and fusion of features from both modalities. In the EEG branch, a multi-scale feature extraction strategy is employed, using a parallel structure to model dynamic information at different time scales, thereby capturing complex temporal features in fine motor imagery tasks. In the fNIRS branch, a single-scale feature aggregation method is used to compress and integrate temporal information related to blood oxygenation changes to obtain discriminative modal feature representations. To mitigate the differences in feature distribution between the two modalities, this invention introduces a feature alignment mechanism based on KL divergence, mapping features extracted from EEG and fNIRS to a unified distribution space and constraining their distribution differences to make the two modalities converge in the representation space. Specifically, by normalizing features and constructing probability distribution representations, the model can actively learn cross-modal consistency relationships during training, thereby enhancing the comparability and complementarity between features. This bi-branch representation learning and KL divergence aggregation method not only fully leverages the feature advantages of each modality, but also achieves cross-modal alignment at the feature level, providing a structured and consistent input representation for the subsequent fusion stage, thereby improving the overall decoding performance.

[0013] (3) EEG-guided modal fusion strategy based on cross-modal attention mechanism

[0014] After completing bimodal feature extraction and alignment, this invention further proposes an EEG-guided cross-modal attention fusion strategy to achieve dynamic modeling and adaptive fusion of multimodal information. First, each modal feature is internally encoded, and long-distance temporal dependencies are modeled using an attention-based structural model to obtain a modal representation containing global contextual information. Then, features from different modalities are uniformly modeled to construct a cross-modal feature interaction space, capturing the temporal and semantic connections between different modalities. Based on this, an EEG-guided cross-modal attention mechanism is employed, using EEG features as the dominant information. Attention weights are used to selectively introduce fNIRS features, dynamically adjusting the contribution of each modality according to task requirements. This mechanism effectively models the conditional dependencies between modalities, transforming the fusion process from static superposition to a context-based dynamic interaction. This fusion strategy not only highlights the dominant role of EEG in motor imagery decoding but also fully utilizes the spatial information supplemented by fNIRS, achieving fine fusion of multimodal information and improving the model's ability to express and discriminate complex brain signals.

[0015] Compared with the prior art, the present invention has the following obvious advantages and beneficial effects:

[0016] First, existing EEG-fNIRS decision-level fusion and simple feature-level fusion methods have the following problems: Decision-level fusion methods typically classify EEG and fNIRS independently before fusing the two classification results. This method does not perform interactive modeling of the features of the two modalities before classification, assuming that the discrimination processes of the two modalities are independent, and cannot fully utilize the complementary relationship between EEG and fNIRS in terms of time, space, and physiological mechanisms. When noise, artifacts, or distribution shifts occur in a certain modality, decision-level fusion can only adjust the weights at the result level and cannot correct them during the feature learning stage, thus limiting the potential performance improvement. For feature-level fusion methods that involve direct feature concatenation or simple weighting, EEG and fNIRS have significant differences in sampling rate, temporal response, spatial distribution, and physiological meaning. EEG directly reflects neural electrical activity and has a fast temporal response; fNIRS reflects changes in blood oxygen dynamics, has hemodynamic delays, and has low temporal resolution. If EEG and fNIRS features are directly concatenated, the model often treats features from different modalities as static, equivalent information. It cannot explicitly model cross-modal dependencies, nor can it adaptively select effective modal information based on different samples, time segments, or tasks. This may introduce redundant and irrelevant information, weakening modal complementarity. To address the problems of existing technologies, this invention offers significant advantages in modality fusion: Unlike traditional decision-level fusion, which simply combines results after classification without modeling intermodal relationships, this invention introduces a cross-modal interaction mechanism during the feature learning stage. This allows EEG and fNIRS to establish connections at the feature level, fully utilizing their complementary information in temporal and spatial resolutions. It also enables early adjustment of noise interference and distribution shifts, improving system robustness. Furthermore, addressing the inability of simple feature concatenation or weighting methods to handle modal differences and dependencies, this invention introduces a feature alignment mechanism based on KL divergence. This maps the two modalities to a unified distribution space, mitigating mismatches caused by differences in sampling rate, temporal response, and physiological characteristics. Further, it combines an EEG-guided cross-modal attention mechanism to dynamically weight and selectively fuse features from different modalities. This allows the model to adaptively adjust modal contributions based on specific samples and tasks, suppressing redundant and irrelevant information and enhancing the expressive power of key features. This achieves a shift from static fusion to dynamic interactive fusion, significantly improving the efficiency of multimodal information utilization and the performance of fine-grained motion imagery decoding.

[0017] Secondly, regarding the shortcomings of existing multimodal data augmentation methods, traditional white noise addition may further contaminate the already noisy EEG signal; fragment reconstruction may disrupt the synchronization relationship between EEG and fNIRS; although ordinary GANs can generate samples, if the joint distribution of EEG-fNIRS and the intermodal alignment relationship are not constrained, the generated multimodal samples may be approximately realistic in a single modality, but cannot maintain the real interaction relationship between EEG and fNIRS. To address the problems of existing multimodal data augmentation methods, this invention offers significant advantages in bimodal data augmentation: Unlike traditional white noise addition methods that may further interfere with the original EEG signal distribution, this invention learns the real data distribution characteristics through a generative model, avoiding the introduction of additional meaningless noise; compared to fragment reconstruction methods that may disrupt the temporal synchronization between EEG and fNIRS, this invention jointly models the two modalities during the generation process, maintaining their inherent temporal correlation and physiological consistency; furthermore, unlike ordinary GANs that only focus on single-modal generation effects while ignoring cross-modal relationships, this invention introduces a multimodal interaction mechanism and distribution alignment constraints, ensuring that the generated data not only possesses realism in its respective modality but also maintains the collaborative change relationship between EEG and fNIRS, thereby generating high-quality samples with cross-modal consistency; therefore, this invention achieves an improvement from "realistic single-modal generation" to "cross-modal consistent generation," effectively enhancing the reliability and usability of augmented data and providing a better data foundation for subsequent multimodal feature learning.

[0018] Experiments have demonstrated that the decoding system constructed using this invention achieves an average classification accuracy of 54.58% for 18 subjects across three categories—hand opening / closing, wrist flexion / extension, and elbow pronation / reversion—on the FineMI fine motor imagery public dataset. This result is at least 6.39% higher than the compared decoding methods. Therefore, applying this invention to decode fine motor intentions in non-invasive brain-computer interfaces can achieve higher decoding accuracy, and has significant application value in fields such as non-invasive brain-computer interface control of external devices and rehabilitation medicine. Attached Figure Description

[0019] Figure 1 Flowchart of a dual-modal fine joint motion imagery decoding system based on generative adversarial network enhancement and cross-modal attention fusion using EEG and near-infrared spectroscopy

[0020] Figure 2 Multimodal interaction generator architecture diagram

[0021] Figure 3 Multimodal interactive discriminator structure diagram

[0022] Figure 4 Dataset experiment time diagram Detailed Implementation

[0023] Based on the above description, the following is a specific implementation process, but the scope of protection of this patent is not limited to this implementation process.

[0024] Step 1: Construction of a bimodal data augmentation method based on generative adversarial networks

[0025] Step 1.1: Construction of the overall structure of the bimodal data augmentation method based on generative adversarial networks

[0026] As attached Figure 1 As shown in section 2, the data augmentation method proposed in this invention consists of two parts: a multimodal generator and a multimodal discriminator. The multimodal generator takes random noise and category information as input to generate EEG-fNIRS data. Subsequently, the generated multimodal signal is fused with the original signal and input together into the multimodal discriminator. Furthermore, the output of the multimodal generator is used to calculate the KL divergence between the generated and original data to align their features. The multimodal discriminator distinguishes between real and generated data and identifies the data category by combining label information. Simultaneously, the discriminator's output is used to calculate the loss function and further update the weights of the generator and discriminator. Notably, both the generator and discriminator promote interaction between different modalities; this design enables the network to better capture multimodal features in the data.

[0027] During training, a distribution mismatch often occurs between generated and real data, and this problem is exacerbated in multimodal scenarios, potentially leading to discriminator overfitting and weakening gradient feedback. To address this issue, we construct hybrid training samples by combining generated and real data. This method helps improve feature alignment, stabilize the adversarial training process, and accelerate the model's convergence to more realistic data. The calculation process is shown in Equation (1).

[0028] (1)

[0029] in, , and These represent the generated multimodal data, the original EEG signal, and the original fNIRS signal, respectively. and These represent Gaussian noise and category labels, respectively. and These represent the synthetic data generated by the generator, which includes EEG and fNIRS features, respectively.

[0030] The samples obtained by interpolating real data and generated data are used as one of the inputs to the multimodal discriminator. Since this study focuses on hybrid FJMI decoding, the interpolated samples of the two modalities are calculated separately, and the calculation method is shown in formula (2).

[0031] (2)

[0032] in, This represents the input to the multimodal discriminator. This represents the random matrix used for interpolation. and These represent the raw EEG and fNIRS data, respectively. and This indicates the corresponding generated data.

[0033] Step 1.2: Multimodal Generator Structure

[0034] The structure of the multimodal generator is shown in the attached figure. Figure 2 As shown, the proposed multimodal generator mainly consists of two parts: the EEG branch and the fNIRS branch. Within each branch, the core structure of the multimodal generator network primarily comprises upsampling layers, convolutional layers, and batch normalization layers. These three layers are combined into a unit called the Generation Block (Gblock), used to progressively enhance the clarity of features and ensure data stability.

[0035] The upsampling layer employs bilinear interpolation to double the input size, thereby improving the spatial resolution of the feature map. This process enables the multimodal generator to progressively recover the spatial structure of the input data and effectively enhance the detail of the generated signal. The convolutional layer achieves cross-channel information exchange through convolution operations and reduces the number of output channels. Simultaneously, it not only extracts local features but also captures complex modal information in the data through multiple convolutional kernels, allowing the generator to learn more latent features. Batch normalization layers are applied to the output of each layer to ensure that the data remains standardized with a mean of 0 and a variance of 1. Batch normalization helps accelerate the training process, alleviate the vanishing gradient problem, stabilize the learning process, and prevent overfitting to some extent.

[0036] In the EEG branch of the multimodal generator, the input high-dimensional features are first reshaped into a 64×4×16 (number of convolutional kernels × number of channels × length) feature matrix to achieve dimensionality reduction. This feature matrix is ​​then progressively expanded through five G-block modules. During this process, the number of convolutional kernels gradually decreases, while the number of channels and the size of the feature map continuously increase, thus better adapting to the characteristics of the EEG signal.

[0037] The structure of the fNIRS branch in the multimodal generator is similar to that of the EEG branch. However, considering the significant difference in sampling rate between fNIRS and EEG signals, the Gblock structure in the fNIRS branch is redesigned based on the Gblock of the EEG branch. This branch contains four Gblock modules and further reconstructs the temporal and spatial characteristics of the channels through two convolutional layers. The detailed process of data dimension changes is attached. Figure 2 As shown in Table 1, the parameters of each layer in the EEG and fNIRS branches of the multimodal generator are as follows.

[0038] Table 1 Parameters of Multimodal Generator

[0039]

[0040] Step 1.3: Multimodal discriminator structure

[0041] The structure of the multimodal discriminator is shown in the attached figure. Figure 3As shown in Table 2, the multimodal discriminator simultaneously outputs discrimination results for each modality's generated samples and the fused data. Since the generator uses two-dimensional convolutional layers to reconstruct the spatiotemporal features of the signal, the multimodal discriminator needs to evaluate whether these spatiotemporal features are consistent with the distribution of the real samples. To achieve this goal, this paper improves upon the classic EEGNet, designing a multimodal discriminator structure suitable for multimodal data processing. Considering the difference in time step between this study and the original EEGNet, the EEGNet structure is adjusted: the kernel size of the convolutional and pooling layers is increased in the EEG branch, while the kernel size is correspondingly decreased in the fNIRS branch. The parameters of each layer of the multimodal discriminator are shown in Table 2. The multimodal discriminator consists of an original spatiotemporal information extraction module, a deep spatiotemporal feature extraction module, and an output feature mapping module. In the original spatiotemporal information extraction module, the EEG branch first uses a 1×65 kernel-size 2D convolutional layer to perform preliminary feature extraction from the input features, with padding used. Then, a normalization layer is used for feature normalization. Next, a 62×1 kernel-size 2D convolutional layer is used to further extract spatial features, and a normalization layer is used for feature standardization, with ELU as the activation function. Finally, a 1×8 kernel-size average pooling layer is used for downsampling, with a dropout rate of P=0.5. In the corresponding fNIRS branch, the first 2D convolutional layer uses a 1×8 kernel, followed by a normalization layer; the second 2D convolutional layer uses a 48×1 kernel and is normalized using a normalization layer, with ELU as the activation function; then, a 1×3 kernel-size average pooling layer is used for downsampling, with a dropout rate of P=0.5. In the deep spatiotemporal feature extraction module, the EEG branch uses a 1×33 kernel-sized deep convolutional layer to further extract deep features, followed by a 1×1 kernel-sized point convolutional layer to achieve channel information fusion, and then a normalization layer for processing. The activation function is ELU, and finally, a 1×16 kernel-sized average pooling layer is used for downsampling, with a dropout rate of P=0.5. The fNIRS branch uses a 1×2 kernel-sized deep convolutional layer, a 1×1 kernel-sized point convolutional layer, and a normalization layer, with the activation function also being ELU. Finally, a 1×3 kernel-sized average pooling layer is used for downsampling, with a dropout rate of P=0.5. In the output feature mapping module, the EEG branch converts the extracted spatiotemporal features into a one-dimensional feature vector by outputting a 16×7 layer and connecting it to a flattened layer of size 1×112. The fNIRS branch corresponds to an output dimension of 16×4 and completes feature mapping through a flattened layer of size 1×64, providing a unified feature representation for subsequent multimodal discrimination.

[0042] Table 2 Parameter Table of Multimodal Discriminator

[0043]

[0044] The EEG and fNIRS signals input to the multimodal discriminator are first processed by an optimized EEGNet (with the last linear layer removed) for feature extraction. The resulting low-dimensional feature vectors characterize the spatiotemporal properties of the EEG and fNIRS signals. Subsequently, these feature vectors are processed according to the modality type and mapped to the three branches of the three outputs of the corresponding multimodal discriminator.

[0045] Specifically, in the EEG discriminant branch, low-dimensional EEG feature vectors are concatenated with category information, and the discriminant result is output through a linear layer; the fNIRS discriminant branch uses the same processing method. For the fusion discriminant branch, the low-dimensional features and category information of the two modalities are fused and then input into a linear layer. This design enables the multimodal discriminator to effectively process multimodal data and generate accurate discriminant results.

[0046] Step 1.4: Loss Function

[0047] Since this study focuses on multimodal data processing, the proposed discriminator has three outputs. Based on the Bulldozer GAN (Wasserstein GAN, WGAN) proposed by Arjovsky et al., this paper designs a loss function specifically suitable for multimodal scenarios. The generator calculates its loss based on the discriminator's evaluation of the generated (pseudo) samples, and the calculation method is shown in Equation (3).

[0048] (3)

[0049] in, Represents the distribution of generated samples Randomly selected samples And calculate the average of the expressions used in the calculation for the selected samples. , and These represent the discrimination results of the generated data in the EEG mode, the discrimination results in the fNIRS mode, and the fused discrimination results, respectively. , and These represent the weights of EEG, fNIRS, and the fusion discrimination result, respectively (set in this paper). =0.6, =0.2, =0.2). The gradient penalty term is calculated based on the discriminator's output of the EEG and fNIRS signals, and its calculation method is shown in formula (4).

[0050] (4)

[0051] in Represents the distribution of generated samples Randomly selected samples And calculate the average of the expressions used in the calculation for the extracted samples; To adjust the hyperparameter of the penalty term strength, it was set to 10 in this experiment. The discriminator loss... It can be calculated according to formula (5), where and These represent the losses for real samples and generated samples, respectively.

[0052]

[0053] (5)

[0054]

[0055] To further constrain the distributional consistency between the generated signal and the original data, a regularization term based on KL divergence is introduced into the loss function of the MMI generator. Considering the high-dimensionality of multimodal physiological time series signals and the limited batch size during training, this paper assumes that both the original and generated data follow a multivariate Gaussian distribution with a diagonal covariance matrix. For each modality, the KL divergence is calculated independently on each channel. Specifically, given a batch of original signals... and generated signal The time dimension of each channel is flattened. Here, B, C, and T represent the number of trials, channels, and time steps in a batch, respectively. For the c-th channel, the mean of the original data is calculated along the batch dimension. The mean of the generated data Variance of the original data and the variance of the generated data Under the above assumptions, the KL divergence of the c-th channel from the original distribution to the generated distribution can be calculated by formula (6), where t represents the feature dimension after time flattening. This formula corresponds to the closed-form solution of the KL divergence between two Gaussian distributions with diagonal covariance matrices.

[0056] (6)

[0057] Subsequently, the KL divergences on all channels are summed and averaged within each mode to obtain the mode-specific KL loss, as shown in Equation (7).

[0058] (7)

[0059] in In the multimodal setting, the final KL divergence term is defined as a weighted combination of the EEG and fNIRS modes, and its calculation method is shown in formula (8).

[0060] (8)

[0061] in In this paper, we take 0.7. To avoid the KL divergence term dominating the adversarial learning objective in the early stages of training, a linear preheating annealing strategy is adopted for the KL coefficients. The annealing factor is defined as shown in formula (9).

[0062] (9)

[0063] Where t represents the current training step number. Set the number of steps to 10,000. Finally, the overall loss function of the generator is shown in equation (10).

[0064] (10)

[0065] in In this paper, it is set to 0.5.

[0066] Step 2: Bimodal Multi-Attention Network

[0067] Step 2.1: Multi-scale feature aggregation module based on KL alignment

[0068] To comprehensively extract features from both modalities and more effectively utilize cross-modal interaction information, this invention proposes a multi-scale feature aggregation module based on KL alignment. This module consists of three parts: an EEG multi-scale feature extraction module, an fNIRS single-scale feature aggregation module, and a multi-modal information alignment mechanism based on KL divergence.

[0069] Step 2.1.1: EEG Multi-Scale Feature Extraction Module

[0070] To effectively capture the multi-scale temporal characteristics of FJMI EEG signals, we designed an EEG multi-scale feature extraction module with a parallel multi-branch structure. The input EEG signal is represented as... ,in The number of leads representing the EEG signal This represents the number of sampling points for the EEG signal. The input is fed into four parallel convolutional branches, each using a different temporal kernel size in the first convolutional layer. This design enables the network to simultaneously model the short-, medium-, and long-term temporal dependencies inherent in the FJMI EEG pattern. The outputs of all branches are concatenated to form a unified multi-scale EEG feature representation, which serves as the input to the subsequent decoding module. Simultaneously, the output features of these four branches are passed through a classifier to obtain four predictions, denoted as... These four results will serve as additional constraints on the final loss function. The parameters of each layer in this module are shown in Table 3. This module consists of four branches. In each branch, we set convolutional kernels of different scales in the first convolutional layer, the size of which is determined by the branch index. sum vector To be determined jointly. and These represent the number and size of the convolution kernels, respectively. The dropout ratio is represented by . D represents the number of spatial filters learned for each feature map in the depthwise separable convolutional layer. For simplicity, only the detailed configuration of each layer in a single branch is listed in the table. The EEG multi-scale feature extraction module includes an input reshaping layer, a 2D convolutional layer, a 2D normalization layer, a 2D depthwise convolutional layer, a 2D average pooling layer, a dropout layer, and a rearrangement layer. First, the input EEG features are reshaped into a size of (1, ..., ...) , The four-dimensional tensor is then used. Subsequently, two-dimensional convolutional layers are used in each parallel branch to extract features at different time scales. The number of input channels in the first two-dimensional convolutional layer is... The kernel size is set to (1, The number of convolution kernels is The output size is ( , , The activation function uses a linear function, where the kernel length for each branch is determined by parameters. The values ​​are set to 15, 31, 63, and 125 respectively. Convolutions use the same padding method, and the bias term is set to False. The convolution output is normalized through a two-dimensional normalization layer, resulting in a normalized feature channel count of 2×10⁻⁶. Subsequently, a two-dimensional depthwise convolutional layer was used to further extract spatial features. The depth factor of the depthwise convolution was set to DD, and the kernel size was ( ,1), The number of output channels is The output size is ( ,1, The activation function is Linear, where the depth factor DD is 2 and the bias term is set to False. After depthwise convolution, two-dimensional normalization is performed again, and the ELU activation function is used to enhance the network's non-linear expressive power. Then, a two-dimensional average pooling layer with a kernel size of (1, 8) is used for temporal downsampling, with an output size of (...). ,1, / 8), then random deactivation is performed through a drop-out layer, with the drop-out rate set to 0. =0.3. The pooled features are further input into a second two-dimensional depthwise convolutional layer, the kernel size of which is (1, The number of convolution kernels is The output size is ( ,1, / 8), the activation function uses Linear, where The value is set to 15, and the convolution still uses the same padding method, with the bias term set to False. It then passes through a two-dimensional normalization layer and the ELU activation function, followed by further downsampling using a two-dimensional average pooling layer with a kernel size of (1, 7). The output size is ( ,1, / 56), and then randomly deactivated again through the drop-out layer, with the drop-out rate set to 0. =0.3. Finally, the output features are resized to size ( ) by rearranging the layers. / 56, The two-dimensional feature representation is used as input for subsequent feature fusion or classification modules.

[0071] Table 3 Parameters of EEG Multiscale Feature Extraction Module

[0072]

[0073] Step 2.1.2: fNIRS Single-Scale Feature Aggregation Module

[0074] To effectively learn the discriminative spatiotemporal representation of the fNIRS signal while maintaining computational efficiency, we designed an fNIRS single-scale feature aggregation module as the representation learning branch for the fNIRS modes. Let the fNIRS signal be denoted as... ,in The number of leads representing the fNIRS signal, This represents the number of sampling points for the fNIRS signal. For the input... First, temporal convolution is used for preliminary feature extraction to capture short-term hemodynamic changes while preserving the channel structure of the input signal. Then, depthwise separable convolution, batch normalization, ELU activation function, and Dropout operation are applied sequentially to independently model the temporal dependence of each feature map, stabilize the training process, enhance nonlinear expressive power, and alleviate overfitting, respectively. To further optimize and compress the learned feature representation, pointwise convolution is introduced to project and reorganize the expanded feature map along the feature dimension without increasing temporal resolution, thereby achieving feature integration and dimensionality adjustment while maintaining high computational efficiency. Finally, the obtained feature tensors are rearranged into compact fNIRS feature embeddings for subsequent multimodal fusion and decoding. The parameters of each layer of this module are shown in Table 4, where... and These represent the number and size of the convolution kernels, respectively. The dropout ratio is represented by . D represents the number of spatial filters learned for each feature map in the depthwise separable convolutional layer. The fNIRS single-scale feature aggregation module includes an input reshaping layer, a 2D convolutional layer, a 2D normalization layer, a 2D depthwise convolutional layer, a 2D pointwise convolutional layer, a 2D average pooling layer, a dropout layer, and a rearrangement layer. First, the input fNIRS signal is reshaped into a size of (1, ..., ...) , The input signal is then subjected to a four-dimensional tensor. Subsequently, a two-dimensional convolutional layer is used to perform preliminary temporal feature extraction, with the number of convolutional kernels set to [value missing]. The kernel size is (1, The output size is ( , , The activation function is Linear. The number of convolutional kernels... Set to 9, kernel length The value is set to 8, convolutions use the same padding method, and the bias term is set to False. The convolution output is normalized by a two-dimensional normalization layer, resulting in 2×10^2 feature channels. Subsequently, a two-dimensional depthwise convolutional layer was used to extract spatial features. The depth factor of the depthwise convolution was set to D, and the kernel size was ( ,1), The number of output channels is The output size is ( ,1, The activation function is Linear, with a depth factor D of 2 and a bias term set to False. After depthwise convolution, the model passes through a 2D normalization layer and an ELU activation function to enhance its non-linear expressive power. Random deactivation is then performed using a dropout layer with a dropout rate of 100%. =0.5. Then, a two-dimensional point convolutional layer is used to fuse the features of each channel, with a kernel size of (1,1) and a number of kernels. The output size is ( ,1, The activation function is Linear, and the convolution is padded using the same method, with the bias term set to False. The pointwise convolution output is normalized by a two-dimensional normalization layer and then non-linearly mapped using the ELU activation function. Subsequently, a two-dimensional average pooling layer with a kernel size of (1, 2) is used to downsample the time dimension, resulting in an output size of ( ,1, / 2), and then randomly deactivated again through the drop-out layer, with the drop-out rate still set to 1. =0.5. Finally, the output features are transformed to a size of ( ) by rearranging the layers. / 2, The two-dimensional feature representation provides a unified feature input for subsequent multimodal feature fusion and decoding modules.

[0075] Table 4 fNIRS Single-Scale Feature Aggregation Module

[0076]

[0077] Step 2.1.3: Multimodal Information Alignment Mechanism Based on KL Divergence

[0078] Due to inherent differences in temporal resolution and physiological characteristics between EEG and fNIRS signals, direct feature fusion may lead to distribution mismatch. Therefore, an alignment constraint based on KL divergence is introduced to regularize the latent feature distributions of the two modalities. First, the latent features extracted from the EEG and fNIRS branches are normalized to probability distributions along the feature dimension using a SoftMax function. Then, the EEG features... and fNIRS features The probability distributions are converted using formulas (11) and (12) respectively, where and Let represent the probability distributions of the EEG and fNIRS features, respectively. Before normalization, the fNIRS features are linearly interpolated along the time dimension to make their feature length consistent with the EEG branch. The interpolated features are denoted as . .

[0079] (11)

[0080] (12)

[0081] Subsequently, the KL divergence measure is used to measure the distribution difference between EEG and fNIRS features, and the specific calculation method is shown in formula (13).

[0082] (13)

[0083] In this system, the KL divergence is averaged over the batch dimension. This regularization term is incorporated into the overall training objective to constrain the sharing of similar feature distributions between the two modalities. By minimizing the KL divergence, the proposed alignment strategy promotes cross-modal consistency at the distribution level, thereby achieving more effective multimodal fusion and improving the robustness of the learned representations.

[0084] Step 2.2: EEG-guided cross-modal attention module

[0085] The above feature extraction methods can yield features relevant to the FJMI task. To leverage complementary information between different modalities for more accurate predictions, this paper designs an EEG-guided cross-modal attention module, which includes intra-modal encoding and inter-modal cross-attention.

[0086] To overcome the limited effective receptive field of convolutional encoders, this paper employs a self-attention-based global feature extractor to model the long-range temporal dependencies within each modality. After extraction by the local encoder, the temporal features of each modality are first organized into word sequences and then processed independently to preserve modal characteristics. For modality The time feature sequence, where Indicates the number of time-related morphemes. Represents the embedding dimension. Introduces a learnable category token. This is used to aggregate global information, as shown in formula (14).

[0087] (14)

[0088] get To encode time sequence information, learnable position embeddings are added to the input token. As shown in formula (15).

[0089] (15)

[0090] in This represents the input embedding of the self-attention Transformer module, and is input into the module by... In the stack of layers of self-attention encoders (in this paper) For the first... layer( The encoding process is shown in formulas (16) and (17).

[0091] (16)

[0092] (17)

[0093] in , and Let A, B, C, and D represent the multi-head self-attention module, layer normalization, and feedforward network, respectively. The feedforward neural network adopts a two-layer fully connected mapping structure, including one hidden layer. Its input feature dimension is 72, the hidden layer dimension is 144, and the output feature dimension is restored to 72. In the multi-head self-attention module, attention is calculated as shown in formulas (18) and (19).

[0094] (18)

[0095]

[0096] (19)

[0097]

[0098] The SoftMax operation normalizes the attention score along the temporal dimension, thus obtaining attention weights that represent the relative contributions of different temporal terms within the same modality. After layer encoding, the output of the corresponding category lexical unit is extracted as the global representation within the modality. This representation summarizes the modalities. The long-range time dependencies and task-related information are then input into the intermodal encoder, along with the modality-specific global representations of EEG and fNIRS.

[0099] To address the physiological time misalignment between EEG and fNIRS, this paper employs a fusion intermodal encoder to learn the temporal correlation between the two modalities. Let... and These represent the EEG and fNIRS temporal word embeddings after intramodal self-attention encoding, respectively. To capture cross-modal temporal correspondences, the two sequences are concatenated along the temporal dimension to form a fused sequence, as shown in formula (20).

[0100] (20)

[0101] in Indicates the fusion sequence, and These represent the temporal feature lengths of the two modalities, respectively. This concatenation method maps the two modalities to a unified temporally aware representation space while preserving their original temporal order. To retain modal characteristics after alignment, learnable modal embeddings are added to the fusion lexical units. As shown in Equation (21), this enables the attention mechanism to distinguish different modal sources within a shared time frame. Represents the input terms of the intermodal self-attention encoder, and is input into the encoder. In the stack of layer encoders (in this article) For the first... layer( , its input is denoted as The intermodal self-attention encoding process is the same as that in formulas (16), (17), (18), and (19), except that the subscripts are changed. Replace with The resulting fusion representation captures complementary temporal information between the two modalities and provides global context for further optimization of modality fusion features through cross-attention.

[0102] (twenty one)

[0103] In multimodal systems, traditional fusion methods (such as concatenation or weighted summation) typically assume that the contributions of each modality are fixed and independent, making it difficult to capture the complex and dynamically changing dependencies between modalities, especially in EEG–fNIRS fusion scenarios. To address this issue, this paper proposes a cross-modal multi-head self-attention mechanism. Given that EEG typically dominates prediction performance, this module uses EEG as the query (Q) and models its fusion relationship with fNIRS through keys (K) and values ​​(V). The input of this module is... and , respectively represent the feature maps after the EEG intramodal encoder and the EEG-fNIRS intermodal encoder. The encoding process is consistent with formulas (16), (17) and (18), except that Q, K and V are replaced with the form defined by formula (22) in formula (18).

[0104]

[0105] (twenty two)

[0106]

[0107] The introduction of cross-modal attention shifts the model from a simple static fusion approach to a more dynamic, context-dependent modal interaction approach. This method enhances the feature representation capabilities of each modality by explicitly modeling the conditional dependencies between EEG and fNIRS. Its key innovation lies in the fact that the effective representation of the EEG modality is modeled under the condition of fNIRS information, thus making the fusion process more adaptive and context-aware.

[0108] Step 2.3: Classifier and Loss Function

[0109] The classifier uses the learned feature maps, flattens them into a one-dimensional representation, and inputs it into a fully connected layer to make the output dimension consistent with the number of target classes. Then, the LogSoftMax function is used to generate the probability distribution of each class, and the class with the highest probability is used as the final prediction result.

[0110] The loss function consists of Negative Log Likelihood Loss (NLLLoss), auxiliary branches, and KL divergence to achieve joint optimization of the model. The main loss... The NLL loss is defined as the difference between the final prediction and the true label; auxiliary loss. This results in four EEG branch outputs (denoted as...). The sum of the NLL losses corresponding to ). As shown in formula (23), where This indicates that the loss is calculated using the negative log-likelihood algorithm, and the total loss is... , and The weighted combination, whose weights are determined by hyperparameters and Control (set to 0.6 and 0.15 respectively in this article).

[0111] ,

[0112] , (twenty three)

[0113]

[0114] Experiment and results:

[0115] To verify the effectiveness of the proposed method, experiments were conducted on a self-collected fine motion imagery dataset (published as a public dataset). The proposed method, existing EEG single-modal methods, and existing EEG-fNIRS bimodal methods were implemented in code, and the effectiveness of the proposed method was verified through comparative experiments. An ablation study was conducted on the three inventive aspects of this invention to verify their effectiveness in the proposed method. Furthermore, to verify that the system composed of the data augmentation method and the pattern recognition network of this invention can achieve optimal results, the data augmentation method was replaced, verifying the excellent performance of the system. The data preprocessing method, experimental setup, and experimental results are described in detail below.

[0116] 1. Dataset and Preprocessing Methods

[0117] This invention uses the FineMI dataset for experiments. This dataset contains multiple degrees of freedom movements of the right upper limb (hand, wrist, elbow, and shoulder) from 18 subjects. Specifically, it includes eight sets of movements: hand opening / closing, wrist flexion / extension, wrist abduction / adduction, elbow pronation / supination, elbow flexion / extension, shoulder pronation / supination, shoulder abduction / adduction, and shoulder flexion / extension. In this experiment, only three movements—hand opening / closing, wrist flexion / extension, and shoulder pronation / supination—were used for a 3-class classification experiment. The EEG signals in this dataset were acquired using 64 electrode channels at an original sampling rate of 1000Hz, then downsampled to 250Hz; the fNIRS signals were acquired using 24 channels consisting of 8 near-infrared light sources and 8 detectors at a sampling rate of 7.8125Hz. The total number of trials used in this invention's experiments was 2160 (class balanced, 40 trials per subject per movement). A flowchart of one trial during data acquisition is shown in the attached figure. Figure 4 As shown. During each trial's data collection, a fixed crosshair is displayed on the screen for 0-2 seconds to help the subject focus their attention; from 2-4 seconds, a video of the current trial's task execution is played along with corresponding text prompts; from 4-8 seconds, the subject begins to visualize the movement; and finally, from 10-12 seconds, the subject relaxes. This invention extracts EEG and fNIRS segments from 0-4 seconds after the prompts as input samples. The data shape of one trial of EEG is 62×1000 (channels × sampling points), and the data shape of one trial of fNIRS is 24×32 (channels × sampling points).

[0118] The data preprocessing method of this invention is described below. For EEG data, firstly, a common average reference (CAR) method is used to correct the signal. Secondly, a third-order Butterworth bandpass filter is used for filtering, with a frequency band set to 0.5–40 Hz. Thirdly, an exponential moving average standardization (EMAS) method is used for standardization to reduce the numerical differences between trials. Finally, the EEG signal is divided into multiple trial samples according to the task label. For fNIRS data, firstly, the changes in oxygenated hemoglobin and deoxygenated hemoglobin concentrations are calculated according to the modified Beer-Lambert law. Secondly, a third-order Butterworth bandpass filter is used for filtering, with a frequency band set to 0.01–0.1 Hz. Thirdly, the EMAS method is used to reduce the differences between trials. Finally, the fNIRS signal is divided into multiple trial samples according to the task label.

[0119] 2. Experimental Setup

[0120] To evaluate the effectiveness of this invention, we conducted comparative experiments with single-modal and multi-modal decoding methods. In the experiments, we implemented all methods using the MNE, scikit-learn, and PyTorch frameworks, and evaluated them on a server equipped with a Xeon(R) Gold6326 CPU, 128GB RAM, and an RTX A6000 GPU. To determine statistical significance, we used a one-sided Wilcoxon signed-rank test combined with Bonferroni-Holm correction, with a significance level set at 0.05.

[0121] This experiment employs a single-subject 5-fold cross-validation method, dividing the dataset into five equal parts. In each iteration, one fold (approximately 24 trials) serves as the test set, while the remaining data is further divided into a training set (approximately 77 trials) and a validation set (approximately 19 trials) in a 4:1 ratio. All proposed data augmentation techniques are applied only to the training set, increasing its size by six times. Therefore, the augmented training set contains 539 trials, the validation set contains 19 trials, and the test set contains 24 trials.

[0122] For the parameter settings and training configuration of MMI-CWGAN, both the generator and discriminator were optimized using the Adam optimizer with a learning rate of 0.0001. All parameters of the MMI generator and MMI discriminator were randomly initialized with a mean of 0 and a standard deviation of 0.02. Furthermore, the random noise input to the generator also followed the same Gaussian distribution as the parameter initialization, i.e., a mean of 0 and a standard deviation of 0.02. The training process consisted of 300 epochs, with the discriminator parameters updated every 5 times and the generator parameters updated once.

[0123] The feature capture module of this invention, along with other deep learning comparison methods, employs the Adam optimizer with a learning rate of 0.001 and a batch size of 64. Other deep learning-based methods use cross-entropy loss to measure the difference between the predicted classification result and the true label. We employ a two-stage training strategy, with each stage training for 400 epochs, and use an early stopping mechanism with a threshold of 200 to avoid overfitting.

[0124] 3. Experimental Results

[0125] (1) Comparison with typical methods

[0126] We conducted a comparative experiment involving two types of methods: EEG unimodal methods and EEG-fNIRS multimodal methods. The EEG unimodal methods included EEGNet, MSVTNet, ShallowConvNet, FB-DCANet, Sinc-ShallowNet, and EEGConformer. The EEG-fNIRS multimodal methods included PFNet, FB-Sinc-FusionNet, STANet, and Bimodal CNN. The experimental results are shown in Table 5, where "*" indicates statistical significance compared to the comparative methods. The proposed BiMA-GAS outperformed other methods in both mean accuracy and F1 score, with statistically significant differences (p<0.05). Compared to unimodal methods, the proposed system maintained high mean performance while exhibiting the lowest standard deviation, indicating not only high decoding accuracy across different subjects but also greater stability. Compared to other multimodal methods, this method also achieved the best overall performance. Although PFNet and STANet showed slightly lower standard deviations, this apparent stability came at the cost of significantly lower classification accuracy. Furthermore, the overall performance of existing EEG-fNIRS multimodal methods remains lower than some methods that only use EEG, indicating that simple or coarse modality fusion does not necessarily lead to performance improvements. In contrast, our proposed method achieves a significant improvement over its corresponding single-modal baseline method, demonstrating its effectiveness in utilizing cross-modal information. These phenomena can be attributed to the inherent characteristics of fine joint motion imagery. Due to the small spatial spacing of cortical activation patterns, the spatiotemporal representations shared between EEG and fNIRS are often ambiguous and lack discriminative power. Moreover, most existing multimodal frameworks are primarily designed for simple motion imagery tasks, making it difficult to capture the subtle neural differences required for decoding fine joint motion imagery. These observations further illustrate the need to develop specialized representation learning strategies to jointly model discriminative features across multiple dimensions, including time, space, and spectrum, thereby more effectively utilizing complementary information between different modalities.

[0127] Table 5 Comparative Experiment Results

[0128]

[0129] (2) Ablation test

[0130] To evaluate the effectiveness of the three key contributions, we conducted ablation experiments and designed three variant models: BiMA-GAS without MMI-CWGAN, BiMA-GAS without KL loss term, and BiMA-GAS with EEG guidance replaced by fNIRS guidance in cross-modal multi-head attention. Among them, BiMA-GAS without MMI-CWGAN means that the model is trained directly using the original data; BiMA-GAS without KL loss term means that the loss function is calculated according to formula (24); and in cross-modal multi-head attention, replacing EEG guidance with fNIRS guidance means replacing the original formula (22) with formula (25).

[0131] (twenty four)

[0132]

[0133] (25)

[0134]

[0135] The ablation experiment results are shown in Table 6, where "*" indicates that the results of the present invention are statistically significant compared with the comparative methods. The results in Table 6 show that the proposed method has higher average accuracy and kappa value, as well as lower standard deviation, indicating that the three key modules proposed in this paper are necessary and play an important role in improving the overall performance.

[0136] Table 6 Ablation Experiment Results

[0137]

Claims

1. A method for non-invasive brain-computer interface fine motor imagery decoding, characterized in that: This method first performs data augmentation on the acquired EEG and fNIRS bimodal signals; then, the augmented bimodal data are input into a bi-branch feature learning network to extract multi-scale temporal features from the EEG signal and aggregate features from the fNIRS signal, and align the feature distributions of the two modalities by introducing KL divergence constraints; a cross-modal attention fusion mechanism based on EEG is constructed to dynamically weight and interactively model information from different modalities, thereby achieving adaptive fusion of multimodal features; finally, a classifier is used to discriminate the fused features to decode the joint motion imagination task.

2. The method according to claim 1, characterized in that, The KL alignment-based multi-scale feature aggregation method is as follows: To address the differences in temporal resolution and physiological characteristics between EEG and fNIRS signals, a dual-branch representation learning structure was designed. In the EEG branch, a multi-scale feature extraction strategy was adopted, and dynamic information at different time scales was modeled through a parallel structure to capture temporal features in the motor imagery task. In the fNIRS branch, a single-scale feature aggregation method was used to compress and integrate temporal information related to blood oxygenation changes to obtain discriminative modal feature representations. A feature alignment mechanism based on KL divergence is introduced to map the features extracted by EEG and fNIRS to a unified distribution space.

3. The method according to claim 1, characterized in that, The EEG-guided modal fusion strategy based on cross-modal attention mechanism is as follows: First, each modal feature is internally encoded, and long-distance temporal dependencies are modeled by introducing an attention-based structure, thereby obtaining a modal representation containing global contextual information. Subsequently, the features of different modalities are modeled in a unified manner to construct a cross-modal feature interaction space in order to capture the temporal correlation and semantic relationship between different modalities; Based on this, an EEG-guided cross-modal attention mechanism is adopted, which uses EEG features as the dominant information and selectively introduces fNIRS features through attention weight calculation.

4. The method according to claim 1, characterized in that, Step 1 is as follows: Step 1.1: Construction of the overall structure of the bimodal data augmentation method based on generative adversarial networks The data augmentation method consists of two parts: a multimodal generator and a multimodal discriminator. The multimodal generator takes random noise and category information as input to generate EEG-fNIRS data. Subsequently, the generated multimodal signal is fused with the original signal and input together into the multimodal discriminator. In addition, the output of the multimodal generator is used to calculate the KL divergence between the generated data and the original data to align their features. The multimodal discriminator is used to distinguish between real data and generated data and identifies the data category by combining label information. At the same time, the output of the discriminator is used to calculate the loss function and further update the weights of the generator and discriminator. The calculation process is shown in formula (1). (1)； in, , and These represent the generated multimodal data, the original EEG signal, and the original fNIRS signal, respectively. and These represent Gaussian noise and category labels, respectively. and These represent the synthetic data generated by the generator, which includes EEG and fNIRS features, respectively. The sample obtained by interpolating real data and generated data is used as one of the inputs to the multimodal discriminator; the interpolated samples of the two modalities are calculated separately, and the calculation method is shown in formula (2). (2)； in, This represents the input to the multimodal discriminator. This represents the random matrix used for interpolation. and These represent the raw EEG and fNIRS data, respectively. and This indicates the corresponding generated data; Step 1.2: Multimodal Generator Structure The proposed multimodal generator consists of two main parts: the EEG branch and the fNIRS branch. In each branch, the core structure of the multimodal generator network mainly consists of upsampling layers, convolutional layers, and batch normalization layers. These three layers are combined into a unit called the Generation Block (Gblock), which is used to progressively enhance the clarity of features and ensure the stability of the data. The upsampling layer uses bilinear interpolation to double the input size, thereby improving the spatial resolution of the feature map. The batch normalization layer is applied to the output of each layer to ensure that the data remains in a standardized state with a mean of 0 and a variance of 1. In the EEG branch of the multimodal generator, the input high-dimensional features are first reshaped into a 64×4×16 (number of convolutional kernels × number of channels × length) feature matrix to achieve dimensionality reduction; then, the feature matrix is ​​gradually expanded through five Gblock modules; the fNIRS branch of the multimodal generator contains four Gblock modules and further reconstructs the temporal and spatial features of the channels through two convolutional layers.

5. The method according to claim 1, characterized in that, The multimodal discriminator consists of a raw spatiotemporal information extraction module, a deep spatiotemporal feature extraction module, and an output feature mapping module. In the raw spatiotemporal information extraction module, the EEG branch first uses a 1×65 kernel-size 2D convolutional layer to perform preliminary feature extraction from the input features, with padding. Then, a normalization layer is used for feature normalization. Next, a 62×1 kernel-size 2D convolutional layer is used to further extract spatial features, and a normalization layer is used for feature standardization, with ELU as the activation function. Finally, a 1×8 kernel-size average pooling layer is used for downsampling, with a dropout rate of P=0.

5. In the corresponding fNIRS branch, the first 2D convolutional layer uses a 1×8 kernel, followed by a normalization layer. The second 2D convolutional layer uses a 48×1 kernel and is normalized by a normalization layer, with ELU as the activation function. Then, a 1×3 kernel-size average pooling layer is used for downsampling, with a dropout rate of P=0.

5. The deep spatiotemporal... In the feature extraction module, the EEG branch uses a 1×33 deep convolutional layer to further extract deep features, followed by a 1×1 point convolutional layer to fuse channel information, and then a normalization layer. The activation function is ELU, and finally, a 1×16 average pooling layer is used for downsampling, with a dropout rate of P=0.

5. The fNIRS branch uses a 1×2 deep convolutional layer, a 1×1 point convolutional layer, and a normalization layer, where the activation... The function also uses ELU, and finally uses an average pooling layer with a kernel size of 1×3 to complete downsampling. The dropout rate of the pooling layer is set to P=0.

5. In the output feature mapping module, the EEG branch uses an output layer with an output dimension of 16×7 and connects to a flattening layer with a size of 1×112 to convert the extracted spatiotemporal features into a one-dimensional feature vector. The fNIRS branch corresponds to an output dimension of 16×4 and completes feature mapping through a flattening layer with a size of 1×64, providing a unified feature representation for subsequent multimodal discrimination.

6. The method according to claim 1, characterized in that, Step 1.4: The loss function is calculated as shown in formula (3). (3) in, Represents the distribution of generated samples Randomly selected samples And calculate the average of the expressions used in the calculation for the selected samples. , and These represent the discrimination results of the generated data in the EEG mode, the discrimination results in the fNIRS mode, and the fused discrimination results, respectively. , and represents the weights of EEG, fNIRS, and the fused discrimination result, respectively; the gradient penalty term is calculated based on the discriminator's output of the EEG and fNIRS signals, and its calculation method is shown in formula (4). (4)； in Represents the distribution of generated samples Randomly selected samples And calculate the average of the expressions used in the calculation for the extracted samples; Hyperparameters for adjusting the strength of the penalty term; the discriminator's loss. Calculated according to formula (5), where and These represent the losses for real samples and generated samples, respectively. ； (5)； ； Given a batch of raw signals and generated signal Flatten the time dimension of each channel; where B, C, and T represent the number of trials, channels, and time steps in a batch, respectively; for the c-th channel, calculate the mean of the original data along the batch dimension. The mean of the generated data Variance of the original data and the variance of the generated data Under the above assumptions, the KL divergence of the c-th channel from the original distribution to the generated distribution can be calculated by formula (6), where t represents the feature dimension after time flattening; this formula corresponds to the closed-form solution of the KL divergence between two Gaussian distributions with diagonal covariance matrices. (6)； Subsequently, the KL divergences on all channels are summed and averaged within each mode to obtain the mode-specific KL loss, as shown in Equation (7). (7)； in Under the multimodal setting, the final KL divergence term is defined as a weighted combination of the EEG and fNIRS modes, and its calculation method is shown in formula (8). (8)； in The value is set to 0.7; to avoid the KL divergence term dominating the adversarial learning objective in the early stage of training, a linear preheating annealing strategy is adopted for the KL coefficients, and the annealing factor is defined as shown in formula (9). (9)； Where t represents the current training step number. Set to 10,000 steps; finally, the overall loss function of the generator is shown in equation (10). (10)； in The value is 0.

5.

7. The method according to claim 1, characterized in that, Step 2: Bimodal Multi-Attention Network Step 2.1: Multi-scale feature aggregation module based on KL alignment This module consists of three parts: an EEG multi-scale feature extraction module, an fNIRS single-scale feature aggregation module, and a multi-modal information alignment mechanism based on KL divergence; Step 2.1.1: EEG Multi-Scale Feature Extraction Module A parallel multi-branch structure EEG multi-scale feature extraction module was designed; the input EEG signal is represented as... ,in The number of leads representing the EEG signal The number of sampling points representing the EEG signal; The input is fed into four parallel convolutional branches; The output features of these four branches are used by a classifier to obtain four prediction results, denoted as follows: These four results will serve as additional constraints on the final loss function; the module consists of four branches; in each branch, convolutional kernels of different scales are set in the first convolutional layer, their size determined by the branch index. sum vector To be determined jointly; and These represent the number and size of the convolution kernels, respectively. denoted by the dropout ratio; D represents the number of spatial filters learned for each feature map in a depthwise separable convolutional layer. The EEG multi-scale feature extraction module includes an input reshaping layer, a two-dimensional convolutional layer, a two-dimensional normalization layer, a two-dimensional depthwise convolutional layer, a two-dimensional average pooling layer, a dropout layer, and a rearrangement layer; among them, the input EEG features are first reshaped into a size of (1, ..., ...) , The four-dimensional tensor is used; subsequently, two-dimensional convolutional layers are used in each parallel branch to extract features at different time scales. The number of input channels of the first two-dimensional convolutional layer is... The kernel size is set to (1, The number of convolution kernels is The output size is ( , , The activation function uses a linear function, where the kernel length for each branch is determined by parameters. The values ​​are set to 15, 31, 63, and 125 respectively. Convolutions are padded using the same padding method, and the bias term is set to False. The convolution outputs are normalized using a two-dimensional normalization layer, resulting in a normalized feature channel count of 2×10⁻⁶. Subsequently, a two-dimensional depthwise convolutional layer is used to further extract spatial features. The depth factor of the depthwise convolution is set to DD, and the kernel size is ( ,1), The number of output channels is The output size is ( ,1, The activation function is Linear, where the depth factor DD is 2 and the bias term is set to False. After depthwise convolution, two-dimensional normalization is performed again, and the ELU activation function is used to enhance the non-linear expressive power of the network. Then, a two-dimensional average pooling layer with a kernel size of (1, 8) is used for temporal downsampling, and the output size is (…). ,1, / 8), then random deactivation is performed through a drop-out layer, with the drop-out rate set to 0. =0.3; The pooled features are further input into a second two-dimensional depthwise convolutional layer, the kernel size of which is (1, The number of convolution kernels is The output size is ( ,1, / 8), the activation function uses Linear, where The value is set to 15, and the convolution still uses the same padding method, with the bias term set to False; then it is processed sequentially through a two-dimensional normalization layer and the ELU activation function, and then further downsampled using a two-dimensional average pooling layer with a convolution kernel size of (1, 7), with an output size of ( ,1, / 56), and then randomly deactivated again through the drop-out layer, with the drop-out rate set to 0. =0.3; Finally, the output features are resized to size ( ) by rearranging the layers. / 56, The two-dimensional feature representation is used as input for subsequent feature fusion or classification modules.

8. The method according to claim 1, characterized in that, Step 2.1.2: fNIRS Single-Scale Feature Aggregation Module To effectively learn the discriminative spatiotemporal representation of the fNIRS signal while maintaining computational efficiency, we designed an fNIRS single-scale feature aggregation module as a representation learning branch for the fNIRS modality; the fNIRS signal is denoted as... ,in The number of leads representing the fNIRS signal, This represents the number of sampling points for the fNIRS signal; for the input... First, temporal convolution is used for preliminary feature extraction to capture short-term hemodynamic changes while preserving the channel structure of the input signal. Then, depthwise separable convolution, batch normalization, ELU activation function, and Dropout operation are applied sequentially to independently model the temporal dependence of each feature map, stabilize the training process, enhance nonlinear expressive power, and alleviate overfitting, respectively. To further optimize and compress the learned feature representation, pointwise convolution is introduced to project and reorganize the expanded feature map along the feature dimension without increasing temporal resolution, thereby achieving feature integration and dimensional adjustment while maintaining high computational efficiency. Finally, the obtained feature tensors are rearranged into compact fNIRS feature embeddings for subsequent multimodal fusion and decoding. In this module, and These represent the number and size of the convolution kernels, respectively. The dropout ratio is represented by ; D represents the number of spatial filters learned for each feature map in the depthwise separable convolutional layer; the fNIRS single-scale feature aggregation module includes an input reshaping layer, a 2D convolutional layer, a 2D normalization layer, a 2D depthwise convolutional layer, a 2D point convolutional layer, a 2D average pooling layer, a dropout layer, and a rearrangement layer; firstly, the input fNIRS signal is reshaped into a size of (1, ..., ... , The input signal is then subjected to a four-dimensional tensor; subsequently, a two-dimensional convolutional layer is used to perform preliminary temporal feature extraction, with the number of convolutional kernels set to [value missing]. The kernel size is (1, The output size is ( , , The activation function is Linear; where the number of convolutional kernels is... Set to 9, kernel length The value is set to 8, convolutions use the same padding method, and the bias term is set to False. The convolution output is normalized by a two-dimensional normalization layer, resulting in 2×10^2 feature channels. Subsequently, a two-dimensional depthwise convolutional layer was used to extract spatial features. The depth factor of the depthwise convolution was set to D, and the kernel size was ( ,1), The number of output channels is The output size is ( ,1, The activation function is Linear, with a depth factor D of 2 and a bias term set to False. After depthwise convolution, the model passes through a 2D normalization layer and an ELU activation function to enhance its non-linear expressive power. Random deactivation is achieved through dropout layers with a dropout rate of 100%. =0.5; then, a two-dimensional point convolutional layer is used to fuse the features of each channel, with a kernel size of (1,1) and a number of kernels. The output size is ( ,1, The activation function is Linear, the convolution is padded using the same method, and the bias term is set to False; the point convolution output is normalized by a two-dimensional normalization layer and non-linearly mapped using the ELU activation function; Subsequently, a two-dimensional average pooling layer with a kernel size of (1, 2) is used to downsample the time dimension, and the output size is ( ,1, / 2), and then randomly deactivated again through the drop-out layer, with the drop-out rate still set to 1. =0.5; Finally, the output features are transformed into a size of ( ) by rearranging the layers. / 2, The two-dimensional feature representation provides a unified feature input for subsequent multimodal feature fusion and decoding modules.

9. The method according to claim 1, characterized in that, Step 2.1.3: Multimodal Information Alignment Mechanism Based on KL Divergence Alignment constraints based on KL divergence are introduced to regularize the latent feature distributions of the two modalities. First, the latent features extracted from the EEG and fNIRS branches are normalized to probability distributions along the feature dimension using the SoftMax function. Then, the EEG features... and fNIRS features The probability distributions are converted using formulas (11) and (12) respectively, where and Let represent the probability distributions of the EEG and fNIRS features, respectively. Before normalization, the fNIRS features are linearly interpolated along the time dimension to make their feature length consistent with the EEG branch. The interpolated features are denoted as . ; (11)； (12)； Subsequently, the KL divergence measure was used to measure the distribution difference between EEG and fNIRS features. The specific calculation method is shown in formula (13). (13)。 10. The method according to claim 1, characterized in that, Step 2.2: EEG-guided cross-modal attention module A self-attention-based global feature extractor is employed to model the long-range temporal dependencies within each modality. After extraction by the local encoder, the temporal features of each modality are first organized into word sequences and then processed independently to preserve modal characteristics. For modality The time feature sequence, where Indicates the number of time-related morphemes. Represents the embedding dimension; introduces a learnable category token. This is used to aggregate global information, as shown in formula (14). (14)； get To encode time sequence information, a learnable position embedding is added to the input token. As shown in formula (15); (15)； in This represents the input embedding of the self-attention Transformer module, and is input into the module by... In a stack of layers of self-attention encoders, For the first layer( The encoding process is shown in formulas (16) and (17). (16)； (17)； in , and Let X represent the multi-head self-attention module, layer normalization, and feedforward network, respectively. The feedforward neural network adopts a two-layer fully connected mapping structure, including one hidden layer. Its input feature dimension is 72, the hidden layer dimension is 144, and the output feature dimension is restored to 72. In the multi-head self-attention module, the attention calculation is shown in formulas (18) and (19). (18)； ； ； (19)； ； The SoftMax operation normalizes the attention score in the time dimension, thereby obtaining the attention weights that represent the relative contributions of different temporal words within the same modality. go through After layer encoding, the output of the corresponding category lexical unit is extracted as the global representation within the modality. This representation summarizes the modalities. The long-range time dependencies and task-related information are then input into the intermodal encoder, along with the modality-specific global representations of EEG and fNIRS. To address the physiological time misalignment between EEG and fNIRS, this paper employs a fusion intermodal encoder to learn the temporal correlation between the two modalities; assuming... and These represent the EEG and fNIRS temporal word embeddings after intramodal self-attention encoding, respectively. To capture cross-modal temporal correspondences, the two sequences are concatenated along the temporal dimension to form a fused sequence, as shown in formula (20). (20)； in Indicates the fusion sequence, and These represent the time feature lengths of the two modes, respectively. This concatenation method preserves the original temporal order of each modality while mapping the two modalities to a unified temporally aware representation space. To retain modal characteristics after alignment, learnable modal embeddings are added to the fused lexical units. As shown in Equation (21), this enables the attention mechanism to distinguish different modal sources within a shared time frame; Represents the input terms of the intermodal self-attention encoder, and is input into the encoder. In the stack of layer encoders; for the first layer( , its input is denoted as The intermodal self-attention encoding process is the same as that in formulas (16), (17), (18) and (19), except that the subscripts are changed. Replace with The resulting fusion representation can capture complementary temporal information between the two modalities and provide global context for further optimization of modality fusion features through cross attention. (21)； A cross-modal multi-head self-attention mechanism is proposed; given that EEG typically dominates prediction performance, this module uses EEG as the query (Q) and models its fusion relationship with fNIRS through keys (K) and values ​​(V); the input of this module is... and , representing the feature maps after the EEG intramodal encoder and the EEG-fNIRS intermodal encoder, respectively; Its encoding process is consistent with formulas (16), (17) and (18), except that Q, K and V are replaced with the form defined in formula (22) in formula (18); ； (22)； 。

Images