Time alignment and dynamic fusion method for electroencephalogram and functional near-infrared signals

Abstract

The invention belongs to the technical field of multi-mode brain-computer interface signal processing, and particularly relates to a time alignment and dynamic fusion method based on electroencephalogram and functional near-infrared signals. The method comprises the following steps: neural network setting: setting a feature decoupling module for decoupling electroencephalogram features and functional near-infrared features into shared features and specific features; setting a deformable time alignment module, and performing adaptive time alignment on the shared features and the specific features based on a deformable alignment mechanism of a learnable offset delta t; the setting diagram distillation module is used for realizing fusion output of specific features and shared features by constructing a cross-modal diagram structure between samples and introducing dynamic edge weight and dual difference measurement; neural network training: constructing an electroencephalogram and functional near-infrared signal fusion loss function, and performing neural network model training; and feature fusion: inputting to-be-fused electroencephalogram and functional near-infrared signals into the trained neural network, and outputting fusion features by the neural network.

Description

Technical Field

[0001] The present invention belongs to the technical field of multimodal brain-computer interface signal processing, and specifically relates to a time alignment and dynamic fusion method based on electroencephalogram (EEG) and functional near-infrared (FNIR) signals. Background Art

[0002] With the development of multimodal neural signal processing technology, the fusion of electroencephalogram (EEG) and functional near-infrared spectroscopy (fNIRS) has shown great promise in brain-computer interface (BCI) applications such as emotion recognition, cognitive load assessment, and motor imagery decoding. EEG has millisecond-level temporal resolution, enabling real-time capture of rapid changes in neural activity, but its spatial resolution is poor. fNIRS, on the other hand, can detect changes in cortical blood oxygen concentration with high spatial accuracy, but exhibits a certain time delay in response. These two signals differ significantly in physiological mechanisms, response timing, and semantic expression, posing numerous challenges to achieving efficient and accurate cross-modal fusion.

[0003] Existing EEG-fNIRS fusion methods often use methods such as feature splicing, decision-level fusion, or attention mechanisms. Although these methods can improve recognition performance to a certain extent, they often limit the fusion effect due to the failure to fully consider the time scale differences and semantic expression inconsistencies between modalities. Therefore, current EEG-fNIRS multimodal emotion recognition methods often use simple feature splicing or synchronous fusion strategies. Although they have made some progress in improving recognition accuracy, they still have the following significant drawbacks:

[0004] First, the time delay difference between modes is ignored.

[0005] EEG signals reflect neural electrical activity and have a time resolution of milliseconds; while functional fNIRS signals reflect changes in blood oxygen concentration and are limited by physiological response mechanisms, with a time lag of 2 to 6 seconds. Currently, most fusion methods assume the time synchronization of the two modalities by default, and fail to fully model this inherent time delay across modalities, which can easily lead to feature misalignment and semantic inconsistency, thus affecting the final recognition performance and model stability. For example, the blood flow response of fNIRS usually lags behind the changes in neural activity reflected by EEG, resulting in significant time misalignment between the data of the two modalities at the same stimulation moment, such as Figure 1 shown.

[0006] Second, there is a lack of flexible modal alignment mechanism

[0007] During multimodal fusion, different modalities exhibit dynamic variations in time. For example, response patterns may vary significantly across subjects and under different emotional states. Traditional fusion methods typically employ fixed alignment schemes (such as synchronous interception and mean alignment), lacking adaptive alignment mechanisms for asynchronous responses. These methods struggle to cope with complex temporal variations, and are particularly ineffective in non-stationary signal environments. For example, EEG focuses on reflecting transient electrical activity, while fNIRS focuses more on stable metabolic processes. This heterogeneity between modalities complicates feature alignment and knowledge transfer.

[0008] Third, the information fusion strategy is single and the modal collaboration efficiency is low.

[0009] Most existing methods use serial splicing or early fusion structures, ignoring the directionality of information flow and semantic level differences between modalities. This leads to redundancy, conflict, or insufficient coupling of fused features, making it difficult to fully leverage the information complementarity potential of EEG and fNIRS.

[0010] To this end, researchers have begun exploring deformable mechanisms for cross-modal temporal alignment and improving the efficiency of inter-modal information transfer through methods such as graph distillation. However, relying solely on a single mechanism often struggles to adapt to complex cross-modal dynamic response relationships. For example, while deformable modules can mitigate inter-modal latency differences to some extent, they remain limited in capturing long-term cross-modal dependencies and semantic consistency.

[0011] Therefore, there is an urgent need for a cross-modal processing method that can adaptively align time scale differences, extract modality-shared and modality-specific information, and achieve effective dynamic fusion, so as to further improve the fusion efficiency and downstream task capabilities of EEG and fNIRS data, and provide key support for high-precision decoding of complex brain cognitive states. Summary of the Invention

[0012] In view of this, the present invention proposes a time alignment and dynamic fusion method based on EEG and functional near-infrared signals. This method can synchronously fuse EEG and near-infrared signals with low latency and high precision, and achieve more physiologically interpretable analysis results in scenarios such as emotion recognition and neurological disease monitoring. It is suitable for portable neurological monitoring systems and intelligent intervention platforms.

[0013] The technical solutions for implementing the present invention are as follows:

[0014] In the first aspect, the present invention provides a method for time alignment and dynamic fusion based on EEG and functional near-infrared signals, comprising:

[0015] Neural network setup: A feature decoupling module, a variability time alignment module, and a graph distillation fusion module are set up. The feature decoupling module is used to decouple EEG features and functional near-infrared features into shared features and specific features. The deformable time alignment module adaptively aligns shared features and specific features based on a deformable alignment mechanism with a learnable offset Δt. The graph distillation module achieves the fusion output of specific features and shared features by constructing a cross-modal graph structure between samples and introducing dynamic edge weights and dual difference metrics.

[0016] Neural network training: Construct the EEG and functional near-infrared signal fusion loss function and perform neural network model training;

[0017] Feature fusion: The EEG and functional near-infrared signals to be fused are input into the trained neural network, and the neural network outputs the fusion features.

[0018] Optionally, the shared features described in the present invention are: stable representations that are consistently expressed in different modalities, strongly related to task semantics, and have cross-modal consistency; the specific features are: specific features: personalized changes caused by differences in factors during acquisition in different modalities, reflecting specific dynamic response laws within the modality.

[0019] Optionally, the execution process of the feature decoupling module described in the present invention is: first, the EEG signal and the functional near-infrared fNIRS signal are band-pass filtered, sliding window sliced ​​and standardized to extract shallow representations; secondly, the features of each modality are decoupled into shared features and specific features; finally, the decoupled features are deeply modeled.

[0020] Optionally, the adaptive time alignment of shared features and specific features according to the present invention is specifically as follows:

[0021] Assume that the shared features of deep modeling are represented as Specific features are represented as mod∈{eeg,fnirs}; then the time alignment is:

[0022]

[0023] Here, Δt represents the temporal offset between modalities, and Align(·) represents the semantic temporal alignment and information compensation between modalities.

[0024] Optionally, the present invention introduces dynamic edge weights and dual difference metrics by constructing a cross-modal graph structure between samples, specifically:

[0025] Constructing cross-modal graph structures using time-aligned features Node Set represents all samples under EEG and fNIRS modalities, and the edge set ε represents the directed edges between different modalities;

[0026] In the cross-modal graph, let the error of the edge from node i to node j be i→j for:

[0027]

[0028] Where D(·) is the cross-modal difference metric function, γ is the hyperparameter for adjusting the spatial distance regularization term, and z i and z j Represent EEG and fNIRS modal features respectively; cross-modal difference measurement functions include KL divergence based on probability distribution and feature difference based on contrastive learning.

[0029] Optionally, the KL divergence based on probability distribution in the present invention is:

[0030]

[0031] in, Represents the classification prediction vector of modality m after linear mapping, and softmax() represents the normalization of the logit vector into a probability form;

[0032] The feature difference based on contrastive learning is:

[0033]

[0034] Where B is the batch size, τ is the temperature coefficient, is the sample label consistency indicator;

[0035] D(z i ,z j )=D logit (z i ,z j )+D repr (z i ,z j ).

[0036] Optionally, in the present invention, during the training phase of the neural network, the fusion loss is:

[0037]

[0038] in, is the downstream task loss after fusion, λ2 and λ3 are weighting factors used to control the importance of different constraints, is the distillation loss, is the reconstruction loss.

[0039] Optionally, the distillation loss of the present invention for:

[0040]

[0041] Among them, α j is the modal weight coefficient, w i→j is the normalized weight, calculated based on the feature similarity between samples.

[0042] Optionally, the fused features of the present invention are represented as:

[0043]

[0044] Among them, the overall weight of W1 and W2 represents that A dyn represents the edge weight matrix of the dynamic graph, D discrepancy represents the structural difference measure between modalities, and σ represents the activation function.

[0045] In the second aspect, the present invention provides a device for time alignment and dynamic fusion based on EEG and functional near-infrared signals, comprising a feature decoupling module, a variability time alignment module, and a graph distillation fusion module; wherein,

[0046] The feature decoupling module is used to decouple EEG features and functional near-infrared features into shared features and specific features;

[0047] The deformable time alignment module adaptively aligns shared features and specific features in time based on a deformable alignment mechanism with a learnable offset Δt;

[0048] The graph distillation module constructs a cross-modal graph structure between samples based on the time-aligned features, and introduces dynamic edge weights and dual difference metrics to achieve the fusion output of specific features and shared features.

[0049] Beneficial effects:

[0050] First, solve the delay alignment problem between modalities

[0051] The present invention introduces a deformable mechanism to model the temporal response offset between modalities, and can adaptively adjust the alignment of EEG and fNIRS according to the temporal characteristics of the sample, significantly improving the semantic matching and feature fusion quality.

[0052] Second, enhance cross-modal information collaboration capabilities

[0053] This paper constructs a dynamic fusion framework to guide deep semantic interaction between modalities, improves the multimodal emotion representation capability, and combines mechanisms such as graph distillation to optimize the efficiency of knowledge transfer between modalities, thereby achieving multi-level and multi-scale information fusion.

[0054] Third, consider both model robustness and generalization

[0055] While improving recognition accuracy, the present invention designs an alignment and fusion module with strong scalability and controllable parameters, which can adapt to different data sources and different task requirements, and has good practical promotion value and engineering deployment potential. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0057] Figure 1 Schematic diagram of modal time misalignment;

[0058] Figure 2 This is a framework diagram of the time alignment and dynamic fusion method based on EEG and functional near-infrared spectroscopy signals;

[0059] Figure 3 Schematic diagram of the feature extraction network and deformable alignment module;

[0060] Figure 4 This is a schematic diagram of the deformable Transformer structure;

[0061] Figure 5 Figure 2 is a diagram of the distillation structure. DETAILED DESCRIPTION

[0062] The embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0063] It should be noted that, in the absence of conflict, the following embodiments and features in the embodiments may be combined with each other; and, based on the embodiments in this disclosure, all other embodiments obtained by persons of ordinary skill in the art without creative work are within the scope of protection of this disclosure.

[0064] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein can be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on this disclosure, it should be understood by those skilled in the art that an aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects described herein can be used to implement an apparatus and / or practice a method. In addition, other structures and / or functionalities other than one or more of the aspects described herein can be used to implement this apparatus and / or practice this method.

[0065] The present invention provides a method for time alignment and dynamic fusion of EEG and fNIRS signals, aiming to solve the problems of inter-modal response delay, semantic inconsistency and low fusion efficiency in the process of multimodal brain signal fusion. Figure 2 As shown in the figure, this method combines the deformable feature alignment mechanism, the modal feature decoupling structure and the graph distillation enhancement strategy to systematically improve the collaborative modeling capabilities of EEG and fNIRS from the three levels of time, structure and semantics. It can be widely used in intelligent brain signal processing scenarios such as emotion recognition, depression screening, and brain-computer interface.

[0066] This paper proposes a temporal alignment and dynamic fusion method based on electroencephalogram (EEG) and functional near-infrared (fNIRS) signals, aiming to address the problems of asynchronous responses and insufficient information fusion between multimodal signals. This method combines three key modules: feature decoupling, deformable temporal alignment, and graph distillation fusion, to improve the expression consistency and discriminative ability of cross-modal features, with good generalization and application value. The specific process is as follows:

[0067] Neural network setup: A feature decoupling module, a variability time alignment module, and a graph distillation fusion module are set up. The feature decoupling module is used to decouple EEG features and functional near-infrared features into shared features and specific features. The deformable time alignment module adaptively aligns shared features and specific features based on a deformable alignment mechanism with a learnable offset Δt. The graph distillation module achieves the fusion output of specific features and shared features by constructing a cross-modal graph structure between samples and introducing dynamic edge weights and dual difference metrics.

[0068] Neural network training: Construct the EEG and functional near-infrared signal fusion loss function and perform neural network model training;

[0069] Feature fusion: The EEG and functional near-infrared signals to be fused are input into the trained neural network, and the neural network outputs the fusion features.

[0070] In this embodiment:

[0071] First, feature decoupling is achieved through the configuration of a feature decoupling module, decoupling the modal features of EEG and fNIRS into two categories: shared features and modality-specific features. Shared features represent stable representations that are consistently expressed across modalities and strongly correlated with task semantics, demonstrating cross-modal consistency. Specific features describe individual variations in each modality due to factors such as time delay and signal heterogeneity during acquisition, reflecting specific dynamic response patterns within the modality. This decoupling structure facilitates the extraction of semantically consistent but morphologically heterogeneous key features from multimodal signals while suppressing redundant interference information.

[0072] Secondly, a deformable time alignment module is used to achieve temporal alignment. Taking into account the inherent latency differences between EEG and fNIRS in their physiological response mechanisms, the present invention introduces a deformable attention mechanism to adaptively time-align specific and shared features. By learning an offset Δt, the response differences between modalities are dynamically captured, achieving semantic temporal alignment of cross-modal signals. This mechanism effectively enhances the ability of cross-modal features to collaboratively express each other in the temporal dimension.

[0073] Finally, a graph distillation fusion module is used to achieve feature fusion output. To further guide cross-modal structural consistency, this paper designs a feature fusion module based on graph distillation. By constructing a cross-modal graph structure between samples and introducing dynamic edge weights and a dual dissimilarity metric (logits KL divergence + feature contrast loss), consistent modeling of EEG and fNIRS in shared and specific spaces is achieved. This module not only promotes inter-modal knowledge transfer but also improves the discriminability and robustness of the model.

[0074] The following is a detailed description of the above process:

[0075] Step 1: Signal modeling and preprocessing

[0076] Assume that the input EEG signal and fNIRS signal are two-dimensional matrices respectively: Among them, C e and C f represents the number of channels of the two modalities, and T represents the uniform resampling length.

[0077] The original signal is bandpass filtered, window sliced, and normalized before being fed into the initial encoder to extract shallow representations:

[0078] F eeg =Enc eeg (X eeg ),F fnirs =Enc fnirs (X fnirs )

[0079] Here, Enc represents the initial encoder.

[0080] Step 2: Modal feature decoupling

[0081] Decouple the features of each modality into shared features and specific features:

[0082]

[0083] in, Represents shared semantics (such as emotional state) between modalities, that is, shared features, Represents modality-specific information, i.e., specific features.

[0084] The decoupling operation is achieved through 1×1 convolution channel projection:

[0085]

[0086] In order to keep the decoupled information intact, reconstruction loss is introduced:

[0087]

[0088] Among them, Conv rec Represents a reconstruction function to keep the dimensions consistent.

[0089] Step 3: Feature extraction and deep modeling

[0090] Deep modeling is performed on the decoupled features, which is performed in the shared feature space and the specific feature space respectively.

[0091] For shared features:

[0092]

[0093] For specific features:

[0094]

[0095] Each uses stacked convolution structure and deformable convolution to enhance local perception and semantic expression capabilities.

[0096] Step 4: Deformable Time Alignment Mechanism

[0097] Considering the temporal response delay differences between modalities, a deformable alignment mechanism with a learnable offset Δt is introduced in the two decoupled spaces. Using EEG and fNIRS as reference benchmarks, the temporal offset relationship between the modalities is dynamically modeled (since the alignment mechanisms for shared and specific features are the same, com and pri are not explicitly labeled for simplicity):

[0098]

[0099] Where Δt represents the temporal offset between modalities, which is adaptively learned by the network. Align(·) represents a deformable alignment module with a learnable offset, which achieves semantic temporal alignment and information compensation between modalities. Figure 3 The feature extraction network and deformable alignment structure are shown.

[0100] Step 5: Graph Distillation Guidance Module

[0101] (1) To enhance the cross-domain knowledge transfer capability between modalities, the present invention introduces a graph distillation mechanism in both the shared space and the specific space, and uses the graph structure distillation loss to implement the consistency constraint on the feature representation structure, such as Figure 4 As shown. In the shared space, build a cross-modal graph structure The node set represents all samples under EEG and fNIRS modalities, and the edge set ε represents the directed edges between different modalities. To simplify the subsequent formula expression, z i and z j Represent the EEG and fNIRS modal features, which are respectively the same as those in step 4 above. and Corresponding.

[0102] For the feature representation z under EEG modality i and the feature representation z under fNIRS modality j , define the error of the edge from i to j in the graph as:

[0103]

[0104] Where D(·) is the cross-modal difference metric function, and γ is a hyperparameter that adjusts the spatial distance regularization term.

[0105] The difference function D(·) is defined from two aspects:

[0106] Based on the KL divergence of probability distribution, it measures the inconsistency between different modalities at the classification probability level:

[0107]

[0108] in, Represents the classification prediction vector of modality m after linear mapping, and softmax() represents normalizing the logit vector into a probability form; Figure 5 shown.

[0109]

[0110] Among them, W logit is the linear transformation weight.

[0111] Feature difference loss based on contrastive learning is used to maintain the consistency of semantic representation between modalities. It is defined as follows:

[0112]

[0113] Where B is the batch size, τ is the temperature coefficient, is the sample label consistency indicator.

[0114] Finally, the graph distillation loss function is:

[0115] D(z i ,z j )=D logit (z i ,z j )+D repr (z i ,z j )

[0116] In order to enhance the adaptability of edge weights, a learnable edge weight mechanism is further introduced to calculate the normalized weight w of cross-modal edges based on the feature similarity between samples. i→j :

[0117]

[0118] Among them, f ca (·,·) represents the edge weight function based on feature similarity, Indicates that z i A set of connected neighbor nodes.

[0119] Finally, a single node v j The graph distillation loss can be defined as:

[0120]

[0121] The graph distillation loss of the entire image is a multimodal weighted sum:

[0122]

[0123] Among them, α j is the modality weight coefficient, which adjusts the contribution of each modality to the overall distillation objective. This graph distillation guidance mechanism dynamically adjusts the strength of knowledge transfer between modalities, enhancing the consistency and complementarity between modalities. While ensuring that differential features remain independent, it also improves the discriminative power of fused features in discriminative tasks.

[0124] Finally, the EEG and fNIRS fusion loss is defined as:

[0125]

[0126] in, is the downstream task loss after fusion (which can be emotion classification or motor imagery classification, etc.), and λ2 and λ3 are weighting factors used to control the importance of different constraints.

[0127] Based on the above strategy, the model finally outputs the fusion features guided by graph distillation:

[0128]

[0129] Among them, W1 and W2 represent the overall weight representation formed by the proposed strategy, A dyn represents the edge weight matrix of the dynamic graph, D discrepancy represents the structural difference measure between modalities. The two together form the difference guidance term, which serves as auxiliary information to guide the fusion. σ represents the activation function.

[0130] Fusion stands for fusion operation. and With z i and z j Equivalence, F fusion It is used for various downstream tasks, such as application and deployment in multimodal physiological signal processing tasks such as depression detection, emotion recognition, and neurofeedback.

[0131] This embodiment provides a device for time alignment and dynamic fusion based on EEG and functional near-infrared signals, including a feature decoupling module, a variability time alignment module, and a graph distillation fusion module; wherein,

[0132] The feature decoupling module is used to decouple EEG features and functional near-infrared features into shared features and specific features;

[0133] The deformable time alignment module adaptively aligns shared features and specific features in time based on a deformable alignment mechanism with a learnable offset Δt;

[0134] The graph distillation module constructs a cross-modal graph structure between samples based on the time-aligned features, and introduces dynamic edge weights and dual difference metrics to achieve the fusion output of specific features and shared features.

[0135] Existing multimodal brain-computer interface research has largely focused on the parallel modeling and static fusion of electroencephalogram (EEG) and functional near-infrared (fNIRS) features, ignoring the significant physiological differences in response time between modalities and the structural misalignment between EEG and fNIRS expression patterns. Furthermore, traditional methods often use fixed windows or simple linear mapping for fusion, which cannot fully capture the dynamic coordination and semantic synchronization characteristics between signals, thus limiting the performance ceiling of multimodal fusion.

[0136] In comparison, this application has the following significant technical advantages:

[0137] (1) Modal delay perception and dynamic alignment capabilities

[0138] This paper introduces a deformable temporal alignment mechanism that learns variable time offsets, using EEG and fNIRS as references to achieve feature-level alignment and compensation between modalities. This mechanism adaptively adjusts the modality alignment strategy based on specific tasks and data, effectively alleviating semantic misalignment caused by physiological delays.

[0139] (2) Structural decoupling and semantic consistency enhancement mechanism

[0140] Through the proposed feature decoupling method, intra-modal features are divided into shared features and unique features, respectively modeling their cross-modal consistency and intra-modal differences. This mechanism enhances the robust expression of shared semantics between modalities while avoiding fusion interference caused by feature redundancy and improving representation accuracy.

[0141] (3) Graph-guided cross-modal information transfer mechanism

[0142] This application innovatively introduces a graph distillation module, combining adjacency weight learning with contrast constraints to construct a cross-modal graph structure in a shared representation space. This guides semantic convergence between EEG and fNIRS features, enhancing knowledge transfer between modalities. Compared to traditional one-way distillation or simple concatenation and fusion methods, this mechanism better preserves the complementarity and synergy of modal structures.

[0143] (4) Efficient scalability and practicality

[0144] The method proposed in this invention has a clear structure and strong modularity. It is applicable to physiological signals with various modal combinations and different sampling rates, and can be flexibly embedded in various deep neural network architectures. It has good system scalability and practical deployment value.

[0145] Therefore, the present invention focuses on the two major technical difficulties of "cross-modal delay alignment" and "dynamic fusion collaboration", and proposes an EEG-fNIRS fusion solution with a reasonable mechanism, efficient structure, and wide application. It effectively makes up for the shortcomings of existing research and promotes the development of multimodal brain signal intelligent analysis technology.

[0146] In summary, the above are only preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for time alignment and dynamic fusion based on EEG and functional near-infrared signals, characterized in that: include Neural network setup: A feature decoupling module, a variability time alignment module, and a graph distillation fusion module are set up. The feature decoupling module is used to decouple EEG features and functional near-infrared features into shared features and specific features. The deformable time alignment module adaptively aligns shared features and specific features based on a deformable alignment mechanism with a learnable offset Δt. The graph distillation module achieves the fusion output of specific features and shared features by constructing a cross-modal graph structure between samples and introducing dynamic edge weights and dual difference metrics. Neural network training: Construct a loss function for fusion of EEG and functional near-infrared signals, and train the neural network model; Feature fusion: The EEG and functional near-infrared signals to be fused are input into the trained neural network, and the neural network outputs the fusion features.

2. The method for time alignment and dynamic fusion based on EEG and functional near-infrared signals according to claim 1 is characterized in that: The shared features are: stable representations that are consistently expressed in different modalities and strongly related to task semantics, with cross-modal consistency; the specific features are: The personalized changes of different modalities due to different factors during acquisition reflect the specific dynamic response laws within the modality.

3. The method for time alignment and dynamic fusion based on EEG and functional near-infrared signals according to claim 2 is characterized in that: The execution process of the feature decoupling module is as follows: first, the EEG signals and functional near-infrared (fNIRS) signals are bandpass filtered, window sliced, and normalized to extract shallow representations; second, the features of each modality are decoupled into shared features and specific features; finally, the decoupled features are deeply modeled.

4. The method for time alignment and dynamic fusion based on EEG and functional near-infrared signals according to claim 3 is characterized in that: The adaptive time alignment of shared features and specific features is specifically as follows: Assume that the shared features of deep modeling are represented as Specific features are represented as mod∈{eeg,fnirs}; then the time alignment is: Here, Δt represents the temporal offset between modalities, and Align(·) represents the semantic temporal alignment and information compensation between modalities.

5. The method for time alignment and dynamic fusion based on EEG and functional near-infrared signals according to claim 1, characterized in that: By constructing a cross-modal graph structure between samples, dynamic edge weights and dual difference metrics are introduced, specifically: Constructing cross-modal graph structures using time-aligned features Node Set represents all samples under EEG and fNIRS modalities, and the edge set ε represents the directed edges between different modalities; In the cross-modal graph, let the error of the edge from node i to node j be i→j for: Where D(·) is the cross-modal difference metric function, γ is the hyperparameter for adjusting the spatial distance regularization term, and z i and z j Represent EEG and fNIRS modal features respectively; cross-modal difference measurement functions include KL divergence based on probability distribution and feature difference based on contrastive learning.

6. The method for time alignment and dynamic fusion based on EEG and functional near-infrared signals according to claim 1, characterized in that: The KL divergence based on probability distribution is: in, Represents the classification prediction vector of modality m after linear mapping, and softmax() represents the normalization of the logit vector into a probability form; The feature difference based on contrastive learning is: Where B is the batch size, τ is the temperature coefficient, is the sample label consistency indicator; D(z i ,z j )=D logit (z i ,z j )+D repr (z i ,z j )。 7. The method for time alignment and dynamic fusion based on EEG and functional near-infrared signals according to claim 1, characterized in that: During the training phase of the neural network, the fusion loss is: in, is the downstream task loss after fusion, λ2 and λ3 are weighting factors used to control the importance of different constraints, is the distillation loss, is the reconstruction loss.

8. The method for time alignment and dynamic fusion based on EEG and functional near-infrared signals according to claim 7, characterized in that: The distillation loss for: Among them, α j is the modal weight coefficient, w i→j is the normalized weight, calculated based on the feature similarity between samples.

9. The method for time alignment and dynamic fusion based on EEG and functional near-infrared signals according to claim 4, characterized in that: The fused features are expressed as: Among them, the overall weight of W1 and W2 represents that A dyn represents the edge weight matrix of the dynamic graph, D discrepancy represents the structural difference measure between modalities, and σ represents the activation function.

10. A device for time alignment and dynamic fusion based on EEG and functional near-infrared signals, characterized in that: It includes feature decoupling module, variability time alignment module and graph distillation fusion module; among them, The feature decoupling module is used to decouple EEG features and functional near-infrared features into shared features and specific features; The deformable time alignment module adaptively aligns shared features and specific features in time based on a deformable alignment mechanism with a learnable offset Δt; The graph distillation module constructs a cross-modal graph structure between samples based on the time-aligned features, and introduces dynamic edge weights and dual difference metrics to achieve the fusion output of specific features and shared features.

Images