An fNIRS signal analysis method based on dynamic functional connection and 3DCNN-Transformer
By fusing dynamic functional connections with 3DCNN-Transformer, the challenge of dynamic interaction and recombination processes in fNIRS signal classification was solved, achieving high-precision multi-class task classification, improving classification accuracy, and providing interpretable neurophysiological evidence.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- CHANGSHA UNIVERSITY OF SCIENCE AND TECHNOLOGY
- Filing Date
- 2026-01-23
- Publication Date
- 2026-06-09
AI Technical Summary
Existing fNIRS signal classification methods are unable to fully capture the dynamic interaction and reorganization process of task-induced brain networks in the temporal dimension. Traditional methods independently process spatiotemporal information or only focus on static connection patterns, making it difficult to achieve high-precision classification.
By employing a fusion of dynamic functional connections and 3DCNN-Transformer, and using the dFC tensor as input, combined with local feature extraction from a 3D convolutional neural network and global temporal modeling from a Transformer, high-precision classification of multi-class tasks can be achieved.
It significantly improves the accuracy of fNIRS signal classification, especially on isometric grip strength and mental arithmetic task datasets, and provides more discriminative features and interpretable neurophysiological evidence.
Smart Images
Figure CN122174038A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of medical signal processing and analysis, and more specifically, to the classification study of functional near-infrared spectroscopy (fNIRS) signals, particularly the application of neural networks and deep learning methods in fNIRS signal analysis. Background Technology
[0002] fNIRS, a non-invasive and portable brain functional imaging technique, reflects neural activity by measuring the relative changes in the concentrations of oxyhemoglobin (HbO) and deoxyhemoglobin (HbR) in the cerebral cortex. It has important application value in motor task decoding, brain-computer interfaces, and rehabilitation assessment.
[0003] Traditional fNIRS signal classification methods often rely on manually extracted statistical features or shallow models that process static features. In recent years, deep learning models such as convolutional neural networks (CNNs), long short-term memory (LSTM) networks, and Transformers have been introduced to automatically extract features. However, these methods typically process spatiotemporal information independently or focus only on static connectivity patterns, making it difficult to fully capture the dynamic interactions and reorganization processes of task-induced brain networks over time. Dynamic Functional Connectivity (dFC) quantifies the time-varying characteristics of inter-brain interactions through sliding window analysis, providing a powerful tool for describing the evolution of brain network states at different stages of a task. Currently, combining full-time dFC tensors with end-to-end deep learning architectures (such as 3DCNNs and Transformers) to fully utilize dynamic information at the network level remains a direction for further exploration in the field of fNIRS research. Summary of the Invention
[0004] This invention aims to overcome the shortcomings of existing technologies and proposes a method for fNIRS signal analysis based on the fusion of dynamic functional connectivity (dFC) and 3DCNN-Transformer. This method constructs a task-related dFC tensor as input and designs a hybrid model combining the local feature extraction capabilities of a 3D convolutional neural network (3DCNN) with the global temporal modeling capabilities of a Transformer to achieve high-precision and highly interpretable classification of multi-category tasks. The core of this invention's technical solution lies in: collecting fNIRS signals from subjects performing different categories of tasks, preprocessing them to construct a dFC tensor; inputting this dFC tensor into an end-to-end 3DCNN-Transformer hybrid model, where the 3DCNN module first extracts local spatiotemporal connectivity patterns between channels, and the Transformer encoder module models the global temporal dependencies throughout the entire task execution cycle; finally, a fully connected classification layer is used to classify the fNIRS signals.
[0005] The specific steps of the method of the present invention are as follows:
[0006] The first step is to acquire and preprocess fNIRS signals. fNIRS signals are acquired from subjects performing the task. The raw signals undergo preprocessing steps including detrending, motion artifact correction, bandpass filtering, trial segmentation, baseline correction, and channel normalization to obtain high-quality HbO and HbR time series. The preprocessed functional near-infrared brain signals are then represented as… Where C represents the number of channels for functional near-infrared brain signals, D is 2, representing oxyhemoglobin signal and deoxyhemoglobin signal respectively, and T represents the number of sampling points within a predetermined time.
[0007] The second step, to characterize the time-varying properties of brain functional connectivity during the task, involves constructing a 4D dFC tensor using a typical sliding window strategy. The process is as follows:
[0008] (2.1) Preprocessed fNIRS signal of experiment n Set the time window length If there are s time points with a step size of s, then the total number of overlapping windows is .for In the l-th window, extract the HbO and HbR time series of all channels within the window, calculate the Pearson correlation coefficient between any two channels, and quantify the functional coupling strength between channels;
[0009] (2.2) Based on the correlation coefficient calculation results in (2.1), channel × channel functional connectivity matrices are constructed for HbO and HbR signals respectively, and then combined into a single-window functional connectivity tensor. Each All are symmetric tensors, and all diagonal elements are 1, ensuring the physical rationality of the functional connection representation;
[0010] (2.3) Connect the functions of all L windows in (2.2) into a tensor. Stack them sequentially in chronological order to obtain a 4D dFC tensor. This tensor fully preserves multi-dimensional information such as "spatial connectivity between channels, differences in signal types, and dynamic evolution over time," providing a structured input for subsequent 3DCNN extraction of local spatiotemporal features and Transformer capture of global temporal dependencies.
[0011] The third step involves inputting the 4D dFC tensor from the second step into the 3DCNN module. This module uses two concatenated sub-blocks of "3D convolution + 3D batch normalization (BatchNorm3D) + ReLU activation" to jointly model the spatial topology and short-term dynamic changes of the dFC, ultimately outputting a compact spatiotemporal feature representation rich in discriminative information. The process is shown below:
[0012] (3.1) The 4D dFC tensor obtained in the second step Input the first 3D convolutional layer. This layer uses a 3×3×3 three-dimensional convolutional kernel (spatial dimension × spatial dimension × temporal window dimension), with 16 kernels (output channels) and a stride of 1. The convolutional kernel slides along the spatial and temporal window dimensions of the tensor, simultaneously covering adjacent temporal windows, enabling the simultaneous capture of "spatial connectivity patterns of neighboring channel pairs + dynamic coupling features within a short time window," thus achieving joint extraction of local spatiotemporal dependencies.
[0013] (3.2) Apply 3D batch normalization and ReLU activation function to the spatiotemporal features output by (3.1) to stabilize and accelerate training, enhance nonlinear characteristics, and enhance the model's ability to express complex dFC patterns;
[0014] (3.3) For the features processed in (3.2), max pooling is performed only in the spatial dimension. The pooling kernel is 2×2×1. This pooling kernel only downsamples the spatial dimension. This operation compresses the spatial dimension and outputs the features of the first pooling layer. ,in The core design principle is to expand the spatial receptive field of the network without losing temporal information, and at the same time, to concentrate the model's representational ability on the temporal dynamic characteristics that are crucial to the dFC trajectory through feature compression, thereby generating a spatiotemporal feature map with higher density and lower redundancy.
[0015] (3.4) Repeat the process in (3.1)-(3.3), and further deepen the feature extraction through the second "3D convolution + 3D batch normalization + ReLU activation" sub-block. The number of convolution kernels in the second 3D convolutional layer is increased to 32, while maintaining a kernel size of 3×3×3, to continue capturing more complex local spatiotemporal combination patterns. After processing by BatchNorm3D and ReLU, the MaxPool3D operation is performed again, maintaining a pooling kernel size of 2×2×1, and finally the output of the second pooling layer is obtained. ,in .
[0016] The fourth step involves modeling global temporal dependencies using the Transformer module. The spatiotemporal feature sequences extracted and flattened by the 3DCNN are mapped to a fixed dimension, and positional encodings are added before being input into the Transformer encoder. The encoder's multi-head self-attention mechanism adaptively balances the importance between different time windows, capturing long-term contextual relationships across the entire trial. Finally, features from a specific location in the encoder's last layer output (e.g., the last time step) are taken as the global representation for the entire trial. The process is as follows:
[0017] (4.1) The local spatiotemporal feature map obtained in the third step Convert to a window-level embedding suitable for sequence modeling. The specific steps are as follows:
[0018] (4.1.1) For each time index The local spatiotemporal feature map obtained in the third step Flatten out the spatial dimensions;
[0019] (4.1.2) The flattened features from (4.1.1) are mapped to a fixed-dimensional embedding space through a learnable linear projection layer: ,in (Weight matrix) and (Bias vector) is a learnable parameter. To enable parameter setting, this step adapts the distribution characteristics of the fNIRS dynamic functional connectivity features through parameter learning, so that all window-level features have a unified dimension, laying the foundation for subsequent global temporal modeling.
[0020] (4.1.3) Combine the embedding vectors of all time windows in (4.1.2) in chronological order to obtain the sequence. The sequence contains the local spatiotemporal features of each window, but does not yet carry temporal order information.
[0021] (4.2) is the initial sequence of (4.1.3). Overlay position encoding The superposition yields an input sequence with temporal information. This ensures that the model can accurately capture the evolution of dFC over time (such as the window order relationship of the prompting period, execution period, and recovery period in the task).
[0022] (4.3) Global temporal dependencies are captured using a Transformer encoder. The encoder consists of six layers of identical coding units stacked together. Each layer contains a "multi-head self-attention sub-layer" and a "feedforward network sub-layer". Each sub-layer uses residual connections and layer normalization to stabilize training and alleviate gradient vanishing.
[0023] (4.4) Take the feature vector of the last time step from all time step features output by the last layer of the Transformer encoder. This vector serves as a global representation of the entire trial. It integrates the local dFC spatiotemporal pattern extracted by 3DCNN with the global temporal dependency captured by Transformer, fully encoding the dynamic brain network information of a single trial from the cueing phase, execution phase to the recovery phase, providing comprehensive feature support for subsequent classification tasks.
[0024] (4.5) Extract the self-attention matrix of the Transformer encoder and visualize the dependency weights between different windows through heatmaps. This method reveals the degree of attention the model pays to different task stages (such as prompting, execution, and recovery), providing a neurophysiological explanation for classification decisions.
[0025] The fifth step, based on the global representation obtained in the previous step, outputs the task category probability through a structured classification head. Combined with attention visualization and feature statistical analysis, this provides an interpretable neurophysiological basis for the classification results, ensuring that the model performance and decision logic are both scientifically sound and credible. The process is as follows:
[0026] (5.1) Global representation of the Transformer encoder output First, the features are mapped to a 64-dimensional space through a fully connected layer, then nonlinearity is introduced through the ReLU activation function, and finally a Dropout layer is added to suppress overfitting.
[0027] (5.2) The (5.1) process is passed through the final fully connected classification layer, mapped to the task category dimension K, and then converted into a probability distribution by the Softmax function.
[0028] The main effect and advantage of this invention lies in its ability to effectively improve the accuracy of fNIRS signal classification. When classifying fNIRS signals from 27 volunteers in a self-built isometric grip strength dataset across three isometric grip strength tasks (20%, 50%, and 80% maximum voluntary contraction), the proposed method, DFC-3DCT, demonstrates superior classification performance under the same initial conditions. The leave-one-out cross-validation accuracy reaches 63.27% ± 5.52%, and the within-subjects validation accuracy reaches 76.30% ± 4.22%. It also achieves excellent results on the publicly available mental arithmetic (MA) dataset (leave-one-out cross-validation accuracy 78.51% ± 6.83%, and within-subjects validation accuracy 85.80 ± 7.23%), significantly outperforming baseline models based on raw signals or static functional connections, demonstrating its powerful ability to handle fine motor tasks and generalize across paradigms. The core advantage of this method lies in its full exploitation of dynamic information. Using the dFC tensor as input, it directly models the time-varying interactive characteristics of the brain network during the task process, thus providing more discriminative features. Simultaneously, through a complementary architecture of 3DCNN and Transformer, it achieves multi-level, multi-scale information fusion mining of local spatiotemporal patterns and global temporal dependencies. Furthermore, the model exhibits good interpretability; its output attention weight map can intuitively correlate different execution stages of the task, revealing the dynamic reorganization patterns of the brain network under different grip strengths, providing credible neural evidence for clinical applications such as rehabilitation assessment. Attached Figure Description
[0029] Attached Figure Figure 1 This is the overall flowchart of the method of the present invention, including data acquisition and preprocessing, dFC tensor construction, DFC-3DCT model processing (including 3DCNN module, Transformer module and classification head) and result output and visualization. Detailed Implementation
[0030] A specific embodiment of the present invention will now be described in detail with reference to the technical solution and accompanying drawings.
[0031] This study used a self-collected isometric grip strength fNIRS dataset (IGF dataset) containing 27 healthy subjects. Each subject performed a maximum voluntary contraction grip strength task at three levels: 20%, 50%, and 80%, repeated 20 times at each level, generating a total of 1620 valid trials. The dataset had 20 channels, a sampling frequency of 22.2 Hz, and 400 sampling points, including both oxyhemoglobin and deoxyhemoglobin concentration data.
[0032] As shown in the attached figure, the model designed in this study mainly includes the following processing flow:
[0033] The first step involved acquiring and preprocessing fNIRS signals. fNIRS signals were acquired from subjects performing multi-level isometric grip strength tasks. The raw signals underwent preprocessing steps including detrending, motion artifact correction, bandpass filtering, trial segmentation, baseline correction, and channel normalization to obtain high-quality HbO and HbR time series. The preprocessed functional near-infrared brain signals were then represented as… ;
[0034] The second step, to characterize the time-varying properties of brain functional connectivity during the grip strength task, involves constructing a 4D dFC tensor using a typical sliding window strategy. The process is shown below:
[0035] (2.1) Preprocessed fNIRS signal of experiment n If the time window length is set to 110 and the step size is 1 time point, then the total number of overlapping windows is . .for In the l-th window, extract the HbO and HbR time series of all channels within the window, calculate the Pearson correlation coefficient between any two channels, and quantify the functional coupling strength between channels;
[0036] (2.2) Based on the correlation coefficient calculation results in (2.1), channel × channel functional connectivity matrices are constructed for HbO and HbR signals respectively, and then combined into a single-window functional connectivity tensor. Each All are symmetric tensors, and all diagonal elements are 1, ensuring the physical rationality of the functional connection representation;
[0037] (2.3) Connect the functions of all 291 windows in (2.2) into a tensor. Stack them sequentially in chronological order to obtain a 4D dFC tensor. This tensor fully preserves multi-dimensional information such as "spatial connectivity between channels, differences in signal types, and dynamic evolution over time," providing a structured input for subsequent 3DCNN extraction of local spatiotemporal features and Transformer capture of global temporal dependencies.
[0038] The third step involves inputting the 4D dFC tensor from the second step into the 3DCNN module. This module uses two concatenated sub-blocks of "3D convolution + 3D batch normalization (BatchNorm3D) + ReLU activation" to jointly model the spatial topology and short-term dynamic changes of the dFC, ultimately outputting a compact spatiotemporal feature representation rich in discriminative information. The process is shown below:
[0039] (3.1) The 4D dFC tensor obtained in the second step Input the first 3D convolutional layer. This layer uses a 3×3×3 three-dimensional convolutional kernel (spatial dimension × spatial dimension × temporal window dimension), with 16 kernels (output channels) and a stride of 1. The convolutional kernel slides along the spatial and temporal window dimensions of the tensor, simultaneously covering adjacent temporal windows, enabling the simultaneous capture of "spatial connectivity patterns of neighboring channel pairs + dynamic coupling features within a short time window," thus achieving joint extraction of local spatiotemporal dependencies.
[0040] (3.2) Apply 3D batch normalization and ReLU activation function to the spatiotemporal features output by (3.1) to stabilize and accelerate training, enhance nonlinear characteristics, and enhance the model's ability to express complex dFC patterns.
[0041] (3.3) For the features processed in (3.2), max pooling is performed only in the spatial dimension. The pooling kernel is 2×2×1. This pooling kernel only downsamples the spatial dimension. This operation compresses the spatial dimension and outputs the features of the first pooling layer. The core design principle is to expand the spatial receptive field of the network without losing temporal information, and at the same time, to concentrate the model's representational ability on the temporal dynamic characteristics that are crucial to the dFC trajectory through feature compression, thereby generating a spatiotemporal feature map with higher density and lower redundancy.
[0042] (3.4) Repeat the process in (3.1)-(3.3), and further deepen the feature extraction through the second sub-block of "3D convolution + 3D batch normalization + ReLU activation". The number of convolution kernels in the second 3D convolutional layer is increased to 32, while maintaining a kernel size of 3×3×3, to continue capturing more complex local spatiotemporal combination patterns. After processing by BatchNorm3D and ReLU, the MaxPool3D operation is performed again, maintaining a pooling kernel size of 2×2×1, and finally the output of the second pooling layer is obtained. .
[0043] The fourth step involves modeling global temporal dependencies using the Transformer module. The spatiotemporal feature sequences extracted and flattened by the 3DCNN are mapped to a fixed dimension, and positional encodings are added before being input into the Transformer encoder. The encoder's multi-head self-attention mechanism adaptively balances the importance between different time windows, capturing long-term contextual relationships across the entire trial. Finally, features from a specific location in the encoder's last layer output (e.g., the last time step) are taken as the global representation for the entire trial. The process is as follows:
[0044] (4.1) The local spatiotemporal feature map obtained in the third step Convert to a window-level embedding suitable for sequence modeling. The specific steps are as follows:
[0045] (4.1.1) For each time index The local spatiotemporal feature map obtained in the third step Flatten out the spatial dimensions;
[0046] (4.1.2) The flattened features from (4.1.1) are mapped to a fixed-dimensional embedding space through a learnable linear projection layer: ,in (Weight matrix) and (Bias vector) is a learnable parameter. Setting it to 128, this step adapts the distribution characteristics of the fNIRS dynamic functional connectivity features through parameter learning, so that all window-level features have a unified dimension, laying the foundation for subsequent global temporal modeling.
[0047] (4.1.3) Combine the embedding vectors of all time windows in (4.1.2) in chronological order to obtain the sequence. The sequence contains the local spatiotemporal features of each window, but does not yet carry temporal order information.
[0048] (4.2) is the initial sequence of (4.1.3). Overlay position encoding The superposition yields an input sequence with temporal information. This ensures that the model can accurately capture the evolution of dFC over time (such as the window sequence relationship of the cueing phase, execution phase, and recovery phase in the grip strength task).
[0049] (4.3) Global temporal dependencies are captured using a Transformer encoder. The encoder consists of six layers of identical coding units stacked together. Each layer contains a "multi-head self-attention sub-layer" and a "feedforward network sub-layer". Each sub-layer uses residual connections and layer normalization to stabilize training and alleviate gradient vanishing.
[0050] (4.4) Take the feature vector of the last time step from all time step features output by the last layer of the Transformer encoder. This vector serves as a global representation of the entire trial. It integrates the local dFC spatiotemporal pattern extracted by 3DCNN with the global temporal dependency captured by Transformer, fully encoding the dynamic brain network information of a single grip strength trial from the cueing phase, execution phase to the recovery phase, providing comprehensive feature support for subsequent classification tasks.
[0051] (4.5) Extract the self-attention matrix of the Transformer encoder and visualize the dependency weights between different windows through heatmaps. This method reveals the degree of attention the model pays to different task stages (such as prompting, execution, and recovery), providing a neurophysiological explanation for classification decisions.
[0052] The fifth step, based on the global representation obtained in the previous step, outputs the grip strength level probability through a structured classification head. Combined with attention visualization and feature statistical analysis, this provides an interpretable neurophysiological basis for the classification results, ensuring that the model performance and decision logic are both scientifically sound and credible. The process is as follows:
[0053] (5.1) Global representation of the Transformer encoder output First, the features are mapped to a 64-dimensional space through a fully connected layer, then nonlinearity is introduced through the ReLU activation function, and finally a Dropout layer is added to suppress overfitting.
[0054] (5.2) The 64-dimensional features of (5.1) are passed through the final fully connected classification layer and mapped to the grip strength category dimension 3, and then converted into a probability distribution by the Softmax function.
[0055] The method described in this invention is not limited to the specific embodiments described above. Those skilled in the art can adjust or modify the specific structural parameters of the model, the dFC construction method, etc., within the framework of the technical solution disclosed in this invention, and all such adjustments should be included within the protection scope of this invention.
Claims
1. A method for fNIRS signal analysis based on dynamic functional connectivity and 3DCNN-Transformer, characterized in that, By constructing a dynamic functional connectivity tensor as model input and combining it with a 3D convolutional neural network and a Transformer encoder, dynamic feature extraction and classification of brain functional networks under multiple categories of tasks are achieved. Includes the following steps: The first step, signal acquisition and preprocessing, involved acquiring functional near-infrared spectroscopy (fNIRS) signals from subjects performing different types of tasks. These signals underwent preprocessing, including detrending, motion artifact correction, bandpass filtering, trial segmentation, baseline correction, and channel normalization. This yielded a multi-channel time series containing both oxyhemoglobin (HbO) and deoxyhemoglobin (HbR) signals, represented as a three-dimensional tensor. Where C represents the number of channels for functional near-infrared brain signals, D is 2, representing oxyhemoglobin signal and deoxyhemoglobin signal respectively, and T represents the number of sampling points within a predetermined time. The second step is the construction of the dynamic functional connectivity tensor: The three-dimensional signal tensor is segmented using a sliding time window method. Within each time window, the Pearson correlation coefficient between all channel pairs is calculated to obtain a matrix reflecting the instantaneous functional connectivity between brain regions. The connectivity matrices of each time window are then stacked in chronological order to construct a four-dimensional dynamic functional connectivity tensor. (Number of channels × Number of signal types × Number of time windows), where L is the number of time windows, which is used as the model input; The third step is local spatiotemporal feature extraction: the 4D dynamic function concatenation tensor from the second step is input into the 3DCNN module. This module uses two concatenated "3D convolution + 3D batch normalization (BatchNorm3D) + ReLU activation" sub-blocks to jointly model the spatial topology and short-term dynamic changes of the dFC, and finally outputs a compact spatiotemporal feature representation rich in discriminative information. The fourth step is global temporal dependency modeling and visualization: The spatiotemporal feature map output from the third step is converted into a feature sequence, which is then linearly projected and positionally encoded before being input into the Transformer encoder. The encoder's multi-head self-attention mechanism adaptively balances the importance between different time windows, capturing long-term contextual relationships across the entire trial. Finally, the features at a specific location in the encoder's last layer output (such as the last time step) are taken as the global representation for the entire trial. Simultaneously, the weight matrix of the multi-head self-attention mechanism is extracted to generate an attention heatmap representing the model's temporal attention patterns. The fifth step is classification decision: Based on the global representation obtained in the fourth step, the task category probability is output through a structured classification head, and attention visualization and feature statistical analysis are combined to provide interpretable neurophysiological basis for the classification results, ensuring that the model performance and decision logic are both scientific and credible.
2. The dynamic functional connectivity tensor construction method in the fNIRS signal analysis method based on dynamic functional connectivity and 3DCNN-Transformer according to claim 1, characterized in that, The method specifically includes the following steps: The first step is to represent the preprocessed single-trial fNIRS signal as a three-dimensional tensor, with the dimensions being the number of channels C × the number of signal types D × the number of time points T. The second step is to set a fixed length. Using a sliding window with a fixed step size s, the three-dimensional tensor is segmented along the time dimension to obtain L overlapping time window data blocks; The third step is to calculate the Pearson correlation coefficients of the HbO signal component and the HbR signal component between all pairs of channels for the data block of the l-th time window, forming two C × C correlation matrices. The fourth step is to stack the D C × C correlation matrices corresponding to the L time windows in chronological order to construct a four-dimensional dynamic functional connectivity tensor with dimensions C × C × D × L.
3. A local spatiotemporal feature extraction module in the fNIRS signal analysis method based on dynamic functional connectivity and 3DCNN-Transformer according to claim 1, characterized in that, This module, a 3D convolutional neural network module, is used to obtain local spatiotemporal features from the fNIRS dynamic functional connectivity tensor. The specific steps are as follows: The first step involves inputting the constructed dynamic functional connectivity tensor into the first 3D convolutional layer. A 3D convolutional kernel of a preset size is then used to perform convolution operations along the channel, signal type, and time window dimensions to extract preliminary local spatiotemporal connectivity patterns. The kernel size of the 3D convolutional layer is [size missing]. , For the size of the spatial kernel, The kernel size is the time dimension; The second step involves performing three-dimensional batch normalization and ReLU activation function mapping on the output of the first step to stabilize the training process and enhance the nonlinear representation capability. The third step is to input the feature map output from the second step into a three-dimensional max pooling layer, and downsample only in the spatial dimension to expand the receptive field and aggregate features while keeping the temporal dimension unchanged. The fourth step involves inputting the feature map output from the third step into the second 3D convolutional layer and the subsequent second 3D max pooling layer to further refine higher-level local spatiotemporal features, ultimately outputting a dense spatiotemporal feature map for sequence modeling.
4. A global time-dependent modeling and visualization module in the fNIRS signal analysis method based on dynamic functional connectivity and 3DCNN-Transformer according to claim 1, characterized in that, This module, the Transformer encoder module, is used to model global temporal dependencies from spatiotemporal feature sequences extracted from 3DCNNs and extract the model's attention patterns to provide a visual explanation of the task stages. Specifically, it performs the following operations: The first step is to flatten and reassemble the local spatiotemporal feature map output by the 3D convolutional neural network module along the spatial dimension to obtain the feature sequence. , where M is the feature dimension; The second step is to project the feature sequence through a linear projection layer. Mapping to dimensions is sequence And add a positional code to each position to obtain the encoded sequence. ; The third step is to process the encoded sequence. The input consists of a Transformer encoder composed of N stacked coding layers; each coding layer contains a multi-head self-attention sublayer and a feedforward neural network sublayer, and all employ residual connections and layer normalization; The fourth step is to extract the multi-head self-attention weight matrix of at least one Transformer coding layer from the forward propagation process after the model training is completed. This matrix quantifies the strength of the dependency between different time windows. The fifth step is to normalize the extracted attention weight matrix and average it according to category or task conditions to generate a two-dimensional heatmap representing the model’s temporal attention pattern. The sixth step is to generate an attention difference map by calculating the differences in the average attention heatmaps under different task conditions, so as to visualize the shifting pattern of the model's attention as the task category changes. The seventh step is to extract the feature vector of the last time step from the feature sequence output by the final layer Transformer encoder, and use it as the global temporal representation for the entire trial for classification.
5. A classification decision module in the fNIRS signal analysis method based on dynamic functional connectivity and 3DCNN-Transformer according to claim 1, characterized in that, This module is used to map the global temporal representation output by the Transformer module to specific task class probabilities and prevent model overfitting. It is a fully connected classification head and performs the following operations: The first step is to receive the global time representation vector output by the global time dependency modeling module; The second step is to input the vector into the first fully connected layer for nonlinear transformation and then pass it through the ReLU activation function; The third step is to apply a Dropout operation to the activated features, randomly discarding some neuron connections to enhance the model's generalization ability. The fourth step is to input the features after Dropout into the second fully connected layer and map their dimensions to match the number of categories. The fifth step is to apply the Softmax function to the output of the second fully connected layer to obtain the predicted probability distribution of the fNIRS signal belonging to each task category in this trial, thus completing the classification decision.