Efficient space-time CNN-Transform fusion network model suitable for fNIRS classification

Abstract

The invention discloses a deep learning model fNIRS-STCT fusing time CNN, space CNN and Transform, and aims to improve the classification accuracy of fNIRS on the premise of considering delayed hemodynamic response, space channel and time information of fNIRS signals. According to the technical scheme, an fNIRS signal passes through a time feature extraction module containing a time CNN and an improved Transform and a space extraction module containing a space CNN and an improved Transform, and time features and space features of the fNIRS are effectively extracted respectively. And then, the time features and the space features are fused, and fNIRS data are classified.

Description

Technical Field

[0001] The present invention relates to the field of medical signal processing and analysis, and more specifically, to the classification research of functional near-infrared spectroscopy (fNIRS) signals, especially the application of neural networks and deep learning methods in fNIRS signal analysis. Background Art

[0002] fNIRS is a neuroimaging technique that analyzes brain activity by measuring cerebral hemodynamic responses and has become a useful tool for monitoring brain activity. It measures the intensity of near-infrared light dispersed through cortical tissue by diffusion to quantify changes in the concentrations of oxyhemoglobin (HbO) and deoxyhemoglobin (HbR) in cortical regions. Compared to other functional brain imaging techniques, fNIRS maintains a balance between temporal and spatial resolution in brain imaging and has irreplaceable advantages. This model has been increasingly used in neuropsychiatric research, including patients with schizophrenia and anxiety disorders. In addition, due to its portability and ease of use, fNIRS has received widespread attention in the field of brain-computer interfaces.

[0003] Methods for classifying fNIRS signals include traditional machine learning and the emerging deep learning approach. Machine learning methods, such as support vector machines, linear discriminant analysis, and artificial neural networks, have been widely used. However, traditional classifiers rely heavily on manually extracted statistical values ​​(mean, variance, slope, kurtosis, peak, and skewness). In recent years, deep learning has made significant progress and has been increasingly used in fNIRS classification research. Convolutional neural networks (CNNs), long short-term memory (LSTM), and Transformers have been used for fNIRS classification. Although many complex models have emerged, the improvement in classification performance of these existing models is limited due to the lack of consideration of the inherent delayed hemodynamic response of fNIRS signals and the failure to comprehensively utilize the spatial and temporal information of fNIRS signals. Summary of the Invention

[0004] This paper proposes a deep learning model, fNIRS-STCT, that integrates temporal CNN, spatial CNN, and Transformer. Its purpose is to improve fNIRS classification accuracy while taking into account the delayed hemodynamic response, spatial channels, and temporal information of the fNIRS signal. The technical solution of the present invention is to pass the fNIRS signal through a temporal feature extraction module comprising a temporal CNN and an improved Transformer, and a spatial extraction module comprising a spatial CNN and an improved Transformer, respectively, to effectively extract the temporal and spatial features of the fNIRS. The temporal and spatial features are then fused, and the fNIRS signal is classified using a cross-entropy loss function.

[0005] The specific steps of the method of the present invention are as follows:

[0006] Step 1: Input fNIRS signal where C is the number of fNIRS channels, T is the number of sampling points within a certain time, and D is 2, representing the concentration of oxygenated hemoglobin and deoxygenated hemoglobin, respectively.

[0007] The second step is to input the fNIRS signal X into the temporal feature extraction module to extract temporal features. The temporal feature extraction module consists of a temporal CNN (ConvT) with large convolution kernels and a Transformer. The process is as follows:

[0008] (2.1) The fNIRS signal is input into the ConvT module to extract temporal features. The kernel length of the ConvT module is set to the number of sampling points T, and the kernel width is 1, that is, the kernel size of ConvT is 1×T;

[0009] (2.2) Input the temporal feature vector extracted by (2.1) into the channel attention mechanism;

[0010] (2.3) The temporal features of (2.2) are input into the improved Transformer module for further processing to further mine temporal information. The specific steps of the improved Transformer module are as follows:

[0011] (2.3.1) Prepend the classification tag [CLS] to the vector. [CLS] will extract information from the vector that helps with classification in the subsequent multi-head self-attention mechanism (MHSA). Then, add a positional encoding to the classification tag [CLS] and the feature vector. The positional encoding is a learnable one-dimensional embedding that represents the positional information of the [CLS] tag and the feature vector. The positional information can be used to distinguish the classification tag [CLS] from the feature vector.

[0012] (2.3.2) Input the classification label [CLS] and feature vector into MHSA;

[0013] (2.3.3) Update the classification label [CLS] using the output of MHSA, but do not update the feature vector. Normalize the classification label [CLS] and add the classification label [CLS] of (2.3.1);

[0014] (2.3.4) The classification label [CLS] and the feature vector are fed into a feedforward network consisting of two linear transformation layers with a GeLU activation function in between. The classification label [CLS] is normalized and added to the classification label [CLS] in (2.3.3). The feature vector is normalized and added to the feature vector in (2.3.1).

[0015] (2.3.5) Repeat (2.3.2) to (2.3.5). The number of repetitions is the number of layers, which is a configurable parameter.

[0016] (2.3.6) The final output of Transformer consists of two parts, one is the multi-layer encoding result of the classification tag [CLS], and the other is the embedded multi-layer encoding result. For classification tasks, only the part corresponding to the tag [CLS] needs to be extracted.

[0017] (2.4) Concatenate the temporal features of (2.2) and the multi-layer encoding results of the classification label [CLS] of (2.3), and linearly transform them into the dimension corresponding to the category.

[0018] The third step is to input the fNIRS signal X into the spatial feature extraction module to extract spatial features. The spatial feature extraction module consists of a spatial CNN (ConvS) with a large convolution kernel and a Transformer. The process is as follows:

[0019] (3.1) The fNIRS signal is input into the ConvS module to extract spatial features. The kernel length of the ConvS module is set to the number of sampling channels C, and the kernel width is as short as possible to aggregate the spatial information of multi-channel fNIRS. Ideally, the kernel size of ConvS is C × (T / C);

[0020] (3.2) The spatial features are input into the improved Transformer module (consistent with step 2.3) for further processing to further mine spatial information;

[0021] (3.3) Concatenate the spatial features of (3.1) and the multi-layer encoding results of the classification label [CLS] of (3.2) and linearly transform them into the dimension corresponding to the category.

[0022] Step 4: Input the temporal features obtained in step 2 and the spatial features obtained in step 3 into the temporal and spatial feature fusion module for joint classification. The specific steps are as follows:

[0023] (4.1) Input the spatial feature vector outputted in the third step into the Tanh function;

[0024] (4.2) Input the spatial feature vector outputted in the third step into the Sigmoid function and multiply it by the square of the result of (4.1);

[0025] (4.3) Add the time feature vector output from the second step to the result of (4.2);

[0026] (4.4) Input the result of (4.3) into the Softmax function to obtain the probability that the fNIRS signal belongs to each category.

[0027] Step 5: Use cross entropy as the loss function and adjust the loss function in combination with the online label smoothing method and flooding strategy to avoid overfitting. After multiple training cycles, output the classification result with the highest accuracy.

[0028] The main effect and advantage achieved by the present invention is that it can effectively improve the accuracy of classification. When classifying the fNIRS signals of the three actions of right hand finger tapping, left hand finger tapping and foot tapping of 30 volunteers in the public unilateral finger and toe tapping dataset, under the premise of ensuring the same initial conditions, the model fNIRS-STCT proposed by the present invention showed superior classification performance. fNIRS-T is a Transformer-based model and is the most accurate fNIRS classification model currently available. There are three different evaluation schemes: subject-dependent, where both the training set and the test set are from the same subject, ensuring that the model effectively learns the unique brain activity pattern of a specific individual; subject-semi-dependent, where all samples from all subjects are mixed together and then divided into training sets and test sets, ensuring that the model is effective and consistent across different individuals; subject-independent, where the training set and test set are from different subjects, ensuring its versatility for individuals that did not appear in the training set, so that it can be applied to any new subject. Across the three evaluation scenarios, fNIRS-STCT achieved accuracy rates of 82.53%, 81.7%, and 79.60%, respectively, outperforming fNIRS-T by 3%, 2%, and 1%, respectively. This model is highly applicable to real-world scenarios, contributing to the development of disease diagnosis and brain-computer interfaces, and possesses significant research and application value. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Attached photos Figure 1 This is an overview diagram of the model architecture proposed in this invention. DETAILED DESCRIPTION

[0030] A specific implementation example of the present invention is described in detail below in conjunction with the technical solution and the accompanying drawings.

[0031] This study used a publicly available unilateral finger and toe tapping fNIRS dataset, which includes 25 right finger taps, 25 left finger taps, and 25 foot taps from 30 volunteers, totaling 2250 samples. The dataset has 20 channels and 258 sampling points, including both oxygenated and deoxygenated hemoglobin concentration data, to construct 2250 three-dimensional tensors.

[0032] As shown in the attached figure, the model designed for this study mainly includes the following processing procedures:

[0033] Step 1: Input fNIRS signal

[0034] The second step is to input the fNIRS signal X into the temporal feature extraction module to obtain temporal features. The temporal feature extraction module consists of a temporal CNN (ConvT) with a large convolution kernel and a Transformer. The process is as follows:

[0035] (2.1) The fNIRS signal is input into the ConvT module to extract temporal features. The kernel size of ConvT is 1×258, the input channels are 2, and the output channels are 8. The convolution result is a 20×1×8 tensor, and the temporal features are arranged into a vector of size 1×160.

[0036] (2.2) Input the temporal feature vector extracted by (2.1) into the channel attention mechanism, and the feature size remains unchanged;

[0037] (2.3) The temporal feature vector from (2.2) is input into the improved Transformer module for further processing to further mine temporal information. The specific steps of the improved Transformer module are as follows:

[0038] (2.3.1) A classification mark [CLS] of size 1×160 is concatenated before the vector. After concatenation, the size of the classification mark [CLS] and the vector is 2×160. Then, a positional code of size 2×160 is added to the classification mark [CLS] and the feature vector.

[0039] (2.3.2) The classification label [CLS] and feature vector are fed into the multi-head self-attention mechanism, and the feature size remains unchanged;

[0040] (2.3.3) Use the output of the multi-head self-attention mechanism to update the classification label [CLS], but do not update the feature vector. Normalize the classification label [CLS] and add the classification label [CLS] of (2.3.1), and the feature size remains unchanged;

[0041] (2.3.4) The classification label [CLS] and the feature vector are fed into a feedforward network consisting of two linear transformation layers with a GeLU activation function in between. The classification label [CLS] is normalized and added with the classification label [CLS] from (2.3.3). The feature vector is normalized and added with the feature vector from (2.3.1), leaving the feature size unchanged.

[0042] (2.3.5) Repeat (2.3.2) to (2.3.5). The number of repetitions is 5, which is a configurable parameter.

[0043] (2.3.6) The final output of Transformer consists of two parts, one is the multi-layer encoding result of the classification tag [CLS], and the other is the embedded multi-layer encoding result. For the task, only the part corresponding to the tag [CLS] needs to be extracted, and the temporal feature size is still 1×160.

[0044] (2.4) Concatenate the temporal features of (2.2) with the multi-layer encoding results of the classification mark [CLS] of (2.3). After concatenation, the temporal feature size is 1×320 and is linearly transformed into the dimension corresponding to the category, which is 1×3.

[0045] The third step is to input the fNIRS signal X into the spatial feature extraction module to obtain spatial features. The spatial feature extraction module consists of a spatial CNN (ConvS) with a large convolution kernel and a Transformer. The steps are as follows:

[0046] (3.1) The fNIRS signal is input into the ConvS module to extract spatial features. The kernel size of ConvS is 40 × 6. The input channels are 2 and the output channels are 8. The convolution result is a 1 × 43 × 8 tensor, which is linearly transformed into a 1 × 20 × 8 tensor. The spatial features are arranged into a 1 × 160 vector.

[0047] (3.2) The spatial features are input into the improved Transformer module (consistent with step 2.3) for further processing to further mine spatial information. The final output of the Transformer consists of two parts: one is the multi-layer encoding result of the classification label [CLS], and the other is the embedded multi-layer encoding result. For the task, only the part corresponding to the label [CLS] needs to be extracted. The spatial feature size is 1×160;

[0048] (3.3) The spatial features of (3.1) are concatenated with the multi-layer encoding results of the classification mark [CLS] of (3.2). The feature size after concatenation is 1×320, and it is linearly transformed into the dimension corresponding to the category, which is 1×3.

[0049] Step 4: Input the temporal features obtained in step 2 and the spatial features obtained in step 3 into the temporal and spatial feature fusion module for joint classification. The specific steps are as follows:

[0050] (4.1) Input the spatial feature vector output in the third step into the Tanh function, and the feature size remains unchanged;

[0051] (4.2) Input the spatial feature vector output in the third step into the Sigmoid function and multiply it by the square of the result of (4.1). The feature size remains unchanged.

[0052] (4.3) Add the time feature vector output from the second step to the result of (4.2), and the feature size is still 1×3;

[0053] (4.4) Input the result of (4.3) into the Softmax function to obtain the probability that the fNIRS signal belongs to each category, and the feature size remains unchanged.

[0054] Step 5: Use cross entropy as the loss function and adjust the loss function in combination with the online label smoothing method and flooding strategy to avoid overfitting. After 120 training cycles, output the classification result with the highest accuracy.

Claims

1. An efficient spatiotemporal CNN-Transformer fusion network model suitable for fNIRS classification, characterized by: The fNIRS data are respectively passed through a temporal feature extraction module containing a temporal convolutional neural network (CNN) and an improved Transformer, and a spatial feature extraction module containing a spatial CNN and an improved Transformer, to effectively extract the temporal and spatial features of fNIRS. Then, the temporal and spatial features are fused to classify the fNIRS data. The following steps are involved: In the first step, the input fNIRS signal is expressed as Where C is the number of fNIRS channels, T is the number of sampling points in a certain period of time, and D is 2 (representing the concentration of oxygenated hemoglobin and deoxygenated hemoglobin); In the second step, the fNIRS signal is input into a temporal feature extraction module to extract the temporal features of fNIRS. The temporal feature extraction module consists of a temporal CNN with a large convolution kernel (abbreviated as ConvT) and an improved Transformer module. In the third step, the fNIRS signal is input into a spatial feature extraction module to extract the spatial features of fNIRS. The spatial feature extraction module consists of a spatial CNN with large convolution kernels (abbreviated as ConvS) and an improved Transformer module. In the fourth step, the temporal features output in the second step and the spatial features output in the third step are input into a temporal feature and spatial feature fusion module to obtain the classification probability of each category, which are then used together for classification. In the fifth step, cross entropy is used as the loss function, and the loss function is adjusted in combination with the online label smoothing method and the flooding strategy to avoid overfitting. After multiple training cycles, the classification result with the highest accuracy is output.

2. A temporal feature extraction module in an efficient spatiotemporal CNN-Transformer fusion network model suitable for fNIRS classification according to claim 1, characterized in that: To obtain temporal features from fNIRS signals, the temporal feature extraction module consists of a temporal CNN with large convolution kernels (ConvT for short) and an improved Transformer. The specific steps are as follows: In the first step, the fNIRS signal is input into the ConvT module to extract temporal features. The kernel length of the ConvT module is set to the number of sampling points T, and the kernel width is 1, that is, the kernel size of ConvT is 1×T; In the second step, the temporal feature vector extracted in the first step is input into the channel attention mechanism; In the third step, the temporal features from the second step are input into the improved Transformer module for further processing, and the multi-layer encoding results of the classification label [CLS] are output to further mine the temporal information; In the fourth step, the temporal features of the second step are concatenated with the multi-layer encoding results of the classification mark [CLS] of the third step, and linearly transformed into the dimension corresponding to the category.

3. A spatial feature extraction module in an efficient spatiotemporal CNN-Transformer fusion network model suitable for fNIRS classification according to claim 1, characterized in that: To obtain spatial features from fNIRS signals, the spatial feature extraction module consists of a spatial CNN (ConvS) with a large convolution kernel and the improved Transformer. The specific steps are as follows: In the first step, the fNIRS signal is input into the ConvS module to extract spatial features. The kernel length of the ConvS module is set to the number of sampling channels C, and the kernel width is as short as possible to aggregate the spatial information of multi-channel fNIRS. Ideally, the kernel size of ConvT is C × (T / C); In the second step, the spatial features are input into the improved Transformer module for further processing, and the multi-layer encoding results of the classification label [CLS] are output to further mine the spatial information; In the third step, the spatial features of the first step are concatenated with the multi-layer encoding results of the classification mark [CLS] of the second step, and linearly transformed into the dimension corresponding to the category.

4. An improved Transformer module in an efficient spatiotemporal CNN-Transformer fusion network model suitable for fNIRS classification according to claim 1, characterized in that: The self-attention mechanism is used to model the dependencies between positions in the input vector, enhance the feature fusion and abstraction of the input vector, and thus produce more semantically robust and discriminative representations. The specific steps are as follows: The first step is to concatenate a classification tag [CLS] before the input vector. [CLS] will extract information from the vector that helps with classification in the subsequent multi-head self-attention (MHSA) mechanism. Then, a positional encoding is added to the classification tag [CLS] and the feature vector. The positional encoding is a learnable one-dimensional embedding that represents the positional information of the [CLS] tag and the feature vector. This positional information can be used to distinguish the classification tag [CLS] from the feature vector. In the second step, the classification label [CLS] and feature vector are input into MHSA; In the third step, the classification mark [CLS] is updated using the output of MHSA, but the feature vector is not updated. The classification mark [CLS] is normalized and added to the classification mark [CLS] of the first step. In the fourth step, the classification label [CLS] and the feature vector are input into the feedforward network, which consists of two linear transformation layers with a GeLU activation function in the middle; the classification label [CLS] is normalized and added to the classification label [CLS] in the third step; the feature vector is normalized and added to the feature vector in the first step; Step 5: Repeat steps 2 to 5. The number of repetitions is the number of layers, which is a configurable parameter. In the sixth step, the final output of the Transformer module consists of two parts: one is the multi-layer encoding result of the classification tag [CLS], and the other is the embedded multi-layer encoding result. For classification tasks, only the part corresponding to the tag [CLS] needs to be extracted.

5. A temporal and spatial feature fusion module in an efficient spatiotemporal CNN-Transformer fusion network model suitable for fNIRS classification according to claim 1, characterized in that: The temporal and spatial features are fused for classification. The specific steps are as follows: The first step is to input the spatial features output by the spatial feature extraction module into the Tanh function; In the second step, the spatial features output by the spatial feature extraction module are input into the Sigmoid function and multiplied by the square of the result of the first step; The third step is to add the time feature output by the time feature extraction module to the result of the second step; In the fourth step, the result of the third step is input into the Softmax function to obtain the probability that the fNIRS signal belongs to each category.

Images