Fmirs motor imagery decoding method based on double-flow cross-attention and functional connection fusion

Abstract

The application discloses a kind of fNIRS motor imagination decoding methods based on double-flow cross attention and function connection fusion, obtain the fNIRS original signal of subject under motor imagination task, calculate HbO signal and HbR signal concentration variation sequence, respectively to HbO signal and HbR signal are first-order differential processing, generate the enhanced feature stream reflecting the change rate of HbO signal and HbR signal, and introduce residual connection;Double-flow feature encoder is constructed, and HbO signal depth feature map and HbR signal depth feature map are extracted using time convolution and depth space convolution respectively;Cross attention module is introduced, and the depth feature after double-flow dynamic complementary fusion is obtained;For each signal sample obtained, calculate the pearson correlation coefficient matrix between all acquisition channels, and flatten it into a global brain network connection vector matrix;The depth feature after double-flow dynamic complementary fusion is spliced with the global brain network connection vector matrix, and the category result of motor imagination is output through fully connected classifier.

Description

Technical Field

[0001] This invention relates to the field of brain-computer interface and deep learning intersection technology, and in particular to an fNIRS motor imagery decoding method based on dual-stream cross-attention and functional connectivity fusion. Background Technology

[0002] Motor imagery (MI) decoding is a core technology in the fields of brain-computer interfaces (BCI) and neurorehabilitation medicine. It offers revolutionary possibilities for paralyzed patients to re-establish motor control pathways and promote the recovery of neural plasticity, possessing significant clinical value and social significance. Motor imagery decoding based on functional near-infrared spectroscopy (fNIRS) aims to accurately identify a patient's motor intentions by capturing the hemodynamic response patterns induced by neural activity in the brain's motor cortex. This enables direct control of external devices or serves as a key component of neurofeedback training to accelerate motor function rehabilitation.

[0003] Currently, traditional convolutional neural network (CNN) models are widely used in this field for MI decoding. The basic idea is to convert the fNIRS time-series signal into a two-dimensional representation such as a topological map or a time-frequency map. By utilizing the local perception and weight sharing mechanism of CNN, spatial topological dependencies and local time-frequency features are automatically captured through multi-layer convolution and pooling operations. Finally, end-to-end intent classification is achieved through a fully connected layer. The characteristics of this type of method are that the network structure is relatively simple and the interpretability is strong. It can effectively extract shallow local patterns in fNIRS data. However, the existing technology has the following defects: (1) Insufficient feature fusion: Simply stacking oxyhemoglobin and deoxyhemoglobin signals ignores the complex hemodynamic coupling relationship between the two. (2) Spatial perception limitation: Convolutional kernels usually only focus on the local features of adjacent channels, making it difficult to capture the global collaborative patterns between multiple brain regions of the motor cortex (such as the primary motor cortex, supplementary motor area, and premotor cortex). Summary of the Invention

[0004] The purpose of this application is to address the technical deficiencies in the existing technology by providing an fNIRS motion imagery decoding method based on dual-stream cross-attention and functional connectivity fusion.

[0005] The technical solution adopted to achieve the purpose of this application is: A method for decoding motion imagery in fNIRS based on dual-stream cross-attention and functional connectivity fusion includes the following steps: Step 1: Obtain the raw fNIRS signal of the subject under the motor imagery task, calculate the concentration change sequence of the two modes of HbO signal and HbR signal, perform first-order differential processing on HbO signal and HbR signal respectively, generate enhanced feature flow reflecting the rate of change of HbO signal and HbR signal, and introduce residual connection. Step 2: Construct a dual-stream feature encoder. Use temporal convolution and depth-space convolution to extract the local spatiotemporal features of the enhanced feature stream reflecting the rate of change of the HbO and HbR signals after enhancement in Step 1, respectively, and generate the depth feature map of the HbO signal. HbR signal depth feature map ; Step 3: Introduce a cross-attention module to enable mutual attention between HbO and HbR signals. Map the signals using either HbO or HbR local spatiotemporal features as query vectors and either HbR or HbO local spatiotemporal features as key vectors. This achieves dynamic complementary fusion of the two streams' features, resulting in the deep features obtained after dynamic complementary fusion. , ; Step 4: For each acquired signal sample, calculate the Pearson correlation coefficient matrix between all acquisition channels and flatten it into a global brain network connectivity vector matrix. ; Step 5: The depth features obtained from the dynamic complementary fusion of the two streams in Step 3 are then processed. , The global brain network connectivity vector matrix obtained in step 4 The data is then concatenated and a fully connected classifier is used to output the category results of the motion imagery.

[0006] In the above technical solution, the categories of the motion visualization task include raising the left hand and raising the right hand.

[0007] In the above technical solution, step 2 includes the following steps: S201: Construct a two-stream feature encoder. Input the enhanced feature stream reflecting the rate of change of HbO signal from step 1 into the two-stream feature encoder. Use a temporal convolutional layer to capture the temporal waveform features of the hemodynamic response function, model the local temporal dependence, and use a spatial convolutional layer to capture the local topological correlation between different spatial channels, generating a depth feature map of the HbO signal. ; S202: Simultaneously, the enhanced feature stream reflecting the rate of change of the HbR signal after enhancement in step 1 is input into a dual-stream feature encoder. Temporal convolutional layers are used to capture the temporal waveform features of the hemodynamic response function to model local temporal dependence, and spatial convolutional layers are used to capture local topological correlations between different spatial channels to generate a depth feature map of the HbR signal. .

[0008] In the above technical solution, step 3 includes the following steps: S301: Introduce a cross-attention module to process the HbO signal depth feature map generated in step 2. Perform 1×1 convolution dimensionality reduction to generate the query vector. Key vector Sum value vector The generated query matrix The dimension was adjusted to B × T ×4, recorded as This allows the HbO signal to pay attention to the HbR signal, resulting in the first weighted attention matrix. ; S302: In the first weighted attention matrix Based on this, residual connectivity is introduced; S303: Depth feature map of the HbR signal generated in step 2 Perform 1×1 convolution dimensionality reduction, introduce a cross-attention module, and generate a query vector. Key vector Sum value vector The generated query matrix The dimension was adjusted to B × T ×4, recorded as This allows the HbR signal to pay attention to the HbO signal, resulting in the second weighted attention matrix. ; S304: In the second weighted attention matrix Based on this, residual connectivity is introduced; S305: Apply the first weighted attention matrix that introduces residual connections. and the second weighted attention matrix that introduces residual connections Mean compression is applied to the temporal dimension to obtain the deep features after dynamic complementary fusion of the two streams. , .

[0009] In the above technical solution, step 4 includes the following steps: S401: For each signal sample obtained in step 1, calculate the Pearson correlation coefficient matrix between all acquisition channels; S402: Perform an inverse hyperbolic tangent transform on the calculated Pearson correlation coefficient matrix; S403: The Pearson correlation coefficient matrix after inverse hyperbolic tangent transformation Flatten and in a batch B The Pearson correlation coefficient matrices, after being flattened, are combined to obtain the global brain network connectivity vector matrix. .

[0010] In the above technical solution, the expression for the Pearson correlation coefficient matrix among all acquisition channels is as follows:

[0011] In the formula, Represents the acquisition channel, 1≤ ≤ ; Represents the acquisition channel, 1≤ ≤ ; This represents the Pearson correlation coefficient matrix among all acquisition channels. ; This represents the total number of time points in the sample. Represents the number of time points, 1≤ ≤ ; This represents the acquired HbO or HbR signal sample; This represents the average of all samples. The expression for the inverse hyperbolic tangent transform of the calculated Pearson correlation coefficient matrix is ​​as follows:

[0012] In the formula, This represents the matrix obtained by the inverse hyperbolic tangent transformation; Represents the inverse hyperbolic tangent function; Represents the clipping function; This represents the Pearson correlation coefficient matrix across all acquisition channels.

[0013] In the above technical solution, the flattened Pearson correlation coefficient matrix The expression is as follows:

[0014] In the formula, This represents the flattened Pearson correlation coefficient matrix; Represents the flattening operation; This represents the matrix obtained by the inverse hyperbolic tangent transformation; Represents the number of channels The square of.

[0015] The global brain network connectivity vector matrix expression:

[0016] In the formula, Represents the global brain network connectivity vector matrix; This represents the first flattened Pearson correlation coefficient matrix; This represents the second flattened Pearson correlation coefficient matrix; Representing the The Pearson correlation coefficient matrix after flattening; Represents the number of samples in a batch; Represents the number of channels The square of.

[0017] In the above technical solution, step 5 includes the following steps: S501: The depth features obtained from the dynamic complementary fusion of the two streams in step 3 are... , The global brain network connectivity vector matrix obtained in step 4 spliced ​​into a matrix ; S502: Concatenate the matrix The data is sequentially input into two fully connected layers for computation, and the output layer decodes and predicts the category of the subject's motor imagery.

[0018] In the above technical solution, the calculation formula for the first fully connected layer is as follows:

[0019] In the formula, This represents the computation result of the first fully connected layer; This represents a discard operation; Represents the modified linear unit activation function; This represents a batch normalization operation; Represents a splicing matrix; This represents the weight matrix of the first fully connected layer. This represents the bias vector of the first fully connected layer. This represents the number of samples in a batch.

[0020] The calculation formula for the second fully connected layer is as follows:

[0021] In the formula, This represents the computation result of the second fully connected layer; Represents the modified linear unit activation function; This represents the computation result of the first fully connected layer; This represents the weight matrix of the second fully connected layer; This represents the bias vector of the second fully connected layer. This represents the number of samples in a batch.

[0022] In the above technical solution, the category expression for the output layer decoding and predicting the subject's motor imagery is as follows:

[0023] In the formula, This represents the calculation result of the output layer; This represents the matrix representing the computation results of the second fully connected layer; The weight matrix represents the output layer; Represents the bias vector of the output layer; Represents the number of samples in a batch; This represents the bias vector of the output layer.

[0024] The beneficial effects of this invention are as follows: 1. The present invention is based on the fNIRS motion imagery decoding method of dual-stream cross-attention and functional connectivity fusion, which can achieve HbO and HbR feature fusion through dual-stream cross structure without stacking HbO and HbR signals, and capture the complex hemodynamic coupling relationship between the two.

[0025] 2. The present invention is based on the fNIRS motion imagery decoding method of dual-stream cross-attention and functional connectivity fusion. It explicitly uses the Pearson correlation coefficient matrix to nonlinearly fuse all channel features, which can achieve global collaborative spatial perception. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 The diagram shown is a schematic of the structural framework of the fNIRS motion imagery decoding method based on dual-stream cross-attention and functional connectivity fusion described in this invention. Detailed Implementation

[0028] To enable those skilled in the art to better understand the present invention, the technical solution of the present invention will be further described below with reference to specific embodiments. Example 1

[0029] A method for decoding motion imagery in fNIRS based on dual-stream cross-attention and functional connectivity fusion, see [link to relevant documentation]. Figure 1 This includes the following steps: Step 1: Obtain the raw fNIRS signal of the subject during the motor imagery task. Calculate the concentration change sequences of two modes: HbO (oxygenated hemoglobin) and HbR (deoxygenated hemoglobin). Perform first-order differentiation on the HbO and HbR signals respectively to generate enhanced feature flows reflecting the rates of change of HbO and HbR signals. Introduce residual connections to increase the reliability of the features. In this embodiment, the motor imagery task category K includes two types: raising the left hand and raising the right hand. The enhanced feature expressions for the rates of change of HbO and HbR signals are as follows:

[0030] In the formula, represent HbO signal at time; The HbR signal representing the time; , These represent learnable scalar parameters, initialized to 0; represent HbO signal at time; represent The HbR signal at time t; This represents the number of time points sampled, preferably 30. Among these, at the endpoints... At this point, the first-order differential cannot be calculated. and The first derivative is set to 0, and the dataset consists of oxyhemoglobin signals and deoxyhemoglobin signals. , This represents the number of samples in a batch, preferably 16. C The number of physiological channels is represented, preferably 10; training optimizer optimizer The preferred choice is Adam; learning rate lr The preferred value is 0.0005; number of iterations epoch The preferred value is 20; the preferred value for random inactivation dropout is 0.6.

[0031] Step 2: Construct a dual-stream feature encoder. Use temporal convolution and depth-space convolution to extract the local spatiotemporal features of the enhanced feature stream reflecting the rate of change of the HbO and HbR signals after enhancement in Step 1, respectively, and generate the depth feature map of the HbO signal. HbR signal depth feature map .

[0032] Step 2 includes the following steps: S201: Construct a two-stream feature encoder. Input the enhanced feature stream reflecting the rate of change of the HbO signal from step 1 into the two-stream feature encoder. Use a temporal convolutional layer to capture the temporal waveform features of the hemodynamic response function, model the local temporal dependence, and use a spatial convolutional layer to capture the local topological correlation between different spatial channels, generating a depth feature map of the HbO signal. The output expression of the temporal convolutional layer is as follows:

[0033] In the formula, Represents the output of the temporal convolutional layer, with dimension . B ×16× C × T ; Represents the activation operation of the exponential linear unit; Representative batch normalization operation; This represents a convolution operation with 16 kernels and a kernel size of 1×31. Represents the oxygenated hemoglobin signal; The weight matrix representing the temporal convolutional layer; This represents the bias vector of the temporal convolutional layer.

[0034] The output expression of the spatial convolutional layer is as follows:

[0035] In the formula, Represents the output of the spatial convolutional layer, with dimension . B ×32× T ; represent; This represents a convolution operation with 32 kernels and a kernel size of 1× C After convolution operation C =1; Represents the output of the temporal convolutional layer; The weight matrix represents the spatial convolutional layer; This represents the bias vector of the spatial convolutional layer.

[0036] S202: Simultaneously, the enhanced feature stream reflecting the rate of change of the HbR signal after enhancement in step 1 is input into a dual-stream feature encoder. Temporal convolutional layers are used to capture the temporal waveform features of the hemodynamic response function to model local temporal dependence, and spatial convolutional layers are used to capture local topological correlations between different spatial channels to generate a depth feature map of the HbR signal. (The depth feature map of the HbR signal) Generation and depth feature map of HbO signal (The generation process is consistent), and its dimension is... B ×32×T .

[0037] Step 3: Introduce a cross-attention module to enable mutual attention between HbO and HbR signals. Map the signals using either HbO or HbR local spatiotemporal features as query vectors and either HbR or HbO local spatiotemporal features as key vectors. This achieves dynamic complementary fusion of the two streams' features, resulting in the deep features obtained after dynamic complementary fusion. , .

[0038] Step 3 includes the following steps: S301: Introduce a cross-attention module to process the HbO signal depth feature map generated in step 2. Perform 1×1 convolution dimensionality reduction to generate the query vector. Key vector Sum value vector The generated query matrix The dimension was adjusted to B × T ×4, recorded as This allows the HbO signal to pay attention to the HbR signal, resulting in the first weighted attention matrix. .

[0039] In the formula, Represents the query vector matrix; This represents a one-dimensional convolution operation; Represents the depth feature map of the HbO signal; Represents the depth feature map of the HbO signal; Represents the key vector matrix; Represents the depth feature map of the HbR signal; The weight matrix representing the key vectors; Represents a value vector matrix; The weight matrix represents the value vector.

[0040] The expression for the first weighted attention matrix is ​​as follows:

[0041] In the formula, This represents the first weighted attention matrix; Represents a value vector matrix; Represents the normalized exponential function; The transpose of the query vector; Represents the key vector matrix; Represents the trace of the matrix.

[0042] S302: In the first weighted attention matrix Based on this, residual connections are introduced to increase the reliability of deep networks. This is done using the first weighted attention matrix. Based on this, the residual join expression is introduced as follows:

[0043] In the formula, This represents the result obtained from residual connection; Represents the depth feature map of the HbO signal; Represents the learnable scaling factor; This represents the first weighted attention matrix.

[0044] S303: Similarly, the depth feature map of the HbR signal generated in step 2... Perform 1×1 convolution dimensionality reduction, introduce a cross-attention module, and generate a query vector. Key vector Sum value vector The generated query matrix The dimension was adjusted to B × T ×4, recorded as This allows the HbR signal to pay attention to the HbO signal, resulting in the second weighted attention matrix. .

[0045] S304: In the second weighted attention matrix Based on this, residual connections are introduced to increase the reliability of deep networks.

[0046] S305: Apply the first weighted attention matrix that introduces residual connections. and the second weighted attention matrix that introduces residual connections Mean compression is applied to the temporal dimension to obtain the deep features after dynamic complementary fusion of the two streams. , This eliminates the impact of duration variability. The depth features resulting from the dynamic complementary fusion of the two streams... , The expression is as follows:

[0047] In the formula, The depth feature matrix representing the HbO signal; The depth feature matrix representing the HbR signal; This represents the number of time points sampled. Represents the current point in time; This represents the first weighted attention matrix that introduces residual connections; This represents the second weighted attention matrix that introduces residual connections; This represents the number of samples in a batch.

[0048] Step 4: For each acquired signal sample (all HbO and HbR signals acquired in Step 1), calculate the Pearson correlation coefficient matrix between all acquisition channels and flatten it into a global brain network connectivity vector matrix. .

[0049] Step 4 includes the following steps: S401: For each signal sample obtained in step 1, calculate the Pearson correlation coefficient matrix among all acquisition channels. The expression for the Pearson correlation coefficient matrix among all acquisition channels is as follows:

[0050] In the formula, Represents a channel, 1≤ ≤ ; Represents a channel, 1≤ ≤ ; This represents the Pearson correlation coefficient matrix among all acquisition channels. ; This represents the total number of time points in the sample. Represents the number of time points, 1≤ ≤ ; This represents the acquired HbO or HbR signal sample; This represents the average of all samples.

[0051] S402: To enhance the normality of the correlation coefficients, an inverse hyperbolic tangent transform is performed on the calculated Pearson correlation coefficient matrix. The expression for the inverse hyperbolic tangent transform on the calculated Pearson correlation coefficient matrix is ​​as follows:

[0052] In the formula, This represents the matrix obtained by the inverse hyperbolic tangent transformation; Represents the inverse hyperbolic tangent function; Represents the clipping function; This represents the Pearson correlation coefficient matrix across all acquisition channels.

[0053] S403: The Pearson correlation coefficient matrix after inverse hyperbolic tangent transformation Flatten and in a batch B The Pearson correlation coefficient matrices, after being flattened, are combined to obtain the global brain network connectivity vector matrix. The flattened Pearson correlation coefficient matrix The expression is as follows:

[0054] In the formula, The Pearson correlation coefficient matrix after flattening; Represents the flattening operation; This represents the matrix obtained by the inverse hyperbolic tangent transformation; Represents the number of channels The square of.

[0055] The global brain network connectivity vector matrix expression:

[0056] In the formula, Represents the global brain network connectivity vector matrix; This represents the first flattened Pearson correlation coefficient matrix; This represents the second flattened Pearson correlation coefficient matrix; This represents the B-th flattened Pearson correlation coefficient matrix; Represents the number of samples in a batch; Represents the number of channels The square of.

[0057] Step 5: The depth features obtained from the dynamic complementary fusion of the two streams in Step 3 are then processed. , The global brain network connectivity vector matrix obtained in step 4 The data is then concatenated and a fully connected classifier is used to output the category results of the motion imagery.

[0058] Step 5 includes the following steps: S501: The depth features obtained from the dynamic complementary fusion of the two streams in step 3 are... , The global brain network connectivity vector matrix obtained in step 4 spliced ​​into a matrix ( The matrix has dimensions of B ×(32+32+ )).

[0059] S502: Concatenate the matrix The data is sequentially input into two fully connected layers for computation, and the output layer decodes and predicts the category of the subject's motor imagery. The calculation formula for the first fully connected layer is as follows:

[0060] In the formula, This represents the computation result of the first fully connected layer; This represents a discard operation; Represents the modified linear unit activation function; This represents a batch normalization operation; Represents a splicing matrix; This represents the weight matrix of the first fully connected layer. This represents the bias vector of the first fully connected layer. This represents the number of samples in a batch.

[0061] The calculation formula for the second fully connected layer is as follows:

[0062] In the formula, This represents the computation result of the second fully connected layer; Represents the modified linear unit activation function; This represents the computation result of the first fully connected layer; This represents the weight matrix of the second fully connected layer; This represents the bias vector of the second fully connected layer; This represents the number of samples in a batch.

[0063] The output layer decodes and predicts the category expression of the subject's motor imagery as follows:

[0064] In the formula, This represents the calculation result of the output layer; This represents the matrix representing the computation results of the second fully connected layer; The weight matrix represents the output layer; Represents the bias vector of the output layer; Represents the number of samples in a batch; The number of categories representing the movement imagination.

[0065] The fNIRS motion imagery decoding method in this embodiment is denoted as fNIRSDCNet, and compared with fNIRSNet, the two methods are compared in " Adam Optimizer, Learning Rate lr =0.005, number of iterations epoch Simulation training was conducted under the condition of 20". The simulation training results are shown in Table 1.

[0066] Table 1 Comparison of simulation results for the motion imagery decoding method Table 1 presents the average results of the five-fold cross-validation, expressed as mean ± standard deviation. The average accuracy of fNIRSDCNet is 86.02%, significantly better than the average accuracy of fNIRSNet (74.80%). Furthermore, the standard deviation of the fNIRSDCNet method is lower than that of fNIRSNet, indicating that the fNIRS motion imagery decoding method of this invention has better reliability. The F1 score, another evaluation metric, also verifies the effectiveness of the fNIRS motion imagery decoding method of this invention. These results demonstrate that the fNIRS motion imagery decoding method of this invention effectively improves the accuracy of motion imagery decoding and performs better in the fNIRS motion imagery decoding problem.

[0067] For ease of explanation, spatial relative terms such as “up,” “down,” “left,” and “right” are used in the embodiments to describe the relationship of one element or feature shown in the figures relative to another element or feature. It should be understood that, in addition to the orientations shown in the figures, spatial terms are intended to include different orientations of the device in use or operation. For example, if the device in the figures is inverted, an element described as being “down” of other elements or features would be positioned “up” of those other elements or features. Therefore, the exemplary term “down” can encompass both up and down orientations. The device may be positioned in other ways (rotated 90 degrees or in other orientations), and the spatial relative descriptions used herein will be interpreted accordingly.

[0068] Moreover, relational terms such as “first” and “second” are used merely to distinguish one component from another that has the same name, without necessarily requiring or implying any such actual relationship or order between the components.

[0069] The above description is only a preferred embodiment of the present invention. It should be noted that, for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for decoding motion imagery in fNIRS based on dual-stream cross-attention and functional connectivity fusion, characterized in that, Includes the following steps: Step 1: Obtain the raw fNIRS signal of the subject under the motor imagery task, calculate the concentration change sequence of the two modes of HbO signal and HbR signal, perform first-order differential processing on HbO signal and HbR signal respectively, generate enhanced feature flow reflecting the rate of change of HbO signal and HbR signal, and introduce residual connection. Step 2: Construct a dual-stream feature encoder. Use temporal convolution and depth-space convolution to extract the local spatiotemporal features of the enhanced feature stream reflecting the rate of change of the HbO and HbR signals after enhancement in Step 1, respectively, and generate the depth feature map of the HbO signal. HbR signal depth feature map ; Step 3: Introduce a cross-attention module to enable mutual attention between HbO and HbR signals. Map the signals using either HbO or HbR local spatiotemporal features as query vectors and either HbR or HbO local spatiotemporal features as key vectors. This achieves dynamic complementary fusion of the two streams' features, resulting in the deep features obtained after dynamic complementary fusion. , ; Step 4: For each acquired signal sample, calculate the Pearson correlation coefficient matrix between all acquisition channels and flatten it into a global brain network connectivity vector matrix. ; Step 5: The depth features obtained from the dynamic complementary fusion of the two streams in Step 3 are then processed. , The global brain network connectivity vector matrix obtained in step 4 The data is then concatenated and a fully connected classifier is used to output the category results of the motion imagery.

2. The fNIRS motion imagery decoding method according to claim 1, characterized in that, The categories of the motor imagery tasks include raising the left hand and raising the right hand.

3. The fNIRS motion imagery decoding method according to claim 1, characterized in that, Step 2 includes the following steps: S201: Construct a two-stream feature encoder. Input the enhanced feature stream reflecting the rate of change of the HbO signal from step 1 into the two-stream feature encoder. Use a temporal convolutional layer to capture the temporal waveform features of the hemodynamic response function, model the local temporal dependence, and use a spatial convolutional layer to capture the local topological correlation between different spatial channels, generating a depth feature map of the HbO signal. ; S202: Simultaneously, the enhanced feature stream reflecting the rate of change of the HbR signal after enhancement in step 1 is input into a dual-stream feature encoder. Temporal convolutional layers are used to capture the temporal waveform features of the hemodynamic response function to model local temporal dependence, and spatial convolutional layers are used to capture local topological correlations between different spatial channels to generate a depth feature map of the HbR signal. .

4. The fNIRS motion imagery decoding method according to claim 1, characterized in that, Step 3 includes the following steps: S301: Introduce a cross-attention module to process the HbO signal depth feature map generated in step 2. Perform 1×1 convolution dimensionality reduction to generate the query vector. Key vector Sum value vector The generated query matrix The dimension was adjusted to B × T ×4, recorded as This allows the HbO signal to pay attention to the HbR signal, resulting in the first weighted attention matrix. ; S302: In the first weighted attention matrix Based on this, residual connectivity is introduced; S303: Depth feature map of the HbR signal generated in step 2 Perform 1×1 convolution dimensionality reduction, introduce a cross-attention module, and generate a query vector. Key vector Sum value vector The generated query matrix The dimension was adjusted to B × T ×4, recorded as This allows the HbR signal to pay attention to the HbO signal, resulting in the second weighted attention matrix. ; S304: In the second weighted attention matrix Based on this, residual connectivity is introduced; S305: Apply the first weighted attention matrix that introduces residual connections. and the second weighted attention matrix that introduces residual connections Mean compression is applied to the temporal dimension to obtain the deep features after dynamic complementary fusion of the two streams. , .

5. The fNIRS motion imagery decoding method according to claim 1, characterized in that, Step 4 includes the following steps: S401: For each signal sample obtained in step 1, calculate the Pearson correlation coefficient matrix between all acquisition channels; S402: Perform an inverse hyperbolic tangent transform on the calculated Pearson correlation coefficient matrix; S403: The Pearson correlation coefficient matrix after inverse hyperbolic tangent transformation Flatten and in a batch B The Pearson correlation coefficient matrices, after being flattened, are combined to obtain the global brain network connectivity vector matrix. .

6. The fNIRS motion imagery decoding method according to claim 5, characterized in that, The expression for the Pearson correlation coefficient matrix among all acquisition channels is as follows: In the formula, Represents the acquisition channel, 1≤ ≤ ; Represents the acquisition channel, 1≤ ≤ ; This represents the Pearson correlation coefficient matrix among all acquisition channels. ; This represents the total number of time points in the sample. Represents the number of time points, 1≤ ≤ ; This represents the acquired HbO or HbR signal sample; This represents the average of all samples. The expression for the inverse hyperbolic tangent transform of the calculated Pearson correlation coefficient matrix is ​​as follows: In the formula, This represents the matrix obtained by the inverse hyperbolic tangent transformation; Represents the inverse hyperbolic tangent function; Represents the clipping function; This represents the Pearson correlation coefficient matrix across all acquisition channels.

7. The fNIRS motion imagery decoding method according to claim 5, characterized in that, The flattened Pearson correlation coefficient matrix The expression is as follows: In the formula, This represents the flattened Pearson correlation coefficient matrix; Represents the flattening operation; This represents the matrix obtained by the inverse hyperbolic tangent transformation; Represents the number of channels The square of. The global brain network connectivity vector matrix expression: In the formula, Represents the global brain network connectivity vector matrix; This represents the first flattened Pearson correlation coefficient matrix; This represents the second flattened Pearson correlation coefficient matrix; Representing the The Pearson correlation coefficient matrix after flattening; Represents the number of samples in a batch; Represents the number of channels The square of.

8. The fNIRS motion imagery decoding method according to claim 1, characterized in that, Step 5 includes the following steps: S501: The depth features obtained from the dynamic complementary fusion of the two streams in step 3 are... , The global brain network connectivity vector matrix obtained in step 4 spliced ​​into a matrix ; S502: Concatenate the matrix The data is sequentially input into two fully connected layers for computation, and the output layer decodes and predicts the category of the subject's motor imagery.

9. The fNIRS motion imagery decoding method according to claim 8, characterized in that, The calculation formula for the first fully connected layer is as follows: In the formula, This represents the computation result of the first fully connected layer; This represents a discard operation; Represents the modified linear unit activation function; This represents a batch normalization operation; Represents a splicing matrix; This represents the weight matrix of the first fully connected layer. This represents the bias vector of the first fully connected layer. This represents the number of samples in a batch. The calculation formula for the second fully connected layer is as follows: In the formula, This represents the computation result of the second fully connected layer; Represents the modified linear unit activation function; This represents the computation result of the first fully connected layer; This represents the weight matrix of the second fully connected layer; This represents the bias vector of the second fully connected layer. This represents the number of samples in a batch.

10. The fNIRS motion imagery decoding method according to claim 8, characterized in that, The category expression for the decoded prediction of the subject's motor imagery in the output layer is as follows: In the formula, This represents the calculation result of the output layer; This represents the matrix representing the computation results of the second fully connected layer; The weight matrix represents the output layer; Represents the bias vector of the output layer; Represents the number of samples in a batch; This represents the bias vector of the output layer.

Images