Motor imagery later fusion decoding method based on EEG-fNIRS data

Abstract

The invention discloses a motor imagery late fusion decoding method based on EEG-fNIRS data, and the method comprises the steps: preprocessing the EEG-fNIRS data, and obtaining EEG data and fNIRS data of a specified size; using a depth separable convolution feature extraction module based on an Inception architecture to extract EEG feature information, and using an fNIRS feature extraction module to extract fNIRS feature information; inputting the EEG feature information into a self-adaptive average pooling layer, changing the dimension of the EEG feature information to be consistent with the dimension of the fNIRS feature information, and splicing the changed EEG feature information and fNIRS feature information to obtain mixed features; and inputting the mixed features into a multi-layer perceptron for motor imagery task classification. According to the method, EEG data with high time resolution and fNIRS data with high spatial resolution can be combined, so that EEG and fNIRS feature information is fused and complemented, and the classification performance of motor imagery tasks is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of motor imagery decoding, and in particular to a late fusion decoding method for motor imagery based on EEG-fNIRS data. Background Art

[0002] With the continuous development of brain-computer interface technology, motor imagery decoding based on EEG and fNIRS signals has become an important research direction in the fields of neuroscience and intelligent medicine in recent years. Most traditional motor imagery signal decoding methods rely on a single signal source, such as EEG or fNIRS alone for the detection of motor imagery. By analyzing EEG signals and fNIRS signals separately, it can be known that EEG signals have problems of poor spatial resolution and being greatly interfered by electroencephalogram activity artifacts, and their signals have strong individual differences, noise interference and non-linear characteristics; fNIRS signals have advantages in spatial resolution, but their sensitivity to blood oxygen changes may make it difficult to fully capture brain activities related to movement. Therefore, the decoding methods of a single signal source often have deficiencies in recognition accuracy and robustness, and it is difficult to meet the requirements of high precision and high stability. To solve the above problems, in recent years, multi-modal signal fusion technology has gradually become a research hotspot in motor imagery decoding. The EEG-fNIRS signal fusion method can more comprehensively extract brain function information during the brain's motor imagery process by comprehensively utilizing the complementary characteristics of EEG and fNIRS. The combination of the two can improve the accuracy and stability of decoding.

[0003] Although the fusion of EEG and fNIRS can effectively improve the decoding performance, most of the existing fusion methods have certain limitations in the design of signal preprocessing, feature extraction and fusion strategies. For example, how to effectively fuse the features of signals from different sources and how to improve the decoding accuracy on the premise of ensuring real-time performance are still difficult points in the research. In addition, the existing methods generally have bottlenecks in the feature extraction and decoding capabilities of multi-modal signals in the late fusion stage, resulting in the decoding accuracy and stability not meeting the requirements in practical applications. Therefore, designing a new late fusion decoding method for motor imagery can better solve the effective fusion of multi-modal EEG-fNIRS signals, improve the decoding accuracy, enhance the anti-interference ability, and meet the actual application requirements. Summary of the Invention

[0004] The purpose of the present invention is to overcome the deficiencies of the prior art and propose a late fusion decoding method for motor imagery based on EEG-fNIRS data, which can combine high-time-resolution EEG data and high-spatial-resolution fNIRS data, improve the quality of feature extraction, strengthen the information fusion and complementarity between modalities, and improve the decoding accuracy of motor imagery tasks.

[0005] To achieve the above object, the technical solution provided by the present invention is: A late fusion decoding method for motor imagery based on EEG-fNIRS data, comprising the following steps:

[0006] S1: Obtain EEG-fNIRS data during a motor imagery task, and perform corresponding preprocessing operations on the EEG and fNIRS data respectively to obtain EEG data and fNIRS data of a specified size;

[0007] S2: For the above EEG data, a depthwise separable convolutional feature extraction module based on the Inception architecture is designed, called the EEG feature extraction module. Specifically, the Inception architecture contains depthwise separable convolutions of multiple different sizes, which can extract rich EEG feature information while reducing the number of network parameters; for the above fNIRS data, an fNIRS feature extraction module is designed, which includes multi-scale depth convolution and pointwise convolution, and is used to extract features and fuse features of the two types of data HbO and HbR contained in the fNIRS data. Then, a Transformer module with positional encoding is used to further extract features from the fused features to obtain fNIRS feature information;

[0008] S3: Input the EEG feature information into an adaptive average pooling layer to change the dimension of the EEG feature information to be consistent with the dimension of the fNIRS feature information, and splice the changed EEG feature information and the fNIRS feature information to obtain a mixed feature;

[0009] S4: Input the mixed feature into a multi-layer perceptron, and output a class probability vector, where the task corresponding to the maximum probability value is the classification category of the motor imagery task.

[0010] Furthermore, in step S1, the preprocessing operations for the EEG data include: downsampling, band-pass filtering, notch filtering, ICA artifact removal, normalization, and data segmentation to obtain EEG data with a time window length of 3 s; the preprocessing operations for the fNIRS data include: downsampling, band-pass filtering, Beer-Lambert law conversion, and data segmentation to obtain fNIRS data with a time window length of 3 s.

[0011] Furthermore, the specific operation steps of step S2 are as follows:

[0012] S21: For the input EEG data X eeg , first use temporal convolution to extract local features of X eeg in time, then use depthwise convolution DepthwiseConv to perform convolution fusion on the information of different channels, and then obtain spatio-temporal features through average pooling and Dropout layers The formula is expressed as follows:

[0013]

[0014] In the formula, AvgPool represents the average pooling layer, F1 represents the number of EEG time convolution kernels, T1 represents the stride of time convolution, D represents the number of depth convolution kernels, and C eeg represents the number of channels of EEG data, and Conv represents two-dimensional convolution;

[0015] S22: In order to extract and fuse the features of D*F1 one-dimensional time signals in the spatio-temporal features a depthwise separable convolution feature extraction module based on the Inception architecture is designed. The Inception architecture includes four branches. Branch one includes ordinary convolution, depthwise separable convolution with a convolution kernel of 1*T2, average pooling, and a Dropout layer. Branch two includes ordinary convolution, depthwise separable convolution with a convolution kernel of 1*T3, average pooling, and a Dropout layer. Branch three includes average pooling and ordinary convolution. Branch four includes ordinary convolution; The intermediate feature of branch one is obtained through branch one The intermediate feature of branch two is obtained through branch two The intermediate feature of branch three is obtained through branch three The intermediate feature of branch four is obtained through branch four The formula is expressed as follows:

[0016]

[0017] In the formula, DepthwiseSeparableConv represents the depthwise separable convolution layer, T2 represents the convolution kernel size of the depthwise separable convolution in branch one, T3 represents the convolution kernel size of the depthwise separable convolution in branch two, and S represents the stride of the ordinary convolution in branch four;

[0018] S23: Concatenate the intermediate features of the four branches obtained in step S22 in the dimension of the number of feature maps to obtain multi-scale features After that, depthwise separable convolution is used for feature fusion, and the output feature of the EEG feature extraction module is obtained through the Dropout layer The formula is expressed as follows:

[0019]

[0020] In the formula, Concat represents the concatenation function, dim represents that multiple features are concatenated along the specified dimension, and C fNIRS represents the number of channels of fNIRS data, T4 represents the convolution kernel size of the depthwise separable convolution, and NC Represents the number of classification categories for the motor imagery task;

[0021] S24: For the fNIRS data X fNIRS , first perform convolution operations on HbO and HbR in the fNIRS data using three convolutional kernels with different scales, and then splice the three temporal features obtained by convolution in the dimension of the number of feature maps to obtain multi-scale features Then, obtain the multi-scale fusion features through the pointwise convolution layer PointwiseConv The formula is expressed as follows:

[0022]

[0023] In the formula, F2 represents the number of fNIRS temporal convolution kernels, N1 represents the size of the first convolution kernel among the three convolution kernels, N2 represents the size of the second convolution kernel among the three convolution kernels, and N3 represents the size of the third convolution kernel among the three convolution kernels;

[0024] S25: Pass the multi-scale fusion features through the designed Transformer module with positional encoding to obtain the fused features of the encoded position information Then, splice it with the number N C category label vectors F CT of the classification categories of the motor imagery task to obtain enhanced features Then input it into L Transformer encoders to obtain the self-attention features Finally, intercept the first N C feature vectors as the output features of the fNIRS feature extraction module The formula is expressed as follows:

[0025]

[0026] In the formula, 3DTo1D represents converting 3D fNIRS data into 1D fNIRS data according to the 3D representation of the fNIRS data, 1DTo3D represents converting 1D fNIRS data into 3D fNIRS data according to the 3D representation of the fNIRS data, a represents the length of the 3D schematic diagram of the fNIRS data, b represents the width of the 3D schematic diagram of the fNIRS data, MSA represents the multi-head self-attention transformation function, LN represents layer normalization, represents the output of the first Transformer encoder, represents the output of the kth Transformer encoder, k = 2, 3,..., L.

[0027] Further, in step S3, for the output features of the EEG feature extraction module through the adaptive average pooling layer, and then using Squeeze for dimensionality reduction, the output features obtained are of the same dimension as those of the fNIRS feature extraction module data of the same dimension Then combine it with to obtain the EEG-fNIRS hybrid feature F Hybrid , which is expressed by the formula as follows:

[0028]

[0029] In the formula, Squeeze represents the dimensionality reduction function, and AdaptiveAvgPool represents the adaptive average pooling layer

[0030] Further, in step S4, input the EEG-fNIRS hybrid feature F Hybrid into the multi-layer perceptron for task classification to obtain the predicted label y test , that is, obtain the classification result of the motor imagery task, which is expressed by the formula as follows:

[0031] y test = MLP(F Hybrid )

[0032] In the formula, MLP represents the multi-layer perceptron

[0033] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0034] 1. The present invention designs a depthwise separable convolution feature extraction module based on the Inception architecture, which can effectively extract rich EEG feature information while reducing the number of network parameters

[0035] 2. The present invention designs multi-scale depth convolution and pointwise convolution, which can deeply fuse HbO and HbR in the fNIRS data and fully explore and utilize the useful information contained in the fNIRS data

[0036] 3. The present invention designs a Transformer module with position encoding, and uses the self-attention mechanism of the Transformer to autonomously learn the importance of different channels of fNIRS for task classification to obtain fNIRS feature information

[0037] 4. The present invention uses the adaptive average pooling layer to align the dimensions of the EEG feature information and the fNIRS feature information to obtain highly distinguishable fusion features Brief Description of the Drawings

[0038] Figure 1It is the framework diagram of the method of the present invention; in the figure, C eeg represents the number of channels of EEG data, N eeg represents the number of sampling points of EEG data, Conv represents two-dimensional convolution, DepthwiseConv represents depth convolution, D represents the number of depth convolution kernels, F1 represents the number of EEG temporal convolution kernels, N m represents the time dimension of EEG spatio-temporal features. Pooling represents average pooling, SeparableConv represents depthwise separable convolution, stride represents the convolution stride, Concat represents the concatenation function, AdaptiveAvgPool represents adaptive average pooling, Rearrange represents feature dimension rearrangement, N C represents the number of classification categories of the motor imagery task, C fNIRS represents the number of channels of fNIRS data, N fNIRS represents the number of sampling points of fNIRS data, PointwiseConv represents pointwise convolution, Multi-Scale DepthwiseConv represents multi-scale depth convolution, ClassTokens represents N C category token vectors, Positional Encoding represents positional encoding, Multi-HeadAttention represents the multi-head attention module, LayerNorm represents layer normalization, MLP Head represents the multi-layer perceptron head, and Class represents the final classification result.

[0039] Figure 2 It is a schematic diagram of the 3D representation of fNIRS data in the dataset used in the embodiment; in the figure, 12 and 11 are the length and width respectively in the 3D representation diagram of fNIRS, and T is the number of sampling points of the corresponding channel.

[0040] Figure 3 It is the schematic diagram of positional encoding principle. Among them, Patch tokens are multi-scale fusion features, 3D Restoration represents converting the multi-scale fusion features into 3D fNIRS data according to the 3D representation of fNIRS data, 1DRestoration represents converting the 3D fNIRS data into position information fusion features according to the 3D representation of fNIRS data, H and W are the length and width respectively in the 3D representation diagram of fNIRS, T is the number of sampling points of the corresponding channel, Conv represents two-dimensional convolution, Identity represents parameter-free direct connection, and Fusion represents adding the two for fusion. Detailed implementation manners

[0041] The present invention will be further described in detail below in conjunction with the embodiments and the accompanying drawings, but the implementation manners of the present invention are not limited thereto.

[0042] As Figures 1 to 3 shown, this embodiment discloses a late fusion decoding method for motor imagery based on EEG-fNIRS data, which includes the following steps:

[0043] 1) Obtain EEG-fNIRS data during a motor imagery task, and perform corresponding preprocessing operations on the EEG and fNIRS data respectively to obtain EEG data and fNIRS data of a specified size.

[0044] The motor imagery task used in this embodiment is a two-class motor imagery task including 29 subjects publicly available in Berlin, Germany. Among them, the preprocessing operations for EEG data include: downsampling, band-pass filtering, notch filtering, ICA artifact removal, normalization, and data segmentation to obtain EEG data with a time window length of 3s. The downsampling operation downsamples the 1000Hz EEG data to 200Hz; the preprocessing operations for fNIRS data include: downsampling, band-pass filtering, Beer-Lambert law conversion, and data segmentation to obtain fNIRS data with a time window length of 3s. The downsampling operation downsamples the 12.5Hz fNIRS data to 10Hz.

[0045] 2) Input the preprocessed EEG and fNIRS data into an EEG feature extraction module and an fNIRS feature extraction module respectively.

[0046] For the above EEG data, a depthwise separable convolution feature extraction module based on the Inception architecture is designed, called the EEG feature extraction module. Specifically, the Inception architecture contains depthwise separable convolutions of multiple different sizes, which can extract rich EEG feature information while reducing the number of network parameters; for the above fNIRS data, an fNIRS feature extraction module is designed, which includes multi-scale depth convolution and pointwise convolution for feature extraction and feature fusion of the two types of data HbO and HbR contained in the fNIRS data. Then, a Transformer module with positional encoding is used to further extract features from the fused features to obtain fNIRS feature information. The specific operation steps are as follows:

[0047] 2.1) For the input EEG data X eeg , first use temporal convolution to extract local features of X eeg in time, then use depthwise convolution DepthwiseConv to perform convolution fusion on the information of different channels, and then obtain spatio-temporal features through average pooling and Dropout layers The formula is expressed as follows:

[0048]

[0049] Where, AvgPool represents the average pooling layer, F1 represents the number of EEG temporal convolution kernels, T1 represents the stride of temporal convolution, D represents the number of depth convolution kernels, and C eeg represents the number of channels of EEG data. In the embodiment, F1 is 64, T1 is 16, D is 2, and C eeg is 30.

[0050] 2.2) In order to extract and fuse the features of the D*F1 one-dimensional temporal signals in the spatio-temporal features obtained in step 2.1) above, a depthwise separable convolution feature extraction module based on the Inception architecture is designed. The Inception architecture includes four branches. The first branch includes a common convolution, a depthwise separable convolution with a convolution kernel of 1*T2, an average pooling, and a Dropout layer. The second branch includes a common convolution, a depthwise separable convolution with a convolution kernel of 1*T3, an average pooling, and a Dropout layer. The third branch includes an average pooling and a common convolution. The fourth branch includes a common convolution. After passing through the first branch, the intermediate feature of the first branch is obtained After passing through the second branch, the intermediate feature of the second branch is obtained After passing through the third branch, the intermediate feature of the third branch is obtained After passing through the fourth branch, the intermediate feature of the fourth branch is obtained After passing through the fourth branch, the intermediate feature of the fourth branch is obtained The formula is expressed as follows:

[0051]

[0052] Where, AvgPool represents the average pooling layer, DepthwiseSeparableConv represents the depthwise separable convolution layer, F1 represents the number of EEG temporal convolution kernels, T2 represents the convolution kernel size of the depthwise separable convolution in the first branch, T3 represents the convolution kernel size of the depthwise separable convolution in the second branch, and S represents the stride of the common convolution in the fourth branch. In the embodiment, F1 is 64, T2 is 32, T3 is 64, and S is 5.

[0053] 2.3) Concatenate the intermediate features of the four branches obtained in step 2.2) in the dimension of the number of feature maps to obtain multi-scale features After that, use depthwise separable convolution to perform feature fusion, and pass through the Dropout layer to obtain the output feature of the EEG feature extraction module The formula is expressed as follows:

[0054]

[0055] Where, Concat represents the concatenation function, dim represents that multiple features are concatenated along the specified dimension, and C fNIRSC represents the number of channels of fNIRS data, T4 represents the convolution kernel size of depthwise separable convolution, and N C represents the number of classification categories of the motor imagery task. In the embodiment, C fNIRS is 36, T4 is 16, and N C is 2.

[0056] 2.4) For the fNIRS data X fNIRS , first use convolution kernels of three different scales to perform convolution operations on HbO and HbR in the fNIRS data, and then splice the three temporal features obtained by convolution in the dimension of the number of feature maps to obtain multi-scale features Then, through the pointwise convolution layer PointwiseConv, multi-scale fusion features are obtained The formula is expressed as follows:

[0057]

[0058]

[0059] In the formula, Concat represents the splicing function, F2 represents the number of fNIRS temporal convolution kernels, N1 represents the size of the first convolution kernel among the three convolution kernels, N2 represents the size of the second convolution kernel among the three convolution kernels, and N3 represents the size of the third convolution kernel among the three convolution kernels; in the embodiment, F2 is 4, N1 is 4, N2 is 8, and N3 is 16.

[0060] 2.5) Figure 2 is the 3D representation of the fNIRS data used in this example, Figure 3 and is the schematic diagram of position encoding principle. The multi-scale fusion features are passed through the designed Transformer module with position encoding to obtain the fused feature of encoded position information C and then spliced with N CT category label vectors F of the motor imagery task classification to obtain enhanced features Then, input into L Transformer encoders to obtain self-attention features Finally, intercept the first N C feature vectors as the output features of the fNIRS feature extraction module The formula is expressed as follows:

[0061]

[0062] Wherein, 3DTo1D means converting 3D fNIRS data into 1D fNIRS data according to the 3D representation of fNIRS data, 1DTo3D means converting 1D fNIRS data into 3D fNIRS data according to the 3D representation of fNIRS data, a means the length of the 3D schematic diagram of fNIRS data, b means the width of the 3D schematic diagram of fNIRS data, Concat means concatenation function, MSA means multi-head self-attention transformation function, LN means layer normalization, represents the output of the first Transformer encoder, represents the output of the kth Transformer encoder (k = 2, 3, ..., L), N C represents the number of classification categories of the motor imagery task. In an embodiment, Figure 2 As shown, the length and width of the fNIRS 3D representation are 12 and 11 respectively, so a is 12 and b is 11.

[0063] 3) Output features of the EEG feature extraction module After the adaptive average pooling layer, Squeeze is used for dimensionality reduction to obtain the output features of the fNIRS feature extraction module. Same dimension data Then compare it with After splicing, we get the EEG-fNIRS mixed feature F Hybrid , the formula is as follows:

[0064]

[0065] In the formula, Squeeze represents the dimensionality reduction function, and AdaptiveAvgPool represents the adaptive average pooling layer.

[0066] 4) Input the mixed features into the multi-layer perceptron and output the category probability vector, where the task corresponding to the maximum probability value is the classification category of the motor imagery task. Hybrid Input the multi-layer perceptron for task classification and get the predicted label y test , that is, the classification result of the motor imagery task is obtained, and the formula is expressed as follows:

[0067] y test =MLP(F Hybrid )

[0068] Where MLP stands for multi-layer perceptron.

[0069] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications made without departing from the spirit and principle of the present invention shall be equivalent replacement methods and are all included in the protection scope of the present invention.

Claims

1. A late fusion decoding method for motor imagery based on EEG-fNIRS data, characterized in that, It includes the following steps: S1: Obtain EEG-fNIRS data during a motor imagery task, and perform corresponding preprocessing operations on the EEG and fNIRS data respectively to obtain EEG data and fNIRS data of a specified size; S2: For the above EEG data, a depthwise separable convolutional feature extraction module based on the Inception architecture is designed, called the EEG feature extraction module. Specifically, the Inception architecture contains depthwise separable convolutions of multiple different sizes, which can extract rich EEG feature information while reducing the number of network parameters. For the above fNIRS data, an fNIRS feature extraction module is designed, which includes multi-scale depth convolution and pointwise convolution, and is used to extract features and fuse features of the two types of data HbO and HbR contained in the fNIRS data. Then, the Transformer module with positional encoding is used to further extract features from the fused features to obtain fNIRS feature information; S3: Input the EEG feature information into an adaptive average pooling layer to change the dimension of the EEG feature information to be consistent with the dimension of the fNIRS feature information, and splice the changed EEG feature information and the fNIRS feature information to obtain mixed features; S4: Input the mixed features into a multi-layer perceptron, and output a class probability vector. The task corresponding to the maximum probability value is the classification category of the motor imagery task.

2. The motion imagination late fusion decoding method based on EEG-fNIRS data according to claim 1, wherein In step S1, the preprocessing operations for the EEG data include: downsampling, band-pass filtering, notch filtering, ICA artifact removal, normalization, and data segmentation to obtain EEG data with a 3s time window length; the preprocessing operations for the fNIRS data include: downsampling, band-pass filtering, Beer-Lambert law conversion, and data segmentation to obtain fNIRS data with a 3s time window length.

3. The method for decoding late fusion of motor imagery based on EEG-fNIRS data according to claim 1, characterized in that, The specific operation steps of step S2 are as follows: S21: For the input EEG data X eeg , first use temporal convolution to extract the local features of X in time eeg , then use depthwise convolution DepthwiseConv to perform convolutional fusion on the information of different channels, and then obtain spatio-temporal features through average pooling and Dropout layers The formula is expressed as follows: Wherein, AvgPool represents the average pooling layer, F1 represents the number of EEG time convolution kernels, T1 represents the stride of time convolution, D represents the number of depth convolution kernels, and C eeg represents the number of channels of EEG data, and Conv represents two-dimensional convolution; S22: To extract and fuse the features of D*F1 one-dimensional time signals in the spatio-temporal features A depthwise separable convolution feature extraction module based on the Inception architecture is designed. The Inception architecture includes four branches. Branch one includes ordinary convolution, depthwise separable convolution with a convolution kernel of 1*T2, average pooling, and a Dropout layer. Branch two includes ordinary convolution, depthwise separable convolution with a convolution kernel of 1*T3, average pooling, and a Dropout layer. Branch three includes average pooling and ordinary convolution. Branch four includes ordinary convolution; The intermediate feature of branch one is obtained through branch one The intermediate feature of branch two is obtained through branch two The intermediate feature of branch three is obtained through branch three The intermediate feature of branch four is obtained through branch four The formula is expressed as follows: In the formula, DepthwiseSeparableConv represents the depthwise separable convolutional layer, T2 represents the convolutional kernel size of the depthwise separable convolution in branch one, T3 represents the convolutional kernel size of the depthwise separable convolution in branch two, and S represents the stride of the ordinary convolution in branch four; S23: Concatenate the intermediate features of the four branches obtained in step S22 in the feature map number dimension to obtain multi-scale features Then, use depthwise separable convolution to perform feature fusion, and obtain the output features of the EEG feature extraction module through the Dropout layer The formula is as follows: In the formula, Concat represents the concatenation function, dim represents that multiple features are concatenated along the specified dimension, and C fNIRS represents the number of channels of fNIRS data, T4 represents the convolutional kernel size of depthwise separable convolution, and N C represents the number of classification categories of the motor imagery task; S24: For the fNIRS data X fNIRS , first perform convolution operations on HbO and HbR in the fNIRS data using three convolutional kernels of different scales, and then splice the three temporal features obtained by convolution in the dimension of the number of feature maps to obtain multi-scale features Then, obtain multi-scale fusion features through the PointwiseConv layer The formula is expressed as follows: In the formula, F2 represents the number of fNIRS temporal convolutional kernels, N1 represents the size of the first convolutional kernel among the three convolutional kernels, N2 represents the size of the second convolutional kernel among the three convolutional kernels, and N3 represents the size of the third convolutional kernel among the three convolutional kernels; S25: Take the multi-scale fusion features Through the designed Transformer module with positional encoding, obtain the fused feature of the encoded positional information Then concatenate with the number of classification categories N C of the motor imagery task CT category marker vectors F to obtain the enhanced feature Then input into L Transformer encoders to obtain the self-attention feature C Finally, intercept the first N feature vectors as the output features of the fNIRS feature extraction module The formula is expressed as follows: wherein, 3DTo1D represents converting 3D fNIRS data into 1D fNIRS data according to the 3D representation of fNIRS data, 1DTo3D represents converting 1D fNIRS data into 3D fNIRS data according to the 3D representation of fNIRS data, a represents the length of the 3D schematic diagram of fNIRS data, b represents the width of the 3D schematic diagram of fNIRS data, MSA represents the multi-head self-attention transformation function, LN represents layer normalization, represents the output of the first Transformer encoder, represents the output of the k-th Transformer encoder, where k = 2, 3, …, L.

4. A method for decoding late fusion of motor imagery based on EEG-fNIRS data according to claim 3, characterized in that, In step S3, for the output features of the EEG feature extraction module through the adaptive average pooling layer, and then using Squeeze for dimensionality reduction, the output features of the fNIRS feature extraction module are obtained data with the same dimension Then it is combined with to obtain the EEG-fNIRS hybrid feature F Hybrid , which is expressed by the formula as follows: In the formula, Squeeze represents the dimensionality reduction function, and AdaptiveAvgPool represents the adaptive average pooling layer.

5. A method for decoding late fusion of motor imagery based on EEG-fNIRS data according to claim 4, characterized in that, In step S4, the EEG-fNIRS hybrid feature F Hybrid is input into a multi-layer perceptron for task classification to obtain a predicted label y test , that is, the classification result of the motor imagery task is obtained, which is expressed by the following formula: y test = MLP(F Hybrid ) where MLP represents a multi - layer perceptron.

Images