Electroencephalogram near-infrared motor imagery recognition method based on modal deformable attention feature fusion

Abstract

The invention provides an electroencephalogram near-infrared motor imagery recognition method based on modal deformable attention feature fusion, which adopts a public electroencephalogram near-infrared motor imagery data set MI for validity verification, and comprises the following steps: firstly, preprocessing the public data set, and establishing a training set and a test set; secondly, constructing an electroencephalogram near-infrared modal deformable attention feature fusion model; inputting the preprocessed training set into the model for training; and finally, inputting the test set into the trained model, and testing the performance of the model. The method has the advantages that a space-time double-branch attention refinement module is designed, and an electroencephalogram near-infrared space-time long-range dependency relationship and local-global effective features are extracted; a modal deformable attention feature fusion strategy is provided, and a multi-scale electroencephalogram near-infrared space-time most discriminative feature is adaptively captured through a multi-modal space-time deformable sampling mechanism. Effectiveness verification is carried out on the method on MI, the average recognition accuracy reaches 88.17%, and the method is superior to an existing optimal method.

Description

Technical Field

[0001] The present invention provides a method for electroencephalogram near-infrared motor imagery recognition based on modal deformable attention feature fusion. Background Technique

[0002] The hybrid Brain-Computer Interface (BCI) technology that fuses multi-modal information has become a research hotspot. Research shows that the hybrid BCI technology based on electroencephalogram near-infrared can improve the recognition accuracy and practicality of the hybrid BCI system by integrating the advantages of high temporal resolution of electroencephalogram signals (Electroencephalogram, EEG) and high spatial resolution of near-infrared signals (Functional near-infrared spectroscopy, NIRS). As a long-time series multi-channel signal, EEG has high temporal and spatial variability. Therefore, identifying the spatio-temporal dependence relationship over long distances in the non-stationary state is the key to EEG decoding. At the same time, as multi-channel complex time series data, NIRS signals are particularly important for accurately decoding NIRS signals by capturing the most discriminative channel information and enhancing the model's ability to extract global features. In addition, on the basis of effectively extracting the features of EEG signals and NIRS signals, studying an adaptive feature fusion strategy considering the characteristics of multi-modal data also has a positive effect on mining the most discriminative fusion features of electroencephalogram near-infrared.

[0003] However, the existing EEG signal feature extraction methods based on the attention mechanism are difficult to capture the global spatio-temporal information interaction in the long-time series, ignoring the long-distance dependence relationship and dynamic spatio-temporal features in the EEG signals. For the research on NIRS signal feature extraction, most of the existing methods use complex attention modules to obtain high recognition accuracy, ignoring the extraction of local key channel information and being difficult to capture the global spatio-temporal interaction relationship. At the same time, the traditional multi-modal fusion method based on direct feature splicing cannot adaptively capture effective fusion features by combining the characteristics of the electroencephalogram near-infrared multi-modal data structure.

[0004] Therefore, it is necessary to develop the most discriminative feature extraction method based on the characteristics of EEG signals and NIRS signals, and effectively realize the electroencephalogram near-infrared feature fusion through an adaptive feature fusion strategy, so as to improve the recognition accuracy and robustness of the hybrid BCI. Summary of the Invention

[0005] To solve the problems existing in the existing methods in aspects such as electroencephalogram (EEG), near-infrared (NIR) feature extraction, and multimodal feature fusion, the present invention proposes an EEG-NIR motor imagery recognition method based on modality-deformable attention feature fusion. Through four steps of data processing, model construction, model training, and model testing, high-accuracy recognition of brain information is achieved. It has been verified for effectiveness on the publicly available motor imagery EEG dataset MI containing EEG-NIR bimodal data. The recognition accuracy of a single subject can reach 98.95%, the K value of a single subject can reach 0.979, the average recognition accuracy of all subjects can reach 88.17%, and the K value of all subjects can reach 0.758, which is better than other existing optimal methods.

[0006] The present invention proposes an EEG-NIR motor imagery recognition method based on modality-deformable attention feature fusion. First, by constructing an EEG spatio-temporal attention feature extraction module, the ability to represent the hidden spatio-temporal dependence relationships between different time series and different electrode channels of EEG signals is enhanced; second, through the NIR spatio-temporal attention refinement feature extraction module, local key channels are adaptively captured, and through the global attention mechanism, the global spatio-temporal information interaction and feature extraction ability of NIR signals are enhanced; then, a deformable attention feature fusion strategy is adopted to automatically capture the most discriminative features in the multi-scale EEG-NIR fusion features, reduce the redundant information of the model, and improve the generalization performance of the model; finally, the motor imagery classification module is used to perform brain movement intention pattern recognition.

[0007] An EEG-NIR motor imagery recognition method based on modality-deformable attention feature fusion according to an embodiment of the present invention includes the following steps:

[0008] Step 1: Preprocess the publicly available motor imagery EEG dataset MI containing EEG-NIR multimodal data obtained in advance, and establish a training set and a test set corresponding to this dataset;

[0009] Step 2: Use the Pytorch deep learning framework to construct the model structure of an EEG-NIR modality-deformable attention feature fusion model with adaptive channel encoding;

[0010] Step 3: Input the preprocessed training set in Step 1 into the constructed model respectively for model training;

[0011] Step 4: Input the preprocessed test set in Step 1 into the trained model respectively to obtain the decoding performance indicators of the motor imagery task,

[0012] Wherein:

[0013] In the said Step 1, the data preprocessing includes band-pass filtering, baseline correction, re-referencing, and bad channel rejection;

[0014] In step 2, the constructed deformable attention feature fusion model for electroencephalogram near-infrared modality with adaptive channel coding includes: an electroencephalogram spatio-temporal attention feature extraction module, a near-infrared spatio-temporal attention refined feature extraction module, an electroencephalogram near-infrared modality deformable attention feature fusion module, and a motor imagery classification module. The electroencephalogram spatio-temporal attention feature extraction module is connected in parallel with the near-infrared spatio-temporal attention refined feature extraction module, and the extracted electroencephalogram and near-infrared features are respectively input into the electroencephalogram near-infrared modality deformable attention feature fusion module. Among them, the electroencephalogram spatio-temporal attention feature extraction module is used to extract the spatio-temporal dependence relationships hidden between different time series and different electrode channels of electroencephalogram signals; the near-infrared spatio-temporal attention refined feature extraction module is used to adaptively capture local key channels, and through the global attention mechanism, enhance the global spatio-temporal information interaction and feature extraction ability of near-infrared signals; the modality deformable attention feature fusion module is used to automatically capture the most discriminative features in the multi-scale electroencephalogram near-infrared fusion features, reduce the redundant information of the model, and improve the generalization performance of the model;

[0015] In step 3:

[0016] When training the intention recognition model, first adopt the early stopping strategy for the first training of the model. After saving the model training parameters, then conduct the second training of the model. Use the cross-entropy function to calculate the error between the output of the lightweight gradient boosting decision tree classification block and the label, and iteratively update the model parameters through error backpropagation and stochastic gradient descent;

[0017] In step 4:

[0018] When calculating the decoding performance index of the motor imagery electroencephalogram signal, select the time window from 0.5 s before the occurrence of the motor imagery task event to 4 s after the task event to extract the electroencephalogram signal of each trial in the test set. The sampling frequency is 250 Hz, and the electroencephalogram signal of each trial contains 1000 sampling points. Input the divided data into the trained spatio-temporal dynamic aggregation and spectral adaptive filtering network model, and use the recognition accuracy (ACC), Kappa value, F1 score, and AUC value to evaluate the electroencephalogram signal decoding effect of the model.

[0019] The main advantages of the electroencephalogram near-infrared motor imagery recognition method based on modality deformable attention feature fusion proposed by the present invention include:

[0020] 1. By designing a learnable embedding attention block, the present invention divides the electroencephalogram signal into spatio-temporal sequence segments and performs position encoding, effectively mining the spatio-temporal dependence relationships between different time points of the electroencephalogram signal, solving the problem of difficult extraction of global spatio-temporal information interaction in long time series, and improving the model recognition accuracy;

[0021] 2. Aiming at the deficiencies of traditional near-infrared signal feature extraction methods, the present invention proposes a local-global cross-channel attention encoding block, which can adaptively capture local details and global spatio-temporal correlations between different near-infrared channels, enhancing the spatio-temporal global information interaction ability of near-infrared signals;

[0022] 3. Combining the characteristics of the electroencephalogram near-infrared multimodal data structure, the present invention proposes a deformable attention feature fusion strategy, which adaptively captures multi-scale electroencephalogram near-infrared fusion features through a deformable sampling mechanism, improving the robustness and fusion effect of the model;

[0023] 4. In the electroencephalogram and near-infrared feature extraction modules, the present invention adopts a multi-head self-attention encoder-decoder structure, and through mechanisms such as layer normalization and residual connection, effectively learns the global spatio-temporal dependence relationships between signal segments and captures discriminative spatio-temporal features;

[0024] 5. During the feature extraction process, the present invention introduces class token vectors and position embedding vectors, enabling the model to learn electroencephalogram category information and the position information of signal segments during training, further enhancing the model's feature extraction ability;

[0025] 6. In the electroencephalogram near-infrared modality deformable attention feature fusion module, the present invention adopts dilated causal convolution and deformable large convolution kernel attention units, and through large convolution kernels, fully understands voxel context, and constructs an effective feature fusion mechanism with fewer parameters and calculations. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 FIG. is a flowchart of an electroencephalogram near-infrared motor imagery recognition method based on modality deformable attention feature fusion according to an embodiment of the present invention.

[0027] Figure 2 FIG. is a structural diagram of an electroencephalogram spatio-temporal attention feature extraction module according to an embodiment of the present invention.

[0028] Figure 3 FIG. is a structural diagram of a near-infrared spatio-temporal attention refinement feature extraction module according to an embodiment of the present invention.

[0029] Figure 4 FIG. is a structural diagram of an electroencephalogram near-infrared modality deformable attention feature fusion module according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0030] The overall process of an electroencephalogram near-infrared motor imagery recognition method based on modality deformable attention feature fusion according to an embodiment of the present invention is as Figure 1 shown, and it includes:

[0031] Step S1: Preprocess the publicly available electroencephalogram near-infrared dual-modal motor imagery electroencephalogram dataset MI obtained in advance, and establish the corresponding training set and test set for this dataset. The data preprocessing includes band-pass filtering, baseline correction, re-referencing, and bad channel rejection, specifically including:

[0032] Step S1.1: Band-pass filtering, including: Selecting MI motor imagery electroencephalogram and near-infrared signals as data inputs, where a band-pass convolution kernel with a bandwidth of 4 - 38 Hz is selected to adapt to the situation that the information of motor imagery in the electroencephalogram signal data exists in the frequency band of 4 - 32 Hz, and performing band-pass filtering on the intercepted data;

[0033] Step S1.2: Baseline calibration, including making each segment of data at the same baseline starting point to avoid the baseline drift phenomenon existing in the long-term acquisition of electroencephalogram and near-infrared signals. Perform baseline correction on the electroencephalogram signal, using the mean value of the data 200 ms before the task event as the baseline, and taking the relative value relative to the baseline after the task event as the new electroencephalogram signal data value;

[0034] Step S1.3: Perform re-referencing, including selecting the central electrode Cz on the top of the head as the reference electrode to observe the changes in the electrode potentials of the whole brain when the task occurs;

[0035] Step S1.4: Reject bad channels, including rejecting bad segments of electroencephalogram signals exceeding plus or minus 100 uV;

[0036] Step S2: Use the Pytorch deep learning framework to construct a model of an electroencephalogram near-infrared modality deformable attention feature fusion model with adaptive channel coding,

[0037] This model includes a parallel electroencephalogram spatio-temporal attention feature extraction module and a near-infrared spatio-temporal attention refinement feature extraction module. The electroencephalogram and near-infrared features extracted by the two modules are respectively input into the electroencephalogram near-infrared modality deformable attention feature fusion module for multi-modal feature fusion, and the fusion result is input into the motor imagery classification module for recognition and classification. Among them, the electroencephalogram spatio-temporal attention feature extraction module is used to extract the spatio-temporal dependence relationships hidden between different time series and different electrode channels of the electroencephalogram signal; the near-infrared spatio-temporal attention refinement feature extraction module is used to adaptively capture local key channels, and through the global attention mechanism, enhances the global spatio-temporal information interaction and feature extraction ability of the near-infrared signal; the modality deformable attention feature fusion module is used to automatically capture the most discriminative features in the multi-scale electroencephalogram near-infrared fusion features, reduce the redundant information of the model, and improve the generalization performance of the model;

[0038] Construct an electroencephalogram spatio-temporal attention feature extraction module, a near-infrared spatio-temporal attention refinement feature extraction module, an electroencephalogram near-infrared modality deformable attention feature fusion module, and a motor imagery classification module,

[0039] Among them:

[0040] A) The EEG spatio-temporal attention feature extraction module is used to extract the temporal and spatial domain features of EEG signals, capture the potential long-range dependencies between different time positions and different spatial electrodes of EEG signals, such as Figure 2 shown, including:

[0041] A1) Divide the input EEG signal into a series of EEG signal segments i = 1, 2,..., N, to capture the potential spatio-temporal long-range dependencies in the EEG signal sequence, where and represent two different three-dimensional arrays, B represents BatchSize, C represents the number of feature map channels, H and W represent the number of electrodes in the spatial domain and the number of time domain sampling points of the EEG signal respectively, and N and D represent the number of divided EEG signal segments x i and the signal segment size;

[0042] A2) Establish a learnable class label vector x class , and splice the class label vector x class with the EEG signal segment x i to generate an EEG signal segment with class labels for learning EEG category information during the training of the EEG spatio-temporal attention feature extraction module, where represents a three-dimensional array, and x class is randomly initialized during training and updated through training;

[0043] A3) Define a learnable position embedding vector E pos , and add the position embedding vector E pos to element-wise to obtain the EEG signal feature M pos with position information embedded, which is used to learn the potential long-range dependencies between different time positions and different spatial electrodes of the EEG signal segment . Among them, has learnable parameters with a normal distribution, and the value range of the learnable parameters is [-1, 1], represents a three-dimensional array, and the calculation formula for the process is:

[0044] M pos = Concat(x i , x class ) + E pos (1),

[0045] In the formula, Concat(·) represents the splicing function;

[0046] A4) For Mpos Use Dropout regularization to reduce model overfitting, and use the EEG signal features M with positional information embedding after regularization pos Input into the multi-head self-attention encoder-decoder to learn the global spatio-temporal dependencies between EEG signal segments and capture discriminative EEG spatio-temporal features. The multi-head self-attention encoder-decoder consists of a multi-head self-attention layer MSA and a multi-layer perceptron layer MLP. Layer normalization is applied before each layer, and residual connections are applied after each layer. The calculation formula for this process is:

[0047] M l = MSA(LN(M pos )) + M pos , l = 1, 2, …, L (2),

[0048] where LN(·) represents layer normalization, L represents the number of stackable multi-head self-attention encoder-decoders. The next multi-head self-attention encoder-decoder receives the output from the previous one, forming a serial structure to sequentially extract EEG features. M l represents the output features of the l-th multi-head self-attention encoder-decoder. The calculation formula for the multi-head self-attention layer MSA is:

[0049]

[0050] where M q , M k , M v are obtained by multiplying M pos with the learnable transformation matrices Q l , K l , V l respectively. is the transpose of M k , Softmax(·) represents the activation function, represents the scaling factor;

[0051] The calculation formula for the multi-layer perceptron layer MLP is:

[0052] where represents the l-th feature output after being processed by MLP. The MLP layer consists of a linear layer, an activation function GELU, and a Dropout layer.

[0053] B) Construct a near-infrared spatio-temporal attention refinement feature extraction module, as shown in Figure 3 , which is used to extract the spatio-temporal local-global features of near-infrared signals, adaptively capture the cross-channel interaction information of spatio-temporal local features, and the inter-channel dependency relationship of spatio-temporal global features. Among them, the near-infrared signals include oxygenated near-infrared signal x hbo and deoxygenated near-infrared signal xhbr , where:

[0054] For the oxygenated near-infrared signal x hbo The spatio-temporal local-global feature extraction method includes:

[0055] B1) Input the oxygenated near-infrared signal x hbo into the global average pooling layer for global average pooling, reduce the dimension of the input data x hbo to 1×1 without changing the number of channels, and determine and output the oxygenated near-infrared feature M hbo , and the calculation formula. The calculation formula for this process is:

[0056] M hbo = GAP(x hbo ) (5),

[0057] where GAP(﹒) represents the global average pooling function, and the specified output size is 1×1;

[0058] B2) After performing dimensional transformation on M hbo , use the spatio-temporal local feature cross-channel adaptive convolution kernel to perform feature extraction on the dimension-transformed M hbo to obtain the near-infrared oxygenated local feature M hbol , where the size of the adaptive convolution kernel is 2×k, and the parameter padding of the convolution operation is k represents the size of the adaptive convolution kernel, which is determined by the variable t. t is determined by the number of channels C of the feature map of the feature M hbo , the decay coefficient a, and the offset coefficient b. The calculation formula for this process is:

[0059]

[0060] where represents floor division, and abs represents taking the absolute value;

[0061] B3) After performing Sigmoid non-linear transformation on the near-infrared oxygenated local feature M hbol , multiply the oxygenated near-infrared signal x hbo element-wise with the near-infrared oxygenated local feature M hbol to apply the extracted adaptive local feature channel attention weight to the input oxygenated near-infrared signal x hbo , and obtain the near-infrared oxygenated local effective feature The calculation process is expressed as:

[0062]

[0063] where AdConv(﹒) is the adaptive convolution operation, ⊙ represents element-wise multiplication of two vectors, and Sigmoid(·) is the non-linear activation function;

[0064] Similarly, for the deoxygenated near-infrared signal x hbr the spatio-temporal local-global feature extraction method

[0065] includes:

[0066] B4) Input the deoxygenated near-infrared signal x hbr into the adaptive two-dimensional average pooling layer, reduce the dimension of the input data x hbr to 1×1 without changing the number of channels, and determine and output the deoxygenated near-infrared feature M hbr , and the calculation formula is as follows:

[0067] M hbr = GAP(x hbr ) (9),

[0068] where GAP(﹒) represents the adaptive two-dimensional average pooling function, and the specified output size is 1×1;

[0069] B5) After performing dimensional transformation on M hbr , use the spatio-temporal local feature cross-channel adaptive convolution kernel to perform feature extraction on the dimension-transformed M hbr to obtain the deoxygenated near-infrared local feature M hbrl . By utilizing the correlation features of the channels to calculate the weight coefficients between channels, where the size of the adaptive convolution kernel is 2×p, and the parameter padding of the convolution operation is p represents the size of the adaptive convolution kernel, which is determined by the variable q. q is determined by the number of channels C, the attenuation coefficient a, and the offset coefficient b of the feature M hbr , and the calculation formula for the process is:

[0070]

[0071]

[0072] where represents rounding down, and abs represents taking the absolute value,

[0073] B6) After performing Sigmoid non-linear transformation on the deoxygenated near-infrared local feature M hbrl , multiply the deoxygenated near-infrared signal x hbr element-wise with the deoxygenated near-infrared local feature M hbrl to apply the extracted adaptive local feature channel attention weight to the input deoxygenated near-infrared signal x hbr , and obtain the deoxygenated near-infrared local effective feature The calculation process is expressed as:

[0074]

[0075] In the formula, AdConv is the adaptive channel coding operation, ⊙ represents the element-wise multiplication of two vectors, and Sigmoid(·) is the non-linear activation function;

[0076] B7) The near-infrared oxygenated local effective features After being divided according to the data division method in the EEG spatio-temporal attention feature extraction module and position-coded, are input into the multi-head self-attention encoder-decoder to obtain the oxygenated near-infrared global discriminative features Among them, the structure of the multi-head self-attention encoder-decoder is consistent with the structure adopted by the EEG spatio-temporal attention feature extraction module, specifically including:

[0077] B71) The input near-infrared oxygenated features Are divided into a series of feature segments to capture the potential spatio-temporal dependence relationships in the near-infrared oxygenated signal sequence;

[0078] B72) A learnable class token vector x hbol is established. The class token vector x hbol is concatenated with the near-infrared oxygenated local effective feature segments to generate near-infrared oxygenated feature segments with class tokens for learning the near-infrared oxygenated feature category information, where x hbol is randomly initialized during training and its parameters are updated through training;

[0079] B73) A learnable position embedding vector E hbol is defined. The position embedding vector E hbol is added element-wise to to obtain the near-infrared oxygenated features with position information embedded for learning the potential long-range dependence relationships between different time positions and different spatial electrodes of the near-infrared oxygenated feature segments. Among them, E hbol has learnable parameters with a normal distribution, and the value range of the learnable parameters is [-1, 1]. The calculation formula for the process is:

[0080]

[0081] B74) M poshbol is processed using Dropout regularization to reduce model overfitting. The near-infrared oxygenated features M with position information embedded after the regularization processing poshbolInput the multi-head self-attention encoder-decoder to learn the global spatio-temporal dependencies among the near-infrared oxygenated feature segments, and capture the discriminative spatio-temporal features of the near-infrared oxygenated features. The multi-head self-attention encoder-decoder consists of a multi-head self-attention layer MSA and a multi-layer perceptron layer MLP. Layer normalization is performed before each layer, and residual connections are used after each layer. The calculation formula for this process is:

[0082] M l = MSA(LN(M poshbol )) + M poshbol , l = 1, 2, …, (14),

[0083] where LN(·) represents layer normalization, L represents the number of stackable multi-head self-attention encoder-decoders, the next multi-head self-attention encoder-decoder receives the output from the previous one, and they form a serial structure to extract EEG features in sequence. M l represents the output features of the l-th multi-head self-attention encoder-decoder. The calculation formula for the multi-head self-attention layer MSA is:

[0084]

[0085] where, M qhbol , M khbol , M vhbol are obtained by multiplying M poshbol with the learnable transformation matrices Q lhbol , K lhbol , V lhbol respectively. is the transpose of M khbol ;

[0086] The calculation formula for the multi-layer perceptron layer MLP is:

[0087]

[0088] where represents the l-th feature output after being processed by the MLP. The MLP layer consists of a linear layer, an activation function GELU, and a Dropout layer;

[0089] B8) After dividing the near-infrared deoxygenated local effective features according to the data division method in the EEG spatio-temporal attention feature extraction module and performing position encoding, input them into the multi-head self-attention encoder-decoder to obtain the deoxygenated near-infrared global discriminative features where the structure of the multi-head self-attention encoder-decoder is consistent with the structure used in the EEG spatio-temporal attention feature extraction module, specifically including:

[0090] B81) The input near-infrared deoxygenated features Divided into a series of feature segments to capture potential spatio-temporal dependencies in the near-infrared deoxygenated signal sequence;

[0091] B82) Establish a learnable class label vector x hbrl , and splice the class label vector x hbrl with the near-infrared deoxygenated local effective feature segments to generate near-infrared deoxygenated feature segments with class labels for learning near-infrared deoxygenated feature category information, where x hbol is randomly initialized during training and its parameters are updated through training;

[0092] B83) Define a learnable position embedding vector E hbrl , and add the position embedding vector E hbrl element-wise with to obtain near-infrared deoxygenated features with position information embedding for learning potential long-range dependencies between different temporal positions and different spatial electrodes of near-infrared deoxygenated feature segments. Among them, E hbol has learnable parameters with a normal distribution, and the value range of the learnable parameters is [-1, 1]. The calculation formula for the process is:

[0093]

[0094] B84) Apply Dropout regularization to M poshbrl to reduce model overfitting. Input the regularized near-infrared deoxygenated features M with position information embedding poshbrl into the multi-head self-attention encoder-decoder to learn the global spatio-temporal dependencies between near-infrared deoxygenated feature segments and capture discriminative spatio-temporal features of near-infrared deoxygenated features. The multi-head self-attention encoder-decoder consists of a multi-head self-attention layer MSA and a multi-layer perceptron layer MLP. Layer normalization is used before each layer, and residual connection is used after each layer. The calculation formula for the process is:

[0095] M l = MSA(LN(M poshbrl )) + M poshbrl , l = 1, 2, …, L (18),

[0096] where LN(·) represents layer normalization processing, L represents the number of stackable multi-head self-attention encoder-decoders. The next multi-head self-attention encoder-decoder receives the output from the previous multi-head self-attention encoder-decoder to form a serial structure to extract EEG features in sequence. M l represents the output features of the l-th multi-head self-attention encoder-decoder. The calculation formula for the multi-head self-attention layer MSA is:

[0097]

[0098] Wherein, M qhbrl , M khbrl , M vhbrl are respectively obtained by multiplying M poshbrl with the learnable transformation matrix Q lhbrl , K lhbrl , V lhbrl ; is the transpose of M khbrl ;

[0099] The calculation formula of the multi-layer perceptron layer MLP is:

[0100]

[0101] Wherein represents the l-th feature output after being processed by the MLP. The MLP layer is composed of a linear layer, an activation function GELU, and a Dropout layer;

[0102] C) Construct an electroencephalogram near-infrared modality deformable attention feature fusion module for fusing electroencephalogram and near-infrared multi-modal features, design a deformable attention feature fusion strategy, and adaptively capture the most discriminative features of multi-scale electroencephalogram and near-infrared through a deformable sampling mechanism, as Figure 4 shown, including:

[0103] C1) Perform feature splicing on to obtain the electroencephalogram near-infrared spliced feature M EEG-fNIRS ;

[0104] C2) Perform layer normalization processing on M EEG-fNIRS ;

[0105] C3) Input M EEG-fNIRS into the electroencephalogram near-infrared modality deformable attention feature fusion module for multi-modal feature fusion to obtain the final electroencephalogram near-infrared fusion feature

[0106] wherein the electroencephalogram near-infrared modality deformable attention feature fusion module is composed of a series of residual blocks, each residual block is composed of two dilated causal convolution layers, a deformable large convolution kernel attention unit is adopted in the dilated causal convolution layer, and batch normalization processing and an ELU activation function are adopted after each convolution layer;

[0107] In the dilated causal convolution layer, a large convolution kernel is used for the feature map obtained by each dilated causal convolution to fully understand the simplified attention mechanism of the voxel context. The large convolution kernel provides a receptive field similar to the self-attention mechanism. By using depth convolution, depth expansion convolution, and 1×1 convolution, a large convolution kernel can be constructed with fewer parameters and calculations.

[0108] E) Construct a motor imagery classification module for motor intention pattern recognition, including After the Flatten operation, it becomes a one-dimensional vector and then perform classification to obtain the final motor imagery recognition result.

[0109]

[0110] where Flatten(·) is used to convert a multi-dimensional array into a one-dimensional array, Linear(·) is used to set the fully connected layer in the model, and Predict represents the final obtained motor imagery classification result.

[0111] When training the model of the electroencephalogram near-infrared modality deformable attention feature fusion network, an early stopping strategy is adopted. The training set divided in step S1 is re-divided into a training set and a validation set. When the recognition accuracy of the electroencephalogram near-infrared modality deformable attention feature fusion network on the validation set remains stable after N consecutive training iterations, stop the training process in advance and save the model parameters of the electroencephalogram near-infrared modality deformable attention feature fusion network at this time;

[0112] Then adopt a secondary training strategy. Based on the model parameters at the end of the first training, start the second training. In the second training, use the training set divided in step S1 as the model training set and follow the above early stopping strategy.

[0113] During the model training process of the electroencephalogram near-infrared modality deformable attention feature fusion network, use the cross-entropy loss function to calculate the classification loss J, and the calculation formula is as follows:

[0114]

[0115] where p i is the i-th conditional probability generated by the model of the electroencephalogram near-infrared modality deformable attention feature fusion network, l i is the i-th class of the label set, ω(·) represents the indicator function, Θ represents the learnable parameters in the electroencephalogram near-infrared modality deformable attention feature fusion network, ||·|| is the regularization term used to alleviate the overfitting problem, λ represents the trade-off regularization weight, B represents the Batchsize size, and the learning rate during the training process is 1×10 -3 .

[0116] Regard the motor imagery task dataset as a classification task with the same number of samples for each class, and use the recognition accuracy Acc to describe the classification performance of the electroencephalogram near-infrared modality deformable attention feature fusion network. The calculation formula is as follows:

[0117]

[0118] where N true represents the number of correctly classified samples, and N total represents the total number of samples;

[0119] Taking the dataset used as a motor imagery classification task, and using the bias of the deformable attention feature fusion network in the EEG-NIRS modality as an evaluation index, the Kappa coefficient κ is used to measure whether there is a classification bias towards a certain class in the classification of the deformable attention feature fusion network in the EEG-NIRS modality. The calculation of the Kappa coefficient is based on the confusion matrix, and its value ranges from -1 to 1. The calculation formula of the Kappa coefficient is as follows:

[0120]

[0121] where P0 is the classification accuracy rate, and P e is the penalty term for classification bias, and its calculation formula is as follows:

[0122]

[0123] where c represents the total number of classifications, and a i and b i represent the sums of the i-th row and the i-th column in the confusion matrix respectively. From this, according to the calculation formula of this Kappa, when there is a bias towards a certain class in the classification of the deformable attention feature fusion network in the EEG-NIRS modality, that is, when the confusion matrix is unbalanced, the value of Kappa will decrease.

[0124] To verify the effectiveness of the proposed method, performance tests were carried out on the MI dataset. As can be seen from the table, the ACC and Kappa values of the proposed EEG-NIRS motor imagery recognition method based on modal deformable attention feature fusion on MI reached 88.17% and 0.758 respectively, both higher than the existing state-of-the-art recognition methods.

[0125] Table 1 Performance of the EEG-NIRS motor imagery recognition method based on modal deformable attention feature fusion on the MI dataset

[0126]

[0127] Table 2 Performance comparison between the EEG-NIRS motor imagery recognition method based on modal deformable attention feature fusion and the existing latest methods

[0128]

[0129] To illustrate the advantages of the method proposed in the present invention in terms of model recognition performance, the performance indicators of the latest motion intention recognition models in recent years on the same test set are compared. As can be seen from Table 1 and Table 2, the motion intention recognition accuracy (ACC), Kappa value, F1 score, and AUC value of the electroencephalogram near-infrared motor imagery recognition method based on modal deformable attention feature fusion proposed in the present invention are all superior to the existing advanced methods and can be applied to the hybrid BCI interaction system.

[0130] The electroencephalogram near-infrared motor imagery recognition method based on modal deformable attention feature fusion provided by the present invention has been described in detail above. However, it is obvious that the scope of the present invention is not limited thereto. Without departing from the scope of protection defined by the appended claims, various changes to the above embodiments are within the scope of the present invention.

Claims

1. Modeling method of EEG near-infrared modality deformable attention feature fusion model for multimodal motor imagery recognition, characterized by include: A) Construct an EEG spatiotemporal attention feature extraction module to extract EEG signal time and space domain features and capture the potential long-range dependencies between EEG signals at different time positions and different spatial electrodes, including: A1) Input EEG signal Divided into a series of EEG signal segments It is used to capture the potential long-range spatiotemporal dependencies in EEG signal sequences. and represents two different three-dimensional arrays, B represents BatchSize, C represents the number of feature map channels, H and W represent the number of electrodes in the spatial domain and the number of sampling points in the temporal domain of the EEG signal, respectively, and N and D represent the EEG signal segments x after division. i The number of and signal fragment size; A2) Create a learnable class label vector x class , the class label vector x class With EEG signal fragment x i Splicing to generate EEG signal segments with class labels It is used to learn EEG category information when training the EEG spatiotemporal attention feature extraction module. Represents a three-dimensional array, x class Initialize randomly during training and update parameters through training; A3) Define a learnable position embedding vector E pos , embed the position into vector E pos and Add the elements together to get the EEG signal feature M embedded with position information pos , used to learn EEG signal fragments Potential long-range dependencies between electrodes at different temporal positions and in different spatial locations, among which, The learnable parameters have a normal distribution and the range of learnable parameters is [-1,1]. Represents a three-dimensional array, and the calculation formula of the process is: M pos =Concat(x i ,x class) +E pos (1), Where Concat(·) represents the concatenation function; A4) for M pos Dropout regularization is used to reduce model overfitting, and the EEG signal feature M with position information embedded after regularization is pos The multi-head self-attention codec is input to learn the global spatiotemporal dependency between EEG signal segments and capture the discriminative EEG spatiotemporal features. The multi-head self-attention codec consists of a multi-head self-attention layer MSA and a multi-layer perceptron layer MLP. Layer normalization is used before each layer and residual connection is used after each layer. The calculation formula of the process is: M l =MSA(LN(M pos ))+M pos ,l=1,2,…,L (2), Where LN(·) represents the layer normalization process, L represents the number of stackable multi-head self-attention codecs, and the next multi-head self-attention codec receives the output from the previous multi-head self-attention codec to form a serial structure to extract EEG features in sequence. l Represents the output feature of the l-th multi-head self-attention encoder-decoder. The calculation formula of the multi-head self-attention layer MSA is: Where M q , M k , M v M pos and the learnable transformation matrix Q l , K l , V l Multiplying them together, M k The transpose of , Softmax(·) represents the activation function, represents the scaling factor; The calculation formula of the multi-layer perceptron layer MLP is: In the formula Represents the lth feature output after MLP processing. The MLP layer consists of a linear layer, an activation function GELU, and a Dropout layer. B) Construct a near-infrared spatiotemporal attention refinement feature extraction module to extract the spatiotemporal local-global features of near-infrared signals, adaptively capture the cross-channel interaction information of spatiotemporal local features and the inter-channel dependency of spatiotemporal global features, where the near-infrared signal includes the oxygen-containing near-infrared signal x hbo and deoxygenated near infrared signal x hbr ,in: Oxygen-containing near-infrared signal x hbo The spatiotemporal local-global feature extraction methods include: B1) The oxygen-containing near-infrared signal x hbo Enter the global average pooling layer for global average pooling, and convert the input data x hbo The dimension is reduced to 1×1 without changing the number of channels, and the oxygen-containing near-infrared feature M is output. hbo , the calculation formula of the process is: M hbo =GAP(x hbo ) (5), Where GAP(·) represents the global average pooling function, and the output size is specified as 1×1; B2) for M hbo After the dimension transformation, the spatial-temporal local feature cross-channel adaptive convolution kernel is used to align the dimension-transformed M hbo Perform feature extraction to obtain the near-infrared oxygen-containing local feature M hbol , where the size of the adaptive convolution kernel is 2×K, and the parameter padding of the convolution operation is k represents the size of the adaptive convolution kernel, which is determined by the variable t, which is determined by the feature M hbo The number of feature map channels C, the attenuation coefficient a and the offset coefficient b are determined, and the calculation formula of the process is: In the formula Indicates rounding down, abs indicates taking the absolute value; B3) In the near-infrared oxygen-containing local feature M hbol After Sigmoid nonlinear transformation, the oxygen-containing near-infrared signal x hbo and near-infrared oxygen local features M hbol Perform element-wise multiplication to apply the extracted adaptive local feature channel attention weights to the input oxygen-containing near-infrared signal x hbo , and obtain the near-infrared oxygen-containing local effective characteristics The calculation process is expressed as: Where AdConv(·) is the adaptive convolution operation, ⊙ represents the element-by-element multiplication of two vectors, and Sigmoid(·) is the nonlinear activation function; Similarly, for the deoxygenated near-infrared signal x hbr The spatiotemporal local-global feature extraction method includes: B4) Deoxygenation near infrared signal x hbr Input the adaptive two-dimensional average pooling layer, and convert the input data x hbr The dimension is reduced to 1×1 without changing the number of channels, and the near-infrared deoxygenation feature M is determined and output hbr , the calculation formula is as follows: M hbr =GAP(x hbr ) (9), Where GAP(·) represents the adaptive two-dimensional average pooling function, and the output size is specified as 1×1; B5) for M hbr After the dimension transformation, the spatial-temporal local feature cross-channel adaptive convolution kernel is used to align the dimension-transformed M hbr Perform feature extraction to obtain the near-infrared deoxygenation local feature M hbrl , the weight coefficients between channels are calculated by using the correlation characteristics of the channels, where the size of the adaptive convolution kernel is 2×p and the parameter padding of the convolution operation is p represents the adaptive convolution kernel size, which is determined by the variable q. q through the feature M hbr The number of channels C, the attenuation coefficient a and the offset coefficient b are determined, and the calculation formula of the process is: In the formula means round down, abs means take the absolute value, B6) Near infrared deoxidation local characteristics M hbrl After Sigmoid nonlinear transformation, the deoxygenated near-infrared signal x hbr Near infrared deoxygenation local characteristics M hbrl Perform element-wise multiplication to apply the extracted adaptive local feature channel attention weights to the input deoxygenated near infrared signal x hbr , and obtain the local effective characteristics of near-infrared deoxygenation The calculation process is expressed as: Where AdConv is the adaptive channel coding operation, ⊙ represents the element-by-element multiplication of two vectors, and Sigmoid(·) is the nonlinear activation function; B7) Near infrared oxygen-containing local effective characteristics After data division and position encoding according to the data division method in the EEG spatiotemporal attention feature extraction module, the data is input into the multi-head self-attention encoder-decoder to obtain the oxygen-containing near-infrared global discriminant feature. The multi-head self-attention encoder-decoder structure is consistent with the structure used in the EEG spatiotemporal attention feature extraction module, including: B71) Input near-infrared oxygen characteristics Divided into a series of characteristic segments to capture the potential spatiotemporal dependencies in the near-infrared oxygen signal sequence; B72) Create a learnable class label vector x hbol , the class label vector x hbol Splice with near-infrared oxygen-containing local effective feature fragments to generate near-infrared oxygen-containing feature fragments with class labels Used to learn near-infrared oxygen feature category information, where x hbol Initialize randomly during training and update parameters through training; B73) Define a learnable position embedding vector E hbol , embed the position into vector E hbol and Add by element to get the near-infrared oxygen feature with embedded position information It is used to learn the potential long-range dependencies between electrodes at different time positions and different spatial locations of near-infrared oxygen-containing feature fragments, where E hbol The learnable parameters have a normal distribution, and the range of the learnable parameters is [-1,1]. The calculation formula of the process is: B74) for M poshbol Dropout regularization is used to reduce model overfitting, and the near-infrared oxygen feature M with position information embedded after regularization is poshbol The multi-head self-attention codec is input to learn the global spatiotemporal dependency between near-infrared oxygen-containing feature fragments and capture the discriminative spatiotemporal features of near-infrared oxygen-containing features. The multi-head self-attention codec consists of a multi-head self-attention layer MSA and a multi-layer perceptron layer MLP. Layer normalization is used before each layer and residual connection is used after each layer. The calculation formula of the process is: M l =MSA(LN(M poshbol ))+M poshbol ,l=1,2,…,L (14), Where LN(·) represents the layer normalization process, L represents the number of stackable multi-head self-attention codecs, and the next multi-head self-attention codec receives the output from the previous multi-head self-attention codec to form a serial structure to extract EEG features in sequence. l Represents the output feature of the l-th multi-head self-attention encoder-decoder. The calculation formula of the multi-head self-attention layer MSA is: Where M qhbol , M khbol , M vhbol M poshbol and the learnable transformation matrix Q lhbol , K lhbol , V lhbol Multiplying them together, M khbol The transpose of , Softmax(·) represents the activation function, is the scaling factor; The calculation formula of the multi-layer perceptron layer MLP is: In the formula Represents the lth feature output by MLP processing. The MLP layer consists of a linear layer, an activation function GELU, and a Dropout layer; B8) Locally effective characteristics of near infrared deoxidation After data division and position encoding according to the data division method in the EEG spatiotemporal attention feature extraction module, the multi-head self-attention encoder-decoder is input to obtain the deoxygenated near-infrared global discriminant feature. The multi-head self-attention encoder-decoder structure is consistent with the structure used in the EEG spatiotemporal attention feature extraction module, including: B81) Input the near infrared deoxygenation feature Divided into a series of characteristic segments to capture the potential spatiotemporal dependencies in the near-infrared deoxygenation signal sequence; B82) Create a learnable class label vector x hbrl , the class label vector x hbrl Combined with the near-infrared deoxygenation local effective feature fragments to generate near-infrared deoxygenation feature fragments with class labels Used to learn near-infrared deoxygenation feature category information, where x hbol Initialize randomly during training and update parameters through training; B83) Define a learnable position embedding vector E hbrl , embed the position into vector E hbrl and Add the elements to get the near-infrared deoxygenation features with position information embedded It is used to learn the potential long-range dependencies between electrodes at different time positions and different spatial locations of near-infrared deoxygenation feature fragments, where E hbol The learnable parameters have a normal distribution, and the range of the learnable parameters is [-1,1]. The calculation formula of the process is: B84) for M poshbrl Dropout regularization is used to reduce model overfitting, and the near-infrared deoxygenation feature M with position information embedded after regularization is poshbrl Input the multi-head self-attention codec to learn the global spatiotemporal dependency between near-infrared deoxygenation feature fragments and capture the discriminative spatiotemporal features of near-infrared deoxygenation features. The multi-head self-attention codec consists of a multi-head self-attention layer MSA and a multi-layer perceptron layer MLP. Layer normalization is used before each layer and residual connection is used after each layer. The calculation formula of the process is: M l =MSA(LN(M poshbrl ))+M poshbrl ,l=1,2,…,L (18), Where LN(·) represents the layer normalization process, L represents the number of stackable multi-head self-attention codecs, and the next multi-head self-attention codec receives the output from the previous multi-head self-attention codec to form a serial structure to extract EEG features in sequence. l Represents the output feature of the l-th multi-head self-attention encoder-decoder. The calculation formula of the multi-head self-attention layer MSA is: Where M qhbrl , M khbrl , M vhbrl M poshbrl and the learnable transformation matrix Q lhbrl , K lhbrl , V lhbrl Multiplying them together, M khbrl The transpose of , Softmax(·) represents the activation function, is the scaling factor; The calculation formula of the multi-layer perceptron layer MLP is: In the formula Represents the lth feature output after MLP processing. The MLP layer consists of a linear layer, an activation function GELU and a Dropout layer. C) Construct an EEG-NIR modality deformable attention feature fusion module to fuse EEG and NIR multimodal features, design a deformable attention feature fusion strategy, and adaptively capture the most discriminative features of multi-scale EEG and NIR through a deformable sampling mechanism, including: C1) Perform feature splicing to obtain the EEG near-infrared splicing feature M EEG-fNIRS ; C2) M EEG-fNIRS Perform layer standardization; C3) M EEG-fNIRS Input the EEG near-infrared modality deformable attention feature fusion module for multimodal feature fusion to obtain the final EEG near-infrared fusion feature The EEG near-infrared modality deformable attention feature fusion module consists of a series of residual blocks, each of which consists of two dilated causal convolutional layers. The dilated causal convolutional layer uses a deformable large convolution kernel attention unit, and batch normalization and ELU activation function are used after each convolutional layer. In the dilated causal convolution layer, a large convolution kernel is used for the feature map obtained by each dilated causal convolution to fully understand the simplified attention mechanism of the voxel context. The large convolution kernel provides a receptive field similar to the self-attention mechanism. By using deep convolution, deep dilated convolution and 1×1 convolution, a large convolution kernel can be constructed with fewer parameters and calculations. D) Construct a motor imagery classification module for motor intention pattern recognition, including After the Flatten operation, it becomes a one-dimensional vector Then, the classification is performed to obtain the final motor imagery recognition result. Where Flatten(·) is used to convert a multidimensional array into a one-dimensional array, Linear(·) is used to set the fully connected layer in the model, and Predict represents the final motor imagery classification result.

2. The EEG near-infrared modality deformable attention feature fusion network modeling method according to claim 1 is characterized by: Use Pytorch deep learning framework to perform modeling methods.

3. A near-infrared EEG motion imagery recognition method based on modal deformable attention feature fusion, characterized in that include: Step S1: preprocessing the pre-acquired public motor imagery dataset MI, and establishing a training set and a test set corresponding to the dataset; Step S2: executing the EEG near-infrared modality deformable attention feature fusion network modeling method according to claim 1 or 2; Step S3: input the training sets preprocessed in step S1 into the constructed EEG near-infrared modality deformable attention feature fusion network respectively to perform model training of the EEG near-infrared modality deformable attention feature fusion network.

4. The motor imagery EEG decoding method as claimed in claim 3, characterized in that Further including: Step S4: Input the preprocessed test sets in step S1 into the trained EEG near-infrared modality deformable attention feature fusion network to obtain the motor imagery EEG signal decoding performance index. in: The step S1 includes bandpass filtering, baseline correction, re-referencing and bad conductor elimination; The original motor imagery signal data of a single subject in the public dataset is recorded as D = {(x i ,y i )|i=1,2,…,M}, where: M represents the total number of original EEG signal segments of a single subject. x i ∈R C×T Represents the i-th EEG signal segment, which contains C signal acquisition channels, each channel contains T sampling points, y i Represents the label corresponding to the i-th signal segment, where, for the MI dataset (two-classification), y i =0 represents the left hand motor imagery label, y i =1 represents the right hand motor imagery label.

5. The motor imagery decoding method according to claim 3 or 4, characterized in that: The step S3 comprises: The training sets preprocessed in step S1 are respectively input into the EEG near-infrared modality deformable attention feature fusion network to perform a model training of the EEG near-infrared modality deformable attention feature fusion network; Then, the test set preprocessed in step S1 is input into the trained EEG near-infrared modality deformable attention feature fusion network for performance testing.

6. The motor imagery decoding method according to claim 5, characterized in that: When training the model of the EEG near-infrared modality deformable attention feature fusion network, an early stopping strategy is adopted to re-divide the training set divided in step S1 into a training set and a validation set. When the recognition accuracy of the EEG near-infrared modality deformable attention feature fusion network on the validation set remains stable after N consecutive training iterations, the training process is stopped in advance, and the model parameters of the EEG near-infrared modality deformable attention feature fusion network at this time are saved; Then, a secondary training strategy is adopted. Based on the model parameters when the first training stops, the second training is started. In the second training, the training set divided in step S1 is used as the model training set, and the above-mentioned early stopping strategy is used. During the model training process of the EEG near-infrared modality deformable attention feature fusion network, the cross entropy loss function is used to calculate the classification loss J, which is calculated as follows: where p i is the i-th conditional probability generated by the model of the EEG near-infrared modality deformable attention feature fusion network, l i is the i-th category of the label set, ω(·) represents the indicator function, Θ represents the learnable parameter in the EEG near-infrared modality deformable attention feature fusion network, ||·|| is the regularization term used to alleviate the overfitting problem, λ represents the trade-off regularization weight, B represents the batch size, and the learning rate during training is 1×10 -3 .

7. The motor imagery EEG decoding method according to claim 3 or 4, characterized in that: The step S4 comprises: The motor imagery task dataset is used as a classification task with the same number of samples in each category. The recognition accuracy Acc is used to describe the classification performance of the EEG near-infrared modality deformable attention feature fusion network. The calculation formula is as follows: Where N true Represents the number of correct classifications, N total Indicates the total number of samples; The dataset used is used as a motor imagery classification task, and the bias of the EEG near-infrared modality deformable attention feature fusion network is used as an evaluation indicator. The Kappa coefficient κ is used to measure whether the classification of the EEG near-infrared modality deformable attention feature fusion network has a classification bias towards a certain category. The Kappa coefficient is calculated based on the confusion matrix and takes a value between -1 and 1. The calculation formula of the Kappa coefficient is as follows: In the formula, P0 is the classification accuracy, P e is a penalty term for biased classification, and its calculation formula is as follows: In the formula, c represents the total number of categories, a i and b i They represent the sum of the i-th row and the i-th column in the confusion matrix respectively. Therefore, according to the calculation formula of Kappa, when the classification of the EEG near-infrared modality deformable attention feature fusion network shows bias towards a certain class, that is, the confusion matrix is ​​unbalanced, the value of Kappa will decrease.

8. The motor imagery EEG decoding method according to claim 7, characterized in that: The step S4 comprises: The motor imagery decoding performance indicators were calculated with a sampling frequency of 250 Hz. The EEG signal of each trial contained 1000 sampling points. The divided data were input into the trained EEG near-infrared modality deformable attention feature fusion network. The recognition accuracy ACC, Kappa value, F1 score and AUC value were used to evaluate the motor imagery decoding effect of the EEG near-infrared modality deformable attention feature fusion network.

9. The motor imagery EEG decoding method according to claim 3 or 4, characterized in that: The step S1 comprises: S1.1: bandpass filtering, including: selecting MI motor imagery EEG signals as data input, wherein a bandpass filter with a bandwidth of 4-38 Hz is selected to adapt to the situation that the motor imagery information in the EEG signals exists in the frequency band of 4-32 Hz, and performing bandpass filtering on the intercepted data; S1.2: Baseline calibration, including making each data segment at the same baseline starting point to avoid baseline drift during long-term signal acquisition, and performing baseline correction on the signal, taking the mean of the 200ms data before the task event as the baseline, and taking the relative value relative to the baseline after the task event occurs as the new signal data value; S1.3: Perform re-referencing, including selecting the central electrode Cz on the top of the head as the reference electrode to observe the changes in the whole-brain electrode potential when the task occurs; S1.4: Eliminate bad conductors, including eliminating bad segments of signals exceeding ±100uV.

10. A computer-readable storage medium storing a computer-executable program, wherein the computer-executable program enables a processor to execute the method according to any one of claims 1 to 9.

Images