Mirror image center loss-based motor imagery electroencephalogram signal recognition algorithm of dual-scale dual soft maximum converter

Abstract

The invention discloses a motor imagery electroencephalogram signal recognition algorithm of a dual-scale dual soft maximum converter based on mirror image center loss. The motor imagery electroencephalogram signal recognition algorithm specifically comprises the following steps: (1) a pre-training stage specifically comprises the following steps: step 1, selecting a data set; 2, data preprocessing; step 3, constructing a DDT model; 4, calculating a loss function; 5, carrying out back propagation and parameter updating; (2) the fine tuning stage specifically comprises the following steps of: 1, preprocessing data; 2, creating a mirror image electroencephalogram signal and generating original and mirror image features; 3, calculating a mirror image center loss function; 4, calculating a cross entropy loss function; 5, calculating a total loss function; and 6, carrying out back propagation and parameter updating. According to the method, the problem of individual difference in multi-source subject transfer learning is effectively solved, and the spatial positioning capability of the model for event-related desynchronization is enhanced.

Description

Technical Field

[0001] The invention belongs to the technical field of motor imagery EEG signal recognition, and relates to a motor imagery EEG signal recognition algorithm based on a dual-scale dual soft maximum transformer (Transformer) with mirror center loss. Background Art

[0002] A brain-computer interface is an artificial system that interacts with external devices by directly interpreting brain activity, allowing humans to control devices without relying on the traditional neuromuscular system. Motor imagery is a classic brain-computer interface paradigm that enables individuals to imagine movements to generate specific EEG signals. By decoding motor imagery EEG signals, the user's movement intention can be obtained. This technology has been widely used in stroke rehabilitation, wheelchair control, robotic arm control, and vehicle control.

[0003] Accurate recognition of EEG signals is crucial for the performance of brain-computer interfaces. However, the limited number of training samples of the target subjects limits the improvement of the model's discriminative ability. Learning and transferring knowledge from the EEG signals of other subjects is an effective means to improve recognition accuracy. However, there are still two major challenges in transfer learning in motor imagery EEG signal recognition: (1) How to model the significant individual differences in motor imagery EEG signals. Due to differences in individual anatomical structure, behavioral habits, and spontaneous low-frequency oscillations of the brain, the effective features of EEG signals of different subjects vary. Existing transfer learning methods in motor imagery EEG signal classification treat EEG signals of different subjects as a single source, which is used to generate a unified feature representation despite the large differences in their distribution. This is not conducive to effective transfer learning because the model cannot quickly and effectively transfer knowledge by selecting similar sub-source domains and discarding different sub-source domains during fine-tuning. At the same time, the discriminative feature scales of different subjects are different, and using a single time resolution feature as the input of the self-attention layer will produce poor results. (2) How to improve the model's ability to spatially localize neural activity in the sensorimotor cortex. Event-related desynchronization is manifested in EEG signals as a response to stimulation or tasks, resulting in a decrease in the power of specific brain waves (usually in the alpha, beta or theta frequency bands). This decrease in power indicates the desynchronization of neural activity in this frequency band. Accurately locating event-related desynchronization is crucial for motor imagery recognition. However, current deep learning models are not capable of locating event-related desynchronization phenomena. Summary of the invention

[0004] The purpose of the present invention is to provide a motor imagery EEG signal recognition algorithm (MCDDT) based on a dual-scale dual softmax transformer (softmax) with mirror center loss, which is used for the application of multi-source subject transfer learning in motor imagery recognition. This method effectively solves the problem of individual differences in multi-source subject transfer learning by preprocessing EEG signals and fine-tuning the model, and enhances the model's spatial positioning ability for event-related desynchronization.

[0005] The technical solution adopted by the present invention is a motor imagery EEG signal recognition algorithm based on a dual-scale dual soft maximum converter with mirror center loss, which includes two stages: pre-training and fine-tuning, specifically:

[0006] (I) The pre-training stage includes the following steps: Step 1: Data set selection; Step 2: Data preprocessing; Step 3: Constructing a dual-scale dual-softmax transformer (DDT) DDT model to process the EEG signals processed in step 2 of the pre-training stage; Step 4: Calculation of the loss function; Step 5: Back propagation and parameter update;

[0007] (ii) The fine-tuning stage includes the following steps: Step 1: preprocessing of data; Step 2: creating mirror EEG signals and generating original and mirror features; Step 3: calculating the mirror center loss function; Step 4: calculating the cross entropy loss function; Step 5: calculating the total loss function; Step 6: back propagation and parameter update.

[0008] The present invention is also characterized in that:

[0009] In step 1 of the pre-training phase: the selected datasets are the 4th International Brain-Computer Interface Competition dataset 2a and the OpenBMI dataset; in the pre-training phase, 80% of the dataset is used as the training set and the remaining 20% ​​is used as the validation set.

[0010] Step 2 of the pre-training phase is specifically implemented as follows: A 0-38 Hz bandpass filter is applied to the EEG signals in the data set selected in step 1 to remove noise and interference signals above 38 Hz in the EEG signals. Specifically, for the 2a data set, the EEG signal data is processed through a 0.5-100 Hz bandpass filter and a 50 Hz notch filter to eliminate common noise such as power line interference; for the OpenBMI data set, the collected 64-channel EEG signals are resampled to 250 Hz, and a 0-38 Hz bandpass filter is used to further remove high-frequency noise;

[0011] In addition, according to the characteristics of the dataset, EEG signal segments of different time windows were captured for analysis. Specifically, for the 2a dataset, a 4.5-second EEG signal segment from 0.5 seconds before the motor imagery prompt to 4 seconds after the prompt was used; for the OpenBMI dataset, a 3.5-second EEG signal segment from the start of the motor imagery prompt to 3.5 seconds after the prompt was used, and a triple sliding window was applied for data enhancement.

[0012] In step 3 of the pre-training phase, the constructed DDT model consists of three modules: feature extraction module, dual-scale self-attention module and classification module; the feature extraction module consists of a temporal convolution layer, a spatial convolution layer and a maximum pooling layer; the dual-scale self-attention module consists of two branches, branch I consists of a multi-head self-attention module and three consecutive convolutional pooling blocks, and branch II consists of a convolutional pooling block, a multi-head self-attention module and two consecutive convolutional pooling blocks; the classification module consists of parallel dual Softmax layers, corresponding to the motor imagery prediction probability and the source subject prediction probability respectively;

[0013] In step 3 of the pre-training stage, the EEG signal processed in step 2 of the pre-training stage is processed according to the following steps: S1: The EEG signal pre-processed in step 2 of the pre-training stage first enters the feature extraction module. In this module, a 10×1 convolution C is first used in the temporal convolution layer. t Extract the temporal features of the incoming EEG signal and then use a 1×E spatial filter C in the spatial convolution layer t Integrate the information of all EEG signal channels; after the convolution operation, use batch normalization and maximum pooling operations in the maximum pooling layer to reduce the dimension of the features, thereby improving the training efficiency. The operation flow of the feature extraction module based on the convolutional neural network is expressed in formula (1):

[0014] F=P(BN(C s (C t (X)))) (1)

[0015] In formula (1), F represents the output of the feature extraction module based on the convolutional neural network, P represents the maximum pooling operation, BN represents the batch normalization operation, and C s represents the spatial convolutional layer, C t represents the temporal convolution layer, and X represents the preprocessed EEG signal;

[0016] S2: The EEG signal processed by the feature extraction module then enters the dual-scale self-attention module, which consists of two self-attention branches:

[0017] S2.1: Branch I: The feature extraction module outputs feature F which is directly input into the self-attention branch, first passing through the multi-head self-attention module in this branch, and then passing through the three-layer convolutional pooling block in sequence;

[0018] In the processing of branch I, first, since there is no available sequence information in the multi-head self-attention module, the features with learnable position embedding are input into the multi-head self-attention layer, which can provide the DDT model with a unique identifier for each position in the sequence, making it position-sensitive when processing time series data. Then, in order to capture the internal similarity of the input features, three different weight matrices are multiplied by The input features are linearly transformed to obtain queries, keys, and values, which are then transformed by performing a dot product operation on the query and key matrices and using the key dimension d k The attention score is calculated by scaling, as shown in formula (3), and then the attention weight is obtained through the Softmax operation. After calculating the attention weight of each self-attention head, the outputs of all heads are connected to generate a unified output O, as shown in formula (4); then, the output O is layer normalized and processed by the position feedforward layer in the residual structure; wherein, the position feedforward layer further processes the features through a nonlinear transformation to enhance the nonlinear representation ability of the DDT model; then the output M1 is generated, as shown in formula (5); finally, M1 is further processed by three convolutional pooling blocks in turn, and the operation process of each convolutional pooling block is as follows:

[0019] The first convolutional pooling block processes: the input feature M1 passes through the first convolutional layer and pooling layer to extract and compress the features; the output is the feature processed by the first convolutional pooling block, denoted as CP(M1);

[0020] The second convolutional pooling block processes: The output CP(M1) of the previous layer is used as input and processed by the second convolutional pooling block to further extract features. The output is CP(CP(M1));

[0021] The third convolutional pooling block processes: The output CP(CP(M1)) of the second layer is used as input, and after being processed by the third convolutional pooling block, features are extracted again. The final output is B1, that is: B1 = CP(CP(CP(M1)))

[0022] Among them, the expressions of formula (2) to formula (5) are as follows:

[0023]

[0024] In formula (2), F is the output of the feature extraction module; Pos is the learnable position embedding; LN is the layer normalization layer; linear is the weighted linear transformation; represents the weight matrix corresponding to the i-th attention head query Q, represents the weight matrix corresponding to the i-th attention head key K, represents the weight matrix corresponding to the i-th attention head value V, Qi represents the query of the ith attention head, K i represents the key of the i-th attention head, V i Represents the value of the i-th attention head.

[0025]

[0026] In formula (3), O i represents the self-attention representation of the i-th attention head, Q i represents the query of the ith attention head, K i represents the key of the i-th attention head, C i represents the value of the i-th attention head, softmax represents the Softmax operation, and d k represents the dimension of key K, and T represents the transpose operation;

[0027] O = Concat(O1, O2, ..., O i , ... h )+F (4)

[0028] In formula (4), Concat means concatenating the outputs of multiple attention heads together. i represents the self-attention representation of the i-th attention head, the value range of i is 1 to h, h represents the number of attention heads, F is the output of the feature extraction module, and O represents the output of multiple attention heads.

[0029] M1=PF(LN(O))+O (5)

[0030] In formula (5), M1 is the output of the self-attention module, PF is the position feed-forward layer, LN represents layer normalization, and O represents the output of multiple attention heads concatenated.

[0031] S2.2: Branch II: The output feature F of the feature extraction module first passes through a convolutional pooling block to extract local features of the input data, and then enters the multi-head self-attention module. In this module, each feature undergoes a linear transformation of the query, key, and value, and then the similarity between the features is calculated through a dot product operation. Finally, the attention weight is obtained through the softmax function, and finally the output feature is processed through two convolutional pooling blocks. The learning process of branch II is summarized as follows:

[0032] Add the input feature F and the position encoding (Pos) to form a new feature representation F2:

[0033] F2=F+Pos (6)

[0034] F2 is input into the self-attention module for attention calculation; the operation of the multi-head self-attention module is called Attention, and the operation of the multi-head self-attention module is consistent with that in branch I. The output of the self-attention module is M2:

[0035] M2=Attention(CP(F2)) (7)

[0036] M2 is further processed through two consecutive convolutional pooling blocks to obtain the output feature B2:

[0037] B2=CP(CP(M2))(8)

[0038] The output B1 of branch I and the output B2 of branch II are finally added to form the output D of the dual-scale self-attention block:

[0039] D=B1+B2 (9)

[0040] The EEG signals used in the pre-training stage are all original EEG signals, so the output features obtained are all original features; the output feature D obtained after processing by the dual-scale self-attention module enters the pre-trained classification module, and the pre-trained classification module directly inputs the output D into the parallel dual Softmax layer. The outputs of the dual Softmax layer correspond to the prediction probability of motor imagery, as shown in formula (10), and the prediction probability of the source subject, as shown in formula (11);

[0041] The expressions of formula (10)-formula (11) are as follows:

[0042] P MI =sm(D) (10)

[0043] P sub =sm(D) (11)

[0044] In formulas (10) and (11), D represents the feature data transferred from the dual-scale self-attention module to the parallel Softmax layer, sm represents the Softmax layer, and P MI represents the predicted probability of motor imagery, P sub represents the predicted probability of the subject.

[0045] Step 4 of the pre-training phase is implemented in the following steps:

[0046] The motor imagery prediction probability P obtained in step 3 of the pre-training phase is MI and the predicted probability P of the source subject sub Input into formula (12), calculate the sum of the two cross entropy losses of motor imagery classification and subject classification to obtain the final loss function, which is defined as follows:

[0047]

[0048] In formula (12), L pre represents the loss function, represents the predicted probability of motor imagery for the ith subject, represents the predicted probability of the i-th source subject, represents the label of the motor imagery of the ith subject, represents the label of the i-th source subject, and N represents the total number of subjects;

[0049] Step 5 of the pre-training phase is implemented in the following specific steps: Use the DDT model to calculate the gradient of the total loss for each parameter. The DDT model obtains the degree of influence of each parameter on the loss function; on this basis, use the Adam optimizer in the Pytorch framework to update the model parameters according to these calculated gradients.

[0050] Step 1 of the fine-tuning stage is specifically as follows: 80% of the data of any subject sub1 in the collected data set is used as training data without data enhancement, and the remaining 20% ​​is used as a validation set. The EEG signal is then filtered to remove irrelevant frequency components and noise, and the EEG signal data is adjusted to a state suitable for analysis. This operation is consistent with the data preprocessing operation in the pre-training stage.

[0051] Step 2 of the fine-tuning phase is implemented as follows:

[0052] First, the electrode channel data of the left and right hemispheres are extracted from the original EEG signal of sub1; a "mirror" EEG signal is generated by exchanging the data of the left hemisphere electrode channel with the data of the right hemisphere electrode channel; the left and right hand labels of the motor imagery task of the corresponding EEG signal are exchanged;

[0053] Then, the original EEG signal and the mirror EEG signal of sub1 are passed into the feature extraction module and the dual-scale self-attention module of the DDT model saved in the pre-training stage. The processing of the original EEG signal and the mirror EEG signal in the feature extraction module and the dual-scale self-attention module is consistent with that in the pre-training stage. Since the original EEG signal and the mirror EEG signal are used in the fine-tuning stage, the outputs after processing by the dual-scale self-attention module are the original features and the mirror features, respectively.

[0054] Step 3 of the fine-tuning stage is specifically implemented according to the following steps: In this stage, the DDT model evolves into the MCDDT model after the mirror center loss is introduced. The MCDDT model consists of a feature extraction module, a dual-scale self-attention module, a pre-trained classification module and a fine-tuning classification module, wherein the fine-tuning classification module consists of a softmax layer, a classification loss and a mirror center loss; In step 2 of the fine-tuning stage, the original features and mirror features generated after being processed by the dual-scale self-attention module enter the fine-tuning classification module, and the original features and mirror features are brought into formula (13). With the help of the mirror EEG signal, the mirror center loss reduces the distance between the EEG signal sample features of the event-related desynchronization on the ipsilateral side of the brain, and increases the distance between the EEG signal sample features of the event-related desynchronization on the contralateral side of the brain:

[0055]

[0056] In formula (13), L M is the mirror center loss function, D i is the feature of the i-th EEG signal sample; C i is the feature center of the original EEG signal of category i; is the feature center of the category different from category i, and n is the number of samples in the current batch.

[0057] Step 4 of the fine-tuning stage is specifically implemented as follows: the original features generated after the dual-scale self-attention module in the fine-tuning stage are processed by the motor imagery Softmax, the original features are converted into motor imagery category prediction probabilities, and the cross entropy loss is calculated (14):

[0058]

[0059] In formula (14), L C represents the classification loss, N is the number of samples in the batch, is the true motor imagery category label of the i-th sample, is the model’s predicted probability of the motor imagery category for the i-th sample.

[0060] Step 5 of the fine-tuning stage is specifically implemented as follows: The mirror loss function L obtained in step 3 of the fine-tuning stage is M And the cross entropy loss function L obtained in step 4 of the fine-tuning phase C The total loss function is added together to guide the optimization of the MDCCT model. The total loss function formula is defined as formula (15):

[0061] L F =L C +ω·L M (15)

[0062] In formula (15), L F is the total loss function, L C is the cross entropy loss, L M is the mirror center loss, ω is the weight coefficient of the mirror center loss;

[0063] Step 6 of the fine-tuning phase is implemented as follows: Use the MCDDT model to calculate the gradient of the total loss for each parameter. It tracks the error transfer process from the output layer to the input layer. The result of back propagation is that the MCDDT model obtains the influence of each parameter on the loss function. On this basis, use the Adam optimizer in the Pytorch framework to update the model parameters according to these calculated gradients.

[0064] The beneficial effects of the present invention are as follows: the dual-scale dual soft-max converter model based on mirror center loss proposed by the method of the present invention can solve the temporal individual differences in motor imagery EEG signals, enhance the model's ability to capture subject-specific temporal features in EEG signals, and significantly improve the accuracy of EEG signal classification. A dual softmax structure is proposed. In the pre-training stage, the dual softmax structure simultaneously predicts the subject label and the motor imagery label, which enables the model to obtain subject-specific distinguishing features, thereby enhancing its adaptability in the subsequent fine-tuning stage. At the same time, by introducing the mirror center loss, by minimizing the distance between the original EEG signal features and the similar mirror EEG signal features, the model's spatial positioning of event-related desynchronization is enhanced, so that the model can learn distinguishing features that are beneficial to the motor imagery recognition task. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 It is a schematic diagram of the model framework of the pre-training stage in the motor imagery EEG signal recognition algorithm of the dual-scale dual soft maximum converter based on the mirror center loss of the present invention;

[0066] Figure 2 It is a schematic diagram of the architecture of the self-attention layer in the motor imagery EEG signal recognition algorithm of the dual-scale dual soft maximum converter based on the mirror center loss of the present invention;

[0067] Figure 3 It is a schematic diagram of the model framework of the fine-tuning stage in the motor imagery EEG signal recognition algorithm based on the dual-scale dual soft maximum converter of the mirror center loss of the present invention;

[0068] Figure 4 Schematic diagram of swapping left and right EEG channels to construct a mirrored EEG signal. DETAILED DESCRIPTION

[0069] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0070] The present invention provides a motor imagery EEG signal recognition algorithm (MCDDT) based on a dual-scale dual soft maximum transformer (Transformer) with mirror center loss. The algorithm is divided into two stages: pre-training and fine-tuning. The framework diagram of the pre-training stage is shown in FIG. Figure 1 As shown in the figure, the pre-training stage refers to the process of training the MCDDT model using data from multiple subjects. Through pre-training, the MCDDT model can learn common features and patterns. The purpose of this stage is to improve the performance of the model and lay the foundation for subsequent fine-tuning on specific tasks. The fine-tuning stage refers to the process of further optimizing the pre-trained MCDDT model using data from a single target subject. Fine-tuning can improve the performance and accuracy of specific target subject tasks.

[0071] The implementation of the method of the present invention specifically comprises the following steps:

[0072] 1. Pre-training stage

[0073] Pre-training stage step 1: selection of data set;

[0074] In the pre-training phase, step 1: the selected datasets are the 4th International Brain-Computer Interface Competition dataset 2a and the OpenBMI dataset. The 2a dataset provides an EEG signal recognition task based on motor imagery (MI), which contains 22-channel EEG signals of 9 subjects. Each subject performed 72 trials of each category in the training set and the test set, and the data were recorded at different time points. The subjects performed motor imagery according to the prompts, including movements of the left hand, right hand, tongue, and foot. In this study, we focused on two types of motor imagery recognition tasks, namely distinguishing left-hand and right-hand motor imagery. The sampling rate of the EEG signal is 250Hz, and the amplifier resolution is 100μV; the OpenBMI dataset was collected by a Korean university and contains EEG signals of 54 subjects. Each subject collected EEG signals twice on different dates, and each collection contained a training set and a test set, each containing 100 trials (50 left-hand motor imagery tasks and 50 right-hand motor imagery tasks). The EEG signal was recorded through 64 channels with a sampling rate of 1000Hz (resampled to 250Hz in this study). Each trial started with a 3-second black fixation cross, followed by a 4-second imagery task in which the subject imagined the corresponding hand grasping when seeing a left or right arrow. In our experiment, 33 subjects who did not respond to motor imagery were excluded, and 21 subjects were focused on (subject numbers: 1, 2, 3, 5, 6, 9, 17, 18, 19, 21, 22, 28, 29, 32, 33, 36, 37, 43, 44, 45, and 52).

[0075] During the pre-training stage, 80% of the dataset was used as a training set, which contained EEG signals from a variety of different subjects to ensure that the MCDDT model could learn effective features related to motor imagery recognition from diverse source data. The remaining 20% ​​was used as a validation set to evaluate the performance of the MCDDT model during the pre-training process.

[0076] Pre-training stage step 2: data preprocessing: filter the EEG signals in the data set selected in step 1, remove irrelevant frequency components and noise, and adjust the EEG signal data to a state suitable for analysis;

[0077] Step 2 of the pre-training phase is specifically implemented according to the following steps: A 0-38 Hz bandpass filter is applied to the EEG signals in the data set selected in step 1 to remove noise and interference signals above 38 Hz in the EEG signals. Specifically, for the 2a data set, the EEG signal data is processed by a 0.5-100 Hz bandpass filter and a 50 Hz notch filter to eliminate common noise such as power line interference; for the OpenBMI data set, the collected 64-channel EEG signals are resampled to 250 Hz, and a 0-38 Hz bandpass filter is used to further remove high-frequency noise; in addition, according to the characteristics of the data set, EEG signal segments of different time windows are intercepted for analysis, specifically: for the 2a data set, a 4.5-second EEG signal segment from 0.5 seconds before the motor imagery prompt to 4 seconds after the prompt is used; for the OpenBMI data set, a 3.5-second EEG signal segment from the start of the motor imagery prompt to 3.5 seconds after the prompt is used, and a triple sliding window is applied for data enhancement.

[0078] Pre-training stage step 3: Construct a dual-scale dual-softmax transformer (DDT) DDT model, such as Figure 1 As shown, the EEG signal processed in step 2 of the pre-training stage is processed. The constructed DDT model consists of three modules: feature extraction module, dual-scale self-attention module and classification module.

[0079] The preprocessed EEG signal first enters the feature extraction module, and uses a convolutional neural network to extract the spatiotemporal features of the EEG signal; then enters the dual-scale self-attention module. To extract features with different temporal resolutions, this module is mainly divided into two branches: branch I is used to capture the dependencies of high temporal resolution features, and branch II is used to capture the dependencies of low temporal resolution features; finally enters the classification module, in which the EEG signal processed by the dual-scale self-attention module is input into the dual Softmax layer. This structure can simultaneously predict the motor imagery (MI) label and the source subject (subject) label;

[0080] The feature extraction module consists of a temporal convolution layer, a spatial convolution layer, and a maximum pooling layer; the dual-scale self-attention module consists of two branches, branch I consists of a multi-head self-attention module and three consecutive convolutional pooling blocks, and branch II consists of a convolutional pooling block, a multi-head self-attention module, and two consecutive convolutional pooling blocks; the classification module consists of parallel dual Softmax layers, corresponding to the motor imagery prediction probability and the source subject prediction probability respectively;

[0081] In step 3 of the pre-training stage, the EEG signal processed in step 2 of the pre-training stage is processed according to the following steps: S1: The EEG signal pre-processed in step 2 of the pre-training stage first enters the feature extraction module. In this module, a 10×1 convolution C is first used in the temporal convolution layer. t Extract the temporal features of the incoming EEG signal and then use a 1×E spatial filter C in the spatial convolution layer t (E is the number of EEG signal channels) integrates the information of all EEG signal channels; after the convolution operation, batch normalization (BN) and maximum pooling (P) operations are used in the maximum pooling layer to reduce the dimension of the features, thereby improving the training efficiency. The operation flow of the feature extraction module based on the convolutional neural network is expressed in formula (1):

[0082] F=P(BN(C s (C t (X)))) (1)

[0083] In formula (1), F represents the output of the feature extraction module based on the convolutional neural network, P represents the maximum pooling operation, BN represents the batch normalization operation, and C s represents the spatial convolutional layer, C t represents the temporal convolution layer, and X represents the preprocessed EEG signal;

[0084] S2: The EEG signal processed by the feature extraction module then enters the dual-scale self-attention module, which consists of two self-attention branches:

[0085] S2.1: Branch I: Focuses on capturing the dependencies of high temporal resolution features. The feature extraction module outputs feature F directly into the self-attention branch, first passing through the multi-head self-attention module in this branch, and then passing through the three-layer convolutional pooling block (CP block) in sequence;

[0086] In the processing of branch I, firstly, due to the multi-head self-attention module (such as Figure 2As shown in the figure, there is no available sequence information in the dataset. The features with learnable position embedding are input into the multi-head self-attention layer, which can provide the DDT model with a unique identifier for each position in the sequence, making it position-sensitive when processing time series data. Then, in order to capture the internal similarity of the input features, three different weight matrices are multiplied by Perform a linear transformation on the input features to obtain the query (Q i ), key (K i ) and value (V i ), as shown in formula (2); then the query and key matrices are dot-producted and the key dimension d is used k The attention score is calculated by scaling, as shown in formula (3), and then the attention weight is obtained through the Softmax operation. After calculating the attention weight of each self-attention head, the outputs of all heads are connected to generate a unified output O, as shown in formula (4); then, the output O is layer normalized (LN) and processed by the position feedforward layer (PF) in the residual structure. Among them, the position feedforward layer further processes the features through a nonlinear transformation (GELU activation function) to enhance the nonlinear representation ability of the DDT model; then the output M1 is generated, as shown in formula (5); finally, M1 is further processed by three convolutional pooling blocks (CPBlock) in turn, and the operation process of each convolutional pooling block is as follows:

[0087] The first convolutional pooling block processes: The input feature M1 passes through the first convolutional layer and pooling layer to extract and compress the features. The output is the feature processed by the first convolutional pooling block, denoted as CP(M1);

[0088] The second convolutional pooling block processes: The output CP(M1) of the previous layer is used as input and processed by the second convolutional pooling block to further extract features. The output is CP(CP(M1));

[0089] The third convolutional pooling block processes: The output CP(CP(M1)) of the second layer is used as input, and after being processed by the third convolutional pooling block, features are extracted again. The final output is B1, that is: B1 = CP(CP(CP(M1)))

[0090] Among them, the expressions of formula (2) to formula (5) are as follows:

[0091]

[0092] In formula (2), F is the output of the feature extraction module; Pos is the learnable position embedding; LN is the layer normalization layer; linear is the weighted linear transformation; represents the weight matrix corresponding to the i-th attention head query Q, represents the weight matrix corresponding to the i-th attention head key K, represents the weight matrix corresponding to the i-th attention head value V, Q i represents the query of the ith attention head, K i represents the key of the i-th attention head, V i Represents the value of the i-th attention head.

[0093]

[0094] In formula (3), O i represents the self-attention representation of the i-th attention head, Q i represents the query of the ith attention head, K i represents the key of the i-th attention head, V i represents the value of the i-th attention head, softmax represents the Softmax operation, and d k represents the dimension of key K, and T represents the transpose operation;

[0095] O = Concat(O1, O2, ..., O i , ... h )+F (4)

[0096] In formula (4), Concat means concatenating the outputs of multiple attention heads together. i represents the self-attention representation of the i-th attention head, the value range of i is 1 to h, h represents the number of attention heads, F is the output of the feature extraction module, and O represents the output of multiple attention heads.

[0097] M1=PF(LN(O))+O (5)

[0098] In formula (5), M1 is the output of the self-attention module, PF is the position feed-forward layer, LN represents layer normalization, and O represents the output of multiple attention heads concatenated.

[0099] S2.2: Branch II: Focuses on capturing the dependencies of low temporal resolution features. The output feature F of the feature extraction module first passes through a convolutional pooling block to extract local features of the input data, and then enters the multi-head self-attention module. In this module, each feature undergoes a linear transformation of the query, key, and value, and then the similarity between features is calculated through a dot product operation. Finally, the attention weight is obtained through the softmax function, and finally the output feature is processed through two convolutional pooling blocks. The learning process of Branch II is summarized as follows:

[0100] Add the input feature F and the position encoding (Pos) to form a new feature representation F2:

[0101] F2=F+Pos (6)

[0102] F2 is input into the self-attention module for attention calculation; the operation of the multi-head self-attention module is called Attention, and the operation of the multi-head self-attention module is consistent with that in branch I. The output of the self-attention module is M2:

[0103] M2=Attention(CP(F2)) (7)

[0104] M2 is further processed through two consecutive convolutional pooling blocks to obtain the output feature B2:

[0105] B2=CP(CP(M2))(8)

[0106] The output B1 of branch I and the output B2 of branch II are finally added to form the output D of the dual-scale self-attention block:

[0107] D=B1+B2 (9)

[0108] The EEG signals used in the pre-training stage are all original EEG signals, so the output features obtained are all original features. The output feature D obtained after processing by the dual-scale self-attention module enters the pre-trained classification module, and the pre-trained classification module directly inputs the output D into the parallel dual Softmax layer. The output of the dual Softmax layer corresponds to the motor imagery prediction probability, as shown in formula (10), and the prediction probability of the source subject, as shown in formula (11); the main function of the pre-trained classification module is to simultaneously predict the motor imagery label and the subject label, so that the model can obtain the discriminant features of specific subjects during training.

[0109] The expressions of formula (10)-formula (11) are as follows:

[0110] P MI =sm(D) (10)

[0111] P sub =sm(D) (11)

[0112] In formulas (10) and (11), D represents the feature data transferred from the dual-scale self-attention module to the parallel Softmax layer, sm represents the Softmax layer, and P MI represents the predicted probability of motor imagery, P sub represents the predicted probability of the subject.

[0113] Pre-training phase Step 4: Calculation of loss function: The loss function is the sum of two cross entropy losses for motor imagery classification and subject classification.

[0114] Step 4 of the pre-training stage is specifically implemented as follows: the motor imagery prediction probability P obtained in step 3 of the pre-training stage is converted into MI and the predicted probability P of the source subject sub Input into formula (12), calculate the sum of the two cross entropy losses of motor imagery classification and subject classification to obtain the final loss function; this loss function aims to simultaneously optimize the model's classification ability for motor imagery tasks and different subject data, thereby laying the foundation for subsequent fine-tuning.

[0115] The loss function is defined as follows:

[0116]

[0117] In formula (12), L pre represents the loss function, represents the predicted probability of motor imagery for the ith subject, represents the predicted probability of the i-th source subject, represents the label of the motor imagery of the ith subject, represents the label of the i-th source subject, and N represents the total number of subjects.

[0118] Pre-training stage step 5: back propagation and parameter update; steps 1-4 are called forward propagation. During the forward propagation process, the DDT model receives the input EEG signal and gradually extracts features and generates prediction results through a series of module calculations (feature extraction module, dual-scale self-attention module, and pre-training classification module). Subsequently, the model calculates the error between the predicted value and the true value to obtain the total loss (i.e., L pre ).

[0119] Next, we enter the back-propagation phase. The DDT model is used to calculate the gradient of the total loss for each parameter to form back-propagation. This step is based on the chain rule, which tracks the error transmission process from the output layer to the input layer. The result of back-propagation is that the DDT model obtains the degree of influence of each parameter relative to the loss function. On this basis, the Adam optimizer in the Pytorch framework is used to update the parameters of the model according to these calculated gradients. The Adam optimizer can adaptively adjust the learning rate and combine the first-order and second-order momentum estimates of the gradient, so as to accelerate convergence when updating parameters and maintain the stability of the update process. Finally, by repeating the cycle of forward propagation, loss calculation, back-propagation, and parameter update, the model is gradually optimized. At this stage, the DDT model learns the common patterns and features in multi-source EEG signals.

[0120] After the pre-training phase, the model is able to predict both motor imagery labels and subject labels, that is, it can distinguish the characteristics of different motor imagery and different people in the EEG signals of a large number of different people. In this way, the model can obtain individualized distinguishing features for each subject and better adapt to the data of the target subject in the subsequent fine-tuning phase.

[0121] The design of the fine-tuning stage aims to personalize and optimize the pre-trained model through a small amount of target subject data. The framework diagram of the fine-tuning stage is shown in the figure below. Figure 3 As shown, in order to improve the recognition accuracy of the motor imagery task. The following will take any subject sub1 in the 2a dataset or the OpenBMI dataset as an example to explain the specific steps of fine-tuning in detail:

[0122] Fine-tuning stage Step 1: Data preprocessing;

[0123] In the fine-tuning stage, 80% of the data of any subject sub1 (hereinafter referred to as sub1) in the collected 2a dataset or OpenBMI dataset was used as training data without data enhancement, and the remaining 20% ​​was used as a validation set. After that, the EEG signal was filtered to remove irrelevant frequency components and noise, and adjusted to a state suitable for analysis. This operation is consistent with the data preprocessing operation in the pre-training stage;

[0124] Fine-tuning stage step 2: Create mirror EEG signals and generate original and mirror features;

[0125] Step 2 of the fine-tuning phase is implemented as follows: In order to further improve the model's recognition ability for motor imagery tasks and enhance the model's learning of EEG signal features in the absence of precise annotations, a mirror EEG signal generation method based on channel exchange is used. The specific process is as follows:

[0126] First, the electrode channel data of the left and right hemispheres are extracted from the original EEG signal of sub1. For example, the electrode channels of the left hemisphere include C3, F3, P3, etc., while the electrode channels of the right hemisphere include C4, F4, P4, etc. Figure 4As shown. Then, a "mirror" EEG signal is generated by exchanging the data of the left hemisphere electrode channel with the data of the right hemisphere electrode channel. This exchange operation simulates the inversion of the original EEG signal between the left and right hemispheres, so that the electrode placement position of the mirror EEG signal is similar to the mirror image of the original signal. After the mirror EEG signal is generated, due to the symmetrical inversion of the left and right hemispheres of the EEG signal, the left and right hand labels of the corresponding EEG signal's motor imagery task also need to be swapped. For example, assuming that the motor imagery task label of the original EEG signal is "left hand", this means that in the original signal, the event-related desynchronization phenomenon of the left motor cortex is more obvious. However, in the generated mirror EEG signal, due to the exchange of the electrode positions of the left and right hemispheres, the corresponding event-related desynchronization phenomenon will appear in the right motor cortex. Therefore, in order to ensure that the label is consistent with the actual motor task of the mirror signal, the motor imagery task label of the mirror EEG signal will be relabeled as "right hand". Then, the original EEG signal and the mirror EEG signal of sub1 are passed into the feature extraction module and the dual-scale self-attention module of the DDT model saved in the pre-training stage. The processing of the original EEG signal and the mirror EEG signal in the feature extraction module and the dual-scale self-attention module is consistent with that in the pre-training stage, that is, step 3 of the pre-training stage. Since the original EEG signal and the mirror EEG signal are used in the fine-tuning stage, the outputs after processing by the dual-scale self-attention module are the original features and the mirror features, respectively.

[0127] Step 3 of the fine-tuning stage: Calculate the mirror center loss function. Specifically, the following steps are implemented: In this stage, the DDT model evolves into the MCDDT model after the mirror center loss is introduced. The MCDDT model consists of a feature extraction module, a dual-scale self-attention module, a pre-trained classification module, and a fine-tuning classification module. Among them, the fine-tuning classification module consists of a softmax layer, a classification loss, and a mirror center loss. The MCDDT model uses the mirror center loss to enhance the spatial positioning ability of event-related desynchronization. The mirror center loss proposed in the present invention is an improved version of the center loss. The original features and mirror features generated by the dual-scale self-attention module in step 2 of the fine-tuning stage enter the fine-tuning classification module, and the original features and mirror features are brought into the formula (13). With the help of the mirror EEG signal, the mirror center loss reduces the distance between the EEG signal sample features of the event-related desynchronization on the ipsilateral side of the brain (the first term of formula (13)), and increases the distance between the EEG signal sample features of the event-related desynchronization on the contralateral side of the brain (the second term of formula (13)):

[0128]

[0129] In formula (13), L M is the mirror center loss function, D iis the feature of the i-th EEG signal sample; C i is the feature center of the original EEG signal of category i; is the feature center of the category different from category i, and n is the number of samples in the current batch.

[0130] Fine-tuning phase Step 4: Calculate the cross entropy loss function: In order to achieve supervised classification training for motor imagery (motor imagery), the cross entropy loss of Softmax is used.

[0131] Step 4 of the fine-tuning stage is implemented in the following steps: The cross-entropy loss is used in the fine-tuning stage to ensure that the model can perform effective classification learning on the target subject's data and directly optimize the performance of the motor imagery classification task. The original features generated by the dual-scale self-attention module in the fine-tuning stage are processed by the running image Softmax, and the original features are converted into motor imagery category prediction probabilities, and the cross-entropy loss is calculated (14):

[0132]

[0133] In formula (14), L C represents the classification loss (cross entropy loss), N is the number of samples in the batch, is the true motor imagery category label of the i-th sample, is the model’s predicted probability of the motor imagery category for the i-th sample.

[0134] Step 5 of the fine-tuning phase: Calculation of the total loss function; the loss function at this stage is expressed as the sum of the image center loss and the classification loss. The image loss function L obtained in step 3 of the fine-tuning phase is M And the cross entropy loss function L obtained in step 4 of the fine-tuning phase C The total loss function is added together to guide the optimization of the MDCCT model. The total loss function formula is defined as formula (15):

[0135] L F =L C +ω·L M (15)

[0136] In formula (15), L F is the total loss function, L C is the cross entropy loss, L M is the mirror center loss, and ω is the weight coefficient of the mirror center loss.

[0137] Fine-tuning stage step 6: back propagation and parameter update; steps 1-5 are called forward propagation. During the forward propagation process, the MCDDT model receives the input EEG signal and gradually extracts features and generates prediction results through a series of module calculations (feature extraction module, dual-scale self-attention module, and fine-tuning classification module). Subsequently, the model calculates the error between the predicted value and the true value to obtain the total loss (i.e., L F ). Next, it enters the back-propagation phase. Through back-propagation, the MCDDT model calculates the gradient of the total loss for each parameter. This step is based on the chain rule, which tracks the error propagation process from the output layer to the input layer. The result of back-propagation is that the MCDDT model obtains the degree of influence of each parameter relative to the loss function. On this basis, the Adam optimizer in the Pytorch framework is used to update the parameters of the model according to these calculated gradients. The Adam optimizer can adaptively adjust the learning rate, combined with the first-order and second-order momentum estimates of the gradient, so as to accelerate convergence when updating parameters and maintain the stability of the update process. Finally, by repeating the cycle of forward propagation, loss calculation, back-propagation, and parameter update, the parameter adjustment in the fine-tuning phase focuses on adapting the MCDDT model to sub1's personalized EEG signal distribution and improving the prediction performance.

[0138] Example 1

[0139] Experimental verification: 1. Experimental parameter setting. In the preprocessing stage, the maximum number of training epochs was set to 500, and the Adam optimizer was used as the optimization method. The EEG signal was filtered using a 0-38Hz bandpass filter. For the 2a dataset, a 4.5-second EEG signal segment from 0.5 seconds before the motor imagery prompt to 4 seconds after the prompt was used, containing 1125 sampling points. For the B motor imagery dataset, a 3.5-second EEG signal segment from the start of the motor imagery prompt to 3.5 seconds after was used, containing 875 sampling points. 80% of the training data of all subjects in the corresponding dataset was used as the training set, and the remaining 20% ​​was used as the validation set, where the training set was enhanced with a three-fold sliding window. In the fine-tuning stage, the maximum number of training epochs was set to 500, and the Adam optimizer was used as the optimization method. The EEG signal was filtered using a 0-38Hz bandpass filter. For the 2a dataset, a 4.5-second EEG signal segment from 0.5 seconds before the motor imagery prompt to 4 seconds after the prompt was used, containing 1125 sampling points. For the B motor imagery dataset, a 3.5-second EEG signal segment from the start of the motor imagery prompt to 3.5 seconds later was used, containing 875 sampling points. 80% of the target subject's training data was used without data augmentation, and the remaining 20% ​​was used as a validation set.

[0140] 2. Experimental results: In order to verify the effectiveness of the motor imagery EEG signal recognition algorithm based on the mirror center loss dual-scale dual softmax Transformer model, multiple groups of experiments were conducted. The experimental results show that the method of the present invention performs well in the motor imagery EEG signal recognition task independent of the subjects.

[0141] This study compares the proposed MCDDT model with six existing state-of-the-art (SOTA) methods on two datasets, 2a and B motor imagery, including Shallow ConvNet, Deep ConvNet, EEG Signal Net, MM Convolutional Neural Network, ADF Convolutional Neural Network, and ATCNet. The comparison results are based on the epoch that minimizes the validation loss on the validation set, and the results are expressed in terms of accuracy and Kappa value.

[0142] Table 1 Comparison of performance (accuracy and Kappa coefficient) of existing methods

[0143]

[0144]

[0145] The experimental results in Table 1 show that the accuracy of MCDDT reaches 89.64% and 86.23%, respectively, which are better than other existing state-of-the-art methods.

[0146] We also conduct extensive ablation experiments to verify the effectiveness of the proposed dual-scale transformer, dual-softmax structure, and mirror center loss. The ablation experiments evaluate the performance of single-branch networks ("Branch I" and "Branch II"), dual-branch networks ("Dual-Branch") in direct training settings, and dual-branch networks with dual Softmax ("DDT") and the proposed "MCDDT" in pre-training and fine-tuning settings. The experimental results are shown in Table 2.

[0147] Table 2 Ablation experiment of MCDDT

[0148]

[0149]

[0150] From Table 2, we can draw the following conclusions: (1) The dual-branch model outperforms the single-branch structure model, which shows that the dual-scale transformer model can effectively integrate features of multiple temporal resolutions. (2) The DDT model performs better than the dual-branch model, which shows that the dual softmax structure can enhance the adaptability of the model during transfer learning. (3) The MCDDT model combined with the mirror center loss performs best because it improves the spatial localization ability of the event-related desynchronization phenomenon.

[0151] Based on the above conclusions, it can be seen that the dual-scale dual-softmax Transformer model based on the mirror center loss solves the problem of individual time differences in motor imagery EEG signals. Secondly, the dual Softmax structure enables the model to obtain the discriminative features of specific subjects, rather than the unified feature distribution of all subjects. This method allows features to be transferred directly from similar source domains, while discarding different domains during fine-tuning, rather than transferring from the unified feature distribution of all subjects. Therefore, it is more effective in knowledge transfer. Third, the mirror center loss enhances the model's spatial localization ability of event-related desynchronization phenomena in a semi-supervised manner by minimizing the distance between the original EEG signal and the mirror EEG signal features of the same category.

[0152] Example 2

[0153] The motor imagery EEG signal recognition algorithm based on the dual-scale dual soft maximum transformer with mirror center loss includes two stages: pre-training and fine-tuning, specifically:

[0154] (I) The pre-training stage includes the following steps: Step 1: Data set selection; Step 2: Data pre-processing; Step 3: Constructing the MCDDT model and processing the EEG signals processed in step 2 of the pre-training stage; Step 4: Calculation of the loss function; Step 5: Back propagation and parameter update;

[0155] (ii) The fine-tuning stage includes the following steps: Step 1: preprocessing of data; Step 2: creating mirror EEG signals and generating original and mirror features; Step 3: calculating the mirror center loss function; Step 4: calculating the cross entropy loss function; Step 5: calculating the total loss function; Step 6: back propagation and parameter update.

[0156] Finally: Experimental evaluation and performance analysis: Experimental evaluation is conducted on two public datasets. The model is compared with other state-of-the-art methods (SOTA) in terms of classification accuracy and Kappa value. The effectiveness of each module in the model is verified through ablation experiments, and the contribution of these modules to the model performance is analyzed.

[0157] Example 3

[0158] The motor imagery EEG signal recognition algorithm based on the dual-scale dual soft maximum transformer with mirror center loss includes two stages: pre-training and fine-tuning, specifically:

[0159] (I) The pre-training stage includes the following steps: Step 1: Dataset selection; In the pre-training stage step 1: The selected datasets are the 4th International Brain-Computer Interface Competition dataset 2a and the OpenBMI dataset; In the pre-training stage, 80% of the dataset is used as the training set, and the remaining 20% ​​is used as the validation set. Step 2: Data preprocessing; Step 3: Construct the MCDDT model and process the EEG signals processed in step 2 of the pre-training stage; Step 4: Calculation of the loss function; Step 5: Back propagation and parameter update;

[0160] (II) The fine-tuning stage includes the following steps: Step 1: Data preprocessing; Step 2: Create mirror EEG signals and generate original and mirror features; Step 3: Calculate the mirror center loss function; Step 4: Calculate the cross entropy loss function; Step 5: Calculate the total loss function; Step 6: Back propagation and parameter update. Finally: Experimental evaluation and performance analysis: Experimental evaluation is performed on two public datasets. The model is compared with other state-of-the-art methods (SOTA) in terms of classification accuracy and Kappa value. The effectiveness of each module in the model is verified through ablation experiments, and the contribution of these modules to the model performance is analyzed.

Claims

1. A motor imagery EEG signal recognition algorithm based on a dual-scale dual soft-maximum converter with mirror center loss, characterized in that: It includes two stages: pre-training and fine-tuning, specifically: (I) The pre-training phase includes the following steps: Step 1: Selection of data set; Step 2: Data preprocessing; Step 3: Construct a dual-scale dual soft maximum transformer (DDT) model to process the EEG signals processed in step 2 of the pre-training phase; Step 4: Calculation of loss function; Step 5: Back propagation and parameter update; (II) The fine-tuning phase includes the following steps: Step 1: Data preprocessing; Step 2: Create a mirror EEG signal and generate original and mirror features; Step 3: Calculate the mirror center loss function; Step 4: Calculate the cross entropy loss function; Step 5: Calculation of total loss function; Step 6: Back propagation and parameter update.

2. The motor imagery EEG signal recognition algorithm based on the dual-scale dual soft maximum converter with mirror center loss according to claim 1 is characterized in that: In step 1 of the pre-training phase: the selected datasets are the 4th International Brain-Computer Interface Competition dataset 2a and the OpenBMI dataset; in the pre-training phase, 80% of the dataset is used as the training set and the remaining 20% ​​is used as the validation set.

3. The motor imagery EEG signal recognition algorithm based on the dual-scale dual soft maximum converter with mirror center loss according to claim 2 is characterized in that: Step 2 of the pre-training phase is implemented in the following steps: A 0-38 Hz bandpass filter was applied to the EEG signals in the selected data set in step 1 to remove noise and interference signals above 38 Hz in the EEG signals. Specifically, for the 2a data set, the EEG signal data was processed by a 0.5-100 Hz bandpass filter and a 50 Hz notch filter; for the OpenBMI data set, the collected 64-channel EEG signals were resampled to 250 Hz, and a 0-38 Hz bandpass filter was used to further remove high-frequency noise; In addition, according to the characteristics of the dataset, EEG signal segments of different time windows were captured for analysis. Specifically, for the 2a dataset, a 4.5-second EEG signal segment from 0.5 seconds before the motor imagery prompt to 4 seconds after the prompt was used; for the OpenBMI dataset, a 3.5-second EEG signal segment from the start of the motor imagery prompt to 3.5 seconds after the prompt was used, and a triple sliding window was applied for data enhancement.

4. The motor imagery EEG signal recognition algorithm based on the dual-scale dual soft maximum converter with mirror center loss according to claim 3 is characterized in that: In step 3 of the pre-training phase, the constructed DDT model consists of three modules: feature extraction module, dual-scale self-attention module, and classification module; The feature extraction module consists of a temporal convolution layer, a spatial convolution layer, and a maximum pooling layer; The dual-scale self-attention module consists of two branches. Branch I consists of a multi-head self-attention module and three consecutive convolutional pooling blocks, and branch II consists of a convolutional pooling block, a multi-head self-attention module, and two consecutive convolutional pooling blocks. The classification module consists of two parallel Softmax layers, corresponding to the motor imagery prediction probability and the source subject prediction probability respectively; In step 3 of the pre-training stage, the EEG signal processed in step 2 of the pre-training stage is processed according to the following steps: S1: Pre-training stage Step 2 The pre-processed EEG signal first enters the feature extraction module. In this module, a 10×1 convolution C is first used in the temporal convolution layer. t Extract the temporal features of the incoming EEG signal and then use a 1×E spatial filter C in the spatial convolution layer t Integrate the information of all EEG signal channels; after the convolution operation, use batch normalization and maximum pooling operations in the maximum pooling layer to reduce the dimension of the features, thereby improving the training efficiency. The operation flow of the feature extraction module based on the convolutional neural network is expressed in formula (1): F=P(BN(C s (C t (X)))) (1) In formula (1), F represents the output of the feature extraction module based on the convolutional neural network, P represents the maximum pooling operation, BN represents the batch normalization operation, and C s represents the spatial convolutional layer, C t represents the temporal convolution layer, and X represents the preprocessed EEG signal; S2: The EEG signal processed by the feature extraction module then enters the dual-scale self-attention module, which consists of two self-attention branches: S2.1: Branch I: The feature extraction module outputs feature F which is directly input into the self-attention branch, first passing through the multi-head self-attention module in this branch, and then passing through the three-layer convolutional pooling block in sequence; In the processing of branch I, first, since there is no available sequence information in the multi-head self-attention module, the features with learnable position embedding are input into the multi-head self-attention layer, which can provide the DDT model with a unique identifier for each position in the sequence, making it position-sensitive when processing time series data. Then, in order to capture the internal similarity of the input features, three different weight matrices are multiplied by The input features are linearly transformed to obtain queries, keys, and values, which are then transformed by performing a dot product operation on the query and key matrices and using the key dimension d k The attention score is calculated by scaling, as shown in formula (3), and then the attention weight is obtained through the Softmax operation. After calculating the attention weight of each self-attention head, the outputs of all heads are connected to generate a unified output O, as shown in formula (4); then, the output O is layer normalized and processed by the position feedforward layer in the residual structure; wherein, the position feedforward layer further processes the features through a nonlinear transformation to enhance the nonlinear representation ability of the DDT model; then the output M1 is generated, as shown in formula (5); finally, M1 is further processed by three convolutional pooling blocks in turn, and the operation process of each convolutional pooling block is as follows: The first convolutional pooling block processes: the input feature M1 passes through the first convolutional layer and pooling layer to extract and compress the features; the output is the feature processed by the first convolutional pooling block, denoted as CP(M1); The second convolutional pooling block processes: The output CP(M1) of the previous layer is used as input, and after being processed by the second convolutional pooling block, features are further extracted and the output is CP(CP(M1)). The third convolutional pooling block processes: The output CP(CP(M1)) of the second layer is used as input. After being processed by the third convolutional pooling block, features are extracted again and the final output is B1, that is, B1 = CP(CP(CP(M1))) Among them, the expressions of formula (2) to formula (5) are as follows: In formula (2), F is the output of the feature extraction module; Pos is the learnable position embedding; LN is the layer normalization layer; linear is the weighted linear transformation; represents the weight matrix corresponding to the i-th attention head query Q, represents the weight matrix corresponding to the i-th attention head key K, represents the weight matrix corresponding to the i-th attention head value V, Q i represents the query of the ith attention head, K i represents the key of the i-th attention head, V i Represents the value of the i-th attention head; In formula (3), O i represents the self-attention representation of the i-th attention head, Q i represents the query of the ith attention head, K i represents the key of the i-th attention head, V i represents the value of the i-th attention head, softmax represents the Softmax operation, and d k represents the dimension of key K, and T represents the transpose operation; O=Concat(O1,O2,...,O i ,...O h )+F (4) In formula (4), Concat means concatenating the outputs of multiple attention heads together. i represents the self-attention representation of the i-th attention head, the value range of i is 1 to h, h represents the number of attention heads, F is the output of the feature extraction module, and O represents the output of multiple attention heads. M1=PF(LN(O))+O (5) In formula (5), M1 is the output of the self-attention module, PF is the position feed-forward layer, LN represents layer normalization, and O represents the output of multiple attention heads concatenated. S2.2: Branch II: The output feature F of the feature extraction module first passes through a convolutional pooling block to extract local features of the input data, and then enters the multi-head self-attention module. In this module, each feature undergoes a linear transformation of the query, key, and value, and then the similarity between the features is calculated through a dot product operation. Finally, the attention weight is obtained through the softmax function, and finally the output feature is processed through two convolutional pooling blocks. The learning process of branch II is summarized as follows: Add the input feature F to the positional encoding to form a new feature representation F2: F2=F+Pos (6) F2 is input into the self-attention module for attention calculation; the operation of the multi-head self-attention module is called Attention, and the operation of the multi-head self-attention module is consistent with that in branch I; the output of the self-attention module is M2: M2=Attention(CP(F2)) (7) M2 is further processed through two consecutive convolutional pooling blocks to obtain the output feature B2: B2=CP(CP(M2)) (8) The output B1 of branch I and the output B2 of branch II are finally added to form the output D of the dual-scale self-attention block: D=B1+B2 (9) The EEG signals used in the pre-training stage are all original EEG signals, so the output features obtained are all original features; the output feature D obtained after processing by the dual-scale self-attention module enters the pre-trained classification module, and the pre-trained classification module directly inputs the output D into the parallel dual Softmax layer. The outputs of the dual Softmax layer correspond to the prediction probability of motor imagery, as shown in formula (10), and the prediction probability of the source subject, as shown in formula (11); The expressions of formula (10)-formula (11) are as follows: P MI =sm(D) (10) P sub =sm(D) (11) In formulas (10) and (11), D represents the feature data transferred from the dual-scale self-attention module to the parallel Softmax layer, sm represents the Softmax layer, and P MI represents the predicted probability of motor imagery, P sub represents the predicted probability of the subject.

5. The motor imagery EEG signal recognition algorithm based on the dual-scale dual soft maximum converter with mirror center loss according to claim 4 is characterized in that: Step 4 of the pre-training phase is implemented in the following steps: The motor imagery prediction probability P obtained in step 3 of the pre-training phase is MI and the predicted probability P of the source subject sub Input into formula (12), calculate the sum of the two cross entropy losses of motor imagery classification and subject classification to obtain the final loss function, which is defined as follows: In formula (12), L pre represents the loss function, represents the predicted probability of motor imagery for the ith subject, represents the predicted probability of the i-th source subject, represents the label of the motor imagery of the ith subject, represents the label of the i-th source subject, and N represents the total number of subjects; Step 5 of the pre-training phase is implemented in the following steps: The DDT model is used to calculate the gradient of the total loss for each parameter. The DDT model obtains the degree of influence of each parameter on the loss function. On this basis, the Adam optimizer in the Pytorch framework is used to update the model parameters according to these calculated gradients.

6. The motor imagery EEG signal recognition algorithm based on the dual-scale dual soft maximum converter with mirror center loss according to claim 5 is characterized in that: Step 1 of the fine-tuning phase is as follows: 80% of the data of any subject sub1 in the collected dataset was used as training data without data augmentation, and the remaining 20% ​​was used as a validation set. The EEG signal was then filtered to remove irrelevant frequency components and noise, and adjusted to a state suitable for analysis. This operation is consistent with the data preprocessing operation in the pre-training stage.

7. The motor imagery EEG signal recognition algorithm based on the dual-scale dual soft maximum converter with mirror center loss according to claim 6 is characterized in that: Step 2 of the fine-tuning phase is implemented as follows: First, the electrode channel data of the left and right hemispheres are extracted from the original EEG signal of sub1; a "mirror" EEG signal is generated by exchanging the data of the left hemisphere electrode channel with the data of the right hemisphere electrode channel; The left and right hand labels of the motor imagery task of the corresponding EEG signals were swapped; Then, the original EEG signal and the mirror EEG signal of sub1 are passed into the feature extraction module and the dual-scale self-attention module of the DDT model saved in the pre-training stage. The processing of the original EEG signal and the mirror EEG signal in the feature extraction module and the dual-scale self-attention module is consistent with that in the pre-training stage. Since the original EEG signal and the mirror EEG signal are used in the fine-tuning stage, the outputs after processing by the dual-scale self-attention module are the original features and the mirror features, respectively.

8. The motor imagery EEG signal recognition algorithm based on the dual-scale dual soft maximum converter with mirror center loss according to claim 7 is characterized in that: Step 3 of the fine-tuning phase is implemented as follows: In this stage, the DDT model is evolved into the MCDDT model after the mirror center loss is introduced. The MCDDT model consists of a feature extraction module, a dual-scale self-attention module, a pre-trained classification module and a fine-tuning classification module. The fine-tuning classification module consists of a softmax layer, a classification loss and a mirror center loss. In the fine-tuning stage, the original features and mirror features generated after being processed by the dual-scale self-attention module in step 2 enter the fine-tuning classification module. The original features and mirror features are brought into formula (13). With the help of the mirror EEG signal, the mirror center loss reduces the distance between the EEG signal sample features of the event-related desynchronization on the ipsilateral side of the brain, and increases the distance between the EEG signal sample features of the event-related desynchronization on the contralateral side of the brain: In formula (13), L M is the mirror center loss function, D i is the feature of the i-th EEG signal sample; C i is the feature center of the original EEG signal of category i; is the feature center of the category different from category i, and n is the number of samples in the current batch.

9. The motor imagery EEG signal recognition algorithm based on the dual-scale dual soft maximum converter with mirror center loss according to claim 8 is characterized in that: Step 4 of the fine-tuning phase is implemented in the following steps: The original features generated by the dual-scale self-attention module in the fine-tuning stage are processed by the motor imagery Softmax, and then converted into the motor imagery category prediction probability, and the cross entropy loss is calculated (14): In formula (14), L C represents the classification loss, N is the number of samples in the batch, is the true motor imagery category label of the i-th sample, is the model’s predicted probability of the motor imagery category for the i-th sample.

10. The motor imagery EEG signal recognition algorithm based on the dual-scale dual soft maximum converter with mirror center loss according to claim 9 is characterized in that: Step 5 of the fine-tuning phase is implemented in the following steps: The mirror loss function L obtained in step 3 of the fine-tuning phase M And the cross entropy loss function L obtained in step 4 of the fine-tuning phase C The total loss function is added together to guide the optimization of the MDCCT model. The total loss function formula is defined as formula (15): L F =L C +ω·L M (15) In formula (15), L F is the total loss function, L C is the cross entropy loss, L M is the mirror center loss, ω is the weight coefficient of the mirror center loss; Step 6 of the fine-tuning phase is implemented as follows: The MCDDT model is used to calculate the gradient of the total loss for each parameter. It tracks the error transfer process from the output layer to the input layer. The result of back propagation is that the MCDDT model obtains the influence of each parameter on the loss function. On this basis, the Adam optimizer in the Pytorch framework is used to update the model parameters according to these calculated gradients.

Images