Motor Imagery Classification Method and System Based on Dual-Scale Brain Region Features

Abstract

The present invention relates to the field of electroencephalogram (EEG) signal classification tasks, and discloses a motor imagery classification method and system based on dual-scale brain region features. The method includes: obtaining EEG signal data and dividing it into a training set and a test set, constructing information representations of the entire acquisition region of the brain and specific brain regions from two scales, namely the global scale and the local scale; training a classification model using the training set, and inputting the test set into the classification model for classification prediction; training and prediction include: extracting temporal and spatial features from the global and local scales, adaptively enhancing the temporal expression of the global scale features based on a temporal recurrent neural network to obtain global features, and adaptively enhancing the temporal expression of the local scale features to obtain local features; extracting dual-scale fusion features for motor imagery classification; the system includes modules for implementing each step. The present invention fully utilizes the differences and connections of dual-scale features from the global and local scales of the brain, improving the recognition accuracy, classification performance, and generalization ability of EEG motor imagery.

Description

Technical Field

[0001] The present invention relates to the field of electroencephalogram (EEG) signal classification tasks, and particularly to a motor imagery classification method and system based on dual-scale brain region features. Background Art

[0002] A brain-computer interface (BCI) refers to creating a connection path for information exchange between the brain or nervous system of an organic life form and a device with processing or computing capabilities to achieve information exchange and control. EEG (Electroencephalogram) signals are bioelectrical information reflecting the physiological activities of the human brain and are widely used in BCIs to convert observed brain activities into meaningful information for communication between the brain and the external environment. With the rapid development of computer technology, the intention of the human brain can be effectively obtained by analyzing EEG. Among them, EEG-based motor imagery BCIs have been widely used in many fields, such as electric wheelchairs, hand exoskeletons, and artificial prosthetics.

[0003] Traditional EEG analysis methods mainly involve an artificial feature extraction stage and a classification stage. Among them, the artificial extraction of features mainly relies on prior knowledge, which is very likely to ignore potential useful information in EEG signals, often consuming a large amount of time cost. And current classifiers often cannot achieve a high generalization ability.

[0004] The emergence of deep learning has also accelerated the development of the motor imagery classification task of EEG signals. More and more deep learning models, such as EEGNet, Sinc-ConvNet, ShallowNet, and CNN+LSTM, have been used in the motor imagery classification task of EEG signals. Deep learning has the advantages of end-to-end learning and high generalization ability. Related research shows that the activation within the frontal and posterior parietal motor regions carries information about the motor imagery content, that is, the type of imagined action. However, in a large number of studies on the motor imagery classification of EEG signals, most only regard the information of all EEG electrodes as a whole, ignoring the connection between the information of specific brain regions and the motor imagery task, as well as the potential information of specific brain regions. It is impossible to pay attention to and make full use of the connections and differences between specific brain regions, resulting in low classification accuracy and generalization ability of the model. Summary of the Invention

[0005] Therefore, the technical problem to be solved by the present invention is to overcome the deficiencies in the prior art and provide a motor imagery classification method and system based on dual-scale brain region features, which can make full use of the differences and connections of dual-scale features from both the global and local scales of the brain to improve the recognition accuracy, classification performance, and generalization ability of EEG motor imagery.

[0006] To solve the above technical problems, the present invention provides a motor imagery classification method based on dual-scale brain region features, including:

[0007] Obtain electroencephalogram (EEG) signal data and divide it into a training set and a test set. According to the correspondence between EEG electrodes and brain regions, obtain information representations of the entire acquisition region and specific brain regions of the brain in the training set and the test set from two scales: global and local.

[0008] Construct a classification model, use the training set to train the classification model, and input the test set into the trained classification model for classification prediction of motor imagery tasks.

[0009] The processes of training and prediction include:

[0010] Extract the temporal and spatial features of the entire acquisition region of the brain from the global-scale information as global-scale features, and adaptively enhance the temporal expression of the global-scale features based on a time recurrent neural network to obtain global features; extract the temporal and spatial features of specific brain regions from the local-scale information respectively and perform feature fusion as local-scale features, and adaptively enhance the temporal expression of the local-scale features after feature fusion based on a time recurrent neural network to obtain local features; extract the differences and connections between the global features and the local features based on a multi-head attention mechanism to obtain dual-scale fusion features, and use a fully connected layer for classification of motor imagery tasks.

[0011] In an embodiment of the present invention, according to the correspondence between EEG electrodes and brain regions, obtaining information representations of the entire acquisition region and specific brain regions of the brain in the training set and the test set from two scales: global and local, includes:

[0012] Preprocess the EEG signal data, and select sequences of all motor imagery tasks for normalization processing.

[0013] Divide the brain region electrodes corresponding to specific brain regions and the electrodes of the entire region of the brain, and combine the time window length of the samples to construct information representations of the entire acquisition region and specific brain regions of the brain.

[0014] In an embodiment of the present invention, the constructing information representations of the entire acquisition region and specific brain regions of the brain by combining the time window length of the samples is specifically:

[0015] Establish the brain region electrodes of a specific brain region as a C i ×T matrix, where C i is the number of brain region electrodes of this brain region, T is the time window length of the samples corresponding to the brain region electrodes, and i is the serial number of the brain region;

[0016] When i ∈ [1, N], it is the information representation of a specific brain region, where N is the total number of brain regions; when i = 0, it is the information representation of the entire acquisition region of the brain.

[0017] In one embodiment of the present invention, when extracting the global-scale features and local-scale features, they are extracted through a spatio-temporal feature extraction module, specifically:

[0018] A spatio-temporal feature extraction module is established. The spatio-temporal feature extraction module includes 2D convolution, depth convolution, and separable convolution. An activation function and a batch normalization layer are provided after the depth convolution and separable convolution, and a batch normalization layer is provided after the 2D convolution;

[0019] The vector sizes of the information representations of the entire acquisition area of the brain and the information representation of a specific brain region are adjusted and respectively input into the spatio-temporal feature extraction module to obtain the global-scale features and local-scale features.

[0020] In one embodiment of the present invention, when adaptively enhancing the temporal expression of the global-scale features based on a time recurrent neural network and the temporal expression of the local-scale features after feature fusion based on a time recurrent neural network, it is through an LSTM-MHSA module, specifically:

[0021] An LSTM-MHSA module is established. The LSTM-MHSA module includes a long short-term memory network, 2D convolution, and a multi-head self-attention mechanism;

[0022] The global-scale features and the local-scale features after feature fusion are respectively input into the LSTM-MHSA module to obtain the global features and local features.

[0023] In one embodiment of the present invention, the multi-head self-attention mechanism combines residual connection and layer normalization to enrich features.

[0024] In one embodiment of the present invention, the differences and connections between the global features and local features are extracted based on the multi-head attention mechanism to obtain a dual-scale fusion feature, and a fully connected layer is used for classification of the motor imagery task, including:

[0025] The global features are used as the Q values of the multi-head self-attention mechanism, the local features are used as the K values and V values of the multi-head self-attention mechanism, and the multi-head self-attention mechanism is used for calculation;

[0026] The global features are added to the calculation result of the multi-head self-attention mechanism as a residual to obtain a dual-scale fusion feature, and the dual-scale fusion feature is input into a fully connected layer for classification of the motor imagery task.

[0027] In one embodiment of the present invention, the calculation result of the multi-head self-attention mechanism is used as the first part, and the global feature is used as the second part. After the second part is residually connected to the first part, the first part and the second part after the residual connection are subjected to layer normalization processing to obtain the dual-scale fusion feature.

[0028] In one embodiment of the present invention, when training the classification model, the cross-entropy loss function is used to minimize the difference between the classification result and the corresponding true label, and the Adam optimizer is used to optimize the loss function and update the learning rate.

[0029] The present invention also provides a motor imagery classification system based on dual-scale brain region features, including:

[0030] A data acquisition module, configured to acquire electroencephalogram signal data and divide it into a training set and a test set, and obtain information representations of the entire acquisition area and specific brain regions of the brain in the training set and the test set from two scales of global and local according to the correspondence between electroencephalogram electrodes and brain regions;

[0031] A model construction module, configured to construct a classification model;

[0032] A training module, configured to train the classification model using the training set;

[0033] A test module, configured to input the test set into the trained classification model for classification prediction of the motor imagery task;

[0034] The processes of training and prediction include:

[0035] Extract the temporal and spatial features of the entire acquisition area of the brain from the global-scale information as the global-scale feature, and adaptively enhance the temporal expression of the global-scale feature based on the time recurrent neural network to obtain the global feature; extract the temporal and spatial features of specific brain regions from the local-scale information respectively and perform feature fusion as the local-scale feature, and adaptively enhance the temporal expression of the local-scale feature after feature fusion based on the time recurrent neural network to obtain the local feature; extract the differences and connections between the global feature and the local feature based on the multi-head attention mechanism to obtain the dual-scale fusion feature, and use the fully connected layer for classification of the motor imagery task.

[0036] The above technical solutions of the present invention have the following advantages compared with the prior art:

[0037] The present invention extracts the temporal and spatial features of the entire acquisition area of the brain and specific brain regions from two scales, namely the global scale and the local scale, and obtains global features and local features based on this. The differences and connections between the global features and the local features are captured and fully utilized through the multi-head attention mechanism to obtain more discriminative dual-scale fusion features, and on this basis, the classification of the motor imagery task is carried out. The recognition accuracy, classification performance, and generalization ability of electroencephalogram motor imagery are improved. Description of the Drawings

[0038] In order to make the content of the present invention easier to be clearly understood, the following further details the present invention according to the specific embodiments of the present invention in conjunction with the drawings, where:

[0039] Figure 1 is the flowchart of the method of the present invention.

[0040] Figure 2 is the schematic diagram of the correspondence between electroencephalogram electrodes and brain regions in the present invention.

[0041] Figure 3 is the overall framework diagram of the method of the present invention.

[0042] Figure 4 is the structural diagram of the time-space feature extraction module and the LSTM-MHSA module in the present invention.

[0043] Figure 5 is the flowchart of obtaining the dual-scale fusion features and the structural diagram of the multi-head attention mechanism in the present invention.

[0044] Figure 6 is the result comparison diagram of the ablation experiment in the embodiment of the present invention. Detailed Embodiments

[0045] The following further illustrates the present invention in conjunction with the drawings and specific embodiments, so that those skilled in the art can better understand the present invention and be able to implement it, but the embodiments cited do not limit the present invention.

[0046] Embodiment 1

[0047] Referring to Figure 1 and Figure 3 as shown, the present invention discloses a motor imagery classification method based on dual-scale brain region features, including the following steps:

[0048] S1: Obtain electroencephalogram signal data and divide it into a training set and a test set. According to the correspondence between electroencephalogram electrodes and brain regions, obtain the information representations of the entire acquisition area of the brain and specific brain regions in the training set and the test set from two scales, namely the global scale and the local scale;

[0049] S1-1: The electroencephalogram (EEG) signal data can be the existing EEG datasets based on motor imagery recorded in the literature, or the EEG information of the subjects collected on-site. In this embodiment, the High Gamma EEG dataset is used. This dataset was collected from 14 healthy subjects, and the data including four types of motor imagery tasks (left hand (LH), right hand (RH), foot (F), and rest (Re)) was selected. The dataset was recorded at a sampling rate of 500 Hz, with a total of 128 acquisition electrodes and the EEG signals of 14 subjects were recorded in total. For each subject, approximately 880 trials were included in the training set, and approximately 160 trials were included in the test set by the dataset provider before release.

[0050] S1-2: According to the correspondence between the EEG electrodes and the brain regions, the information representations of the entire acquisition region and specific brain regions of the brain in the training set and the test set are obtained from two scales, global and local, including:

[0051] S1-2-1: Preprocess the EEG signal data, select the sequences of all motor imagery tasks for normalization processing, and the data processing methods for the training set and the test set are the same; in this embodiment, 44 EEG electrodes highly correlated with the motor imaging tasks are selected, the EEG signals are resampled to 250 Hz, and the signals are filtered using a 0.5 Hz high-pass filter; Z-score is used to normalize each electrode channel without any data augmentation.

[0052] S1-2-2: Divide the electrode regions corresponding to specific brain regions and the electrodes of the entire region of the brain, and combine the time window length of the samples to construct the information representations of the entire acquisition region and specific brain regions of the brain.

[0053] Figure 2The part boxed by the dashed line corresponds to the specific brain regions pointed by the dashed arrows, including Frontal (frontal lobe), Center (central), Parietal (parietal lobe), which is called the local scale; the part boxed by the solid line represents the Global (global) pointed by the solid arrow, which is called the global scale. In the electrode channel names, F represents the frontal lobe, C represents the center, and P represents the parietal lobe. Therefore, the divided brain regions include the frontal lobe region, the central region, and the parietal lobe region. The brain region electrodes under the divided specific brain regions: the frontal lobe region includes FFC5h, FFC3h, FFC1h, FFC2h, FFC4h, FFC6h, FC5, FC3, FC1, FCz, FC2, FC4, FC6, FCC5h, FCC3h, FCC1h, FCC2h, FCC4h, FCC6h, a total of 19 electrodes; the central region includes C5, C3, C1, C2, C4, C6, a total of 6 electrodes; the parietal lobe region includes CCP5h, CCP3h, CCP1h, CCP2h, CCP4h, CCP6h, CP5, CP3, CP1, CPz, CP2, CP4, CP6, CPP5h, CPP3h, CPP1h, CPP2h, CPP4h, CPP6h, a total of 19 electrodes. The entire acquisition area of the brain, that is, the global scale, consists of all 44 electrodes.

[0054] Establish the brain region electrodes of the specific brain region as C i ×T matrix, C i is the number of brain region electrodes in this brain region, T is the sample time window length corresponding to the brain region electrodes, and i is the serial number of the brain region; in this embodiment, the sample time window length adopted is 4.5 s, that is, 1125 sampling points. When i ∈ [1, N], it is the information representation of the specific brain region, and N is the total number of brain regions; when i = 0, it is the information representation of the entire acquisition area of the brain.

[0055] S2: Construct a classification model including a time and space feature extractor, an attention mechanism, and a time recurrent neural network, train the classification model using the training set, and input the test set into the trained classification model for classification prediction of the motor imagery task.

[0056] The specific processes of training and prediction are as follows:

[0057] S2-1: Extract the time and space features of the entire acquisition area of the brain (referred to as global scale features) from the information of the global scale, and adaptively enhance the time expression of the global scale features based on the time recurrent neural network to obtain global features;

[0058] Extract the global scale features through the time-space feature extraction module, such as Figure 4As shown, the time-space feature extraction module includes a 2D convolution module, a depth convolution, and a separable convolution. After the depth convolution and the separable convolution, there are an activation function elu and a batch normalization layer batch_norm. After the 2D convolution, there is a batch normalization layer batch_norm.

[0059] S2-1-1: Use the 2D convolution module and the depth separable convolution module to extract the global scale features.

[0060] S2-1-1-1: Reset the input vector size to 1×C0×T, and use the 2D convolution module to extract the time features of the entire acquisition area of the brain;

[0061] S2-1-1-2: Use the depth separable convolution module to extract the spatial features of the time features to obtain the global scale features.

[0062] S2-1-2: Input the global scale features (the entire acquisition area of the brain) into the LSTM-MHSA module to obtain the global features.

[0063] As Figure 4 shown, the LSTM-MHSA module includes a long short-term memory network (Long Short-Term Memory, LSTM), a 2D convolution module, and a multi-head self-attention mechanism module; the long short-term memory network is used to extract richer time features, the 2D convolution module is used to reduce the feature dimension, and the multi-head self-attention mechanism module pays better attention to the features themselves to enhance the feature expression ability.

[0064] The multi-head self-attention mechanism combines residual connection and layer normalization layer_norm to enrich the features. The calculation formula of the multi-head self-attention mechanism is:

[0065] MultiHead(Q,K,V)=Concat(head1,…,head i ,…,head h )W 0 ,

[0066]

[0067]

[0068] where Q, K, and V are the Q value, K value, and V value obtained by respectively passing the input features through three different linear transformations; is the scaling factor, W i Q , W 0is a learnable matrix; MultiHead(Q, K, V) is the calculation result of the multi-head self-attention mechanism, Concat() is the residual connection operation, and h is the number of heads in the multi-head self-attention mechanism.

[0069] S2-1-2-1: Use a long short-term memory module to extract the potential temporal features of the global-scale features, and use 2D convolution for feature dimensionality reduction;

[0070] S2-1-2-2: Use the multi-head self-attention mechanism to better focus on the potential temporal features themselves to obtain the global features.

[0071] S2-2: Extract the temporal and spatial features of specific brain regions from the local-scale information and perform feature fusion (referred to as local-scale features), and adaptively enhance the temporal expression of the local-scale features after feature fusion based on a time recurrent neural network to obtain local features;

[0072] S2-2-1: Use a 2D convolution module and a depthwise separable convolution module to extract the temporal and spatial features of the specific brain region respectively.

[0073] S2-2-1-1: Resize the input vector size to 1×C i ×T (i ∈ [1, N]), and use a 2D convolution module to extract the temporal features of the specific brain region.

[0074] S2-2-1-2: Use a depthwise separable convolution module to extract the spatial features of the temporal features to obtain the temporal and spatial features of the specific brain region.

[0075] S2-2-2: Perform feature fusion on the temporal and spatial features of the specific brain region to obtain the local-scale features.

[0076] In this embodiment, the method of feature fusion is feature summation. Use the method of feature summation to perform feature fusion on the time-space features of the specific brain region to obtain local-scale features.

[0077] S2-2-3: Input the local-scale features (specific brain region) after feature fusion into the LSTM-MHSA module to obtain the local features.

[0078] S2-3: Extract the differences and connections between the global features and the local features based on the multi-head attention mechanism to obtain dual-scale fusion features, and higher discriminative dual-scale fusion features can be obtained; use a fully connected layer for classification of motor imagery tasks;

[0079] S2-3-1: Use the global features as the Q value of the multi-head self-attention mechanism, use the local features as the K value and V value of the multi-head self-attention mechanism, and perform calculations using the multi-head self-attention mechanism.

[0080] Take the calculation result of the multi-head self-attention mechanism as the first part, take the global feature as the second part, connect it to the first part by residual connection, and then perform layer normalization layer_norm processing on the first part and the second part after residual connection to obtain the dual-scale fusion feature.

[0081] The calculation method of the multi-head self-attention mechanism in this step is the same as that in S2-1-2. At this time, Q, K, and V are not obtained from the same input, so it is called the multi-head attention mechanism. The dual-scale fusion feature obtained through this step can capture and make full use of the differences and connections between the global feature and the local feature, ensure the main role of the global information in the classification task, and avoid generating negative results.

[0082] S2-3-2: Take the global feature as the residual and add it to the calculation result of the multi-head self-attention mechanism to obtain the dual-scale fusion feature, and input the dual-scale fusion feature into the fully connected layer for classification of the motor imagery task.

[0083] When performing motor imagery classification on the output dual-scale fusion feature through the fully connected layer, use the softmax function to calculate the motor imagery category with the highest probability:

[0084]

[0085] where, x i represents the output value of the i-th neuron, x c represents the output value of the c-th neuron, C is the number of neurons, represents the output values of all neurons.

[0086] When training the classification model, use the cross-entropy (CrossEntropy) loss function to minimize the difference between the classification result and the corresponding true label, and use the Adam optimizer to optimize the loss function and update the learning rate. Use cross-entropy as the loss function during the training process to calculate the loss metric during the training process, and define the loss function as L:

[0087]

[0088] where, N represents the number of samples, M represents the number of categories, the classification category is the same as the true category, y ic is 1, otherwise it is 0, p ic represents the probability that sample i belongs to category c.

[0089] Embodiment 2

[0090] The present invention also discloses a motor imagery classification system based on dual-scale brain region features, including a data acquisition module, a model construction module, a training module, and a testing module.

[0091] The data acquisition module is used to obtain electroencephalogram (EEG) signal data, which is divided into a training set and a testing set. According to the correspondence between EEG electrodes and brain regions, information representations of the entire acquisition region and specific brain regions of the brain in the training set and the testing set are obtained from two scales, namely the global scale and the local scale. The model construction module is used to construct a classification model. The training module is used to train the classification model using the training set. The testing module is used to input the testing set into the trained classification model for classification prediction of motor imagery tasks.

[0092] The processes of training and prediction include:

[0093] Extract the temporal and spatial features of the entire acquisition region of the brain from the information at the global scale (referred to as global-scale features), and adaptively enhance the temporal expression of the global-scale features based on a temporal recurrent neural network to obtain global features; extract the temporal and spatial features of specific brain regions from the information at the local scale respectively and perform feature fusion (referred to as local-scale features), and adaptively enhance the temporal expression of the local-scale features after feature fusion based on a temporal recurrent neural network to obtain local features; extract the differences and connections between the global features and the local features based on a multi-head attention mechanism to obtain dual-scale fusion features, and use a fully connected layer for classification of motor imagery tasks.

[0094] Aiming at the problem existing in the current existing classification methods that all EEG channels are regarded as a whole as input, thus ignoring the connection between the information of specific brain regions and motor imagery tasks, as well as the potential information of each brain region, resulting in limited classification accuracy. To solve this problem, the present invention realizes the classification of motor imagery from two perspectives: one is to construct information representations of the entire acquisition region and specific brain regions of the brain from two scales, namely the global scale and the local scale, through the correspondence between EEG acquisition electrodes and brain regions, and obtain the corresponding global features and local features through feature extraction and fusion, and retain the features of each motor imagery category from the global and local scales; the other is to capture and make full use of the differences and connections between the global features and the local features through a multi-head attention mechanism to obtain dual-scale fusion features with higher discriminability and strengthen the learning ability of the network.

[0095] The present invention includes the following: reading electroencephalogram (EEG) signal data, and constructing information representations of the entire acquisition area of the brain and specific brain regions from two scales, namely the global scale and the local scale, according to the correspondence between EEG electrodes and brain regions; extracting the spatio-temporal features (global scale features) of the entire acquisition area of the brain from the global scale information, and enhancing the temporal expression of the global scale features adaptively through LSTM-MHSA to obtain global features; respectively extracting the spatio-temporal features of specific brain regions, and enhancing the temporal expression of the local scale features after feature fusion through feature fusion and LSTM-MHSA to obtain local features; using the multi-head attention mechanism to capture and make full use of the differences and connections between the global features and the local features to obtain a more discriminative dual-scale fusion feature, and completing the motor imagery classification task. The present invention can achieve a four-class EEG signal motor imagery classification task, and has obtained a relatively high classification accuracy and a relatively low standard deviation.

[0096] To further illustrate the beneficial effects of the present invention, in this embodiment, the method of the present invention is compared with currently advanced deep learning algorithms, namely EEGNet, Sinc-ConvNet, ShallowNet, and CNN+LSTM.

[0097] The parameters set in the experiment are as follows: The Adam optimizer is used to minimize the categorical cross-entropy loss function, and the learning rate is 3e-3. The number of heads of the multi-head self-attention mechanism is set to 4, and a total of 40 epochs are trained. When one epoch of training is completed and the loss does not decrease, the learning rate is reduced by 50%, and the early stopping strategy is used during the training process.

[0098] Based on the High Gamma four-class motor imagery EEG dataset, the classification results are compared with advanced deep learning algorithms, and the specific results are shown in Table 1.

[0099] Table 1 Comparison table of classification results based on the High Gamma four-class motor imagery EEG dataset

[0100]

[0101] As can be seen from Table 1, the classification accuracy, kappa value (the value in parentheses), and standard deviation of each subject are shown. The present invention has obtained a relatively high average classification accuracy of 93.93% and a standard deviation of 3.69%.

[0102] Meanwhile, the effectiveness of the global-local features extracted by the present invention is analyzed by means of ablation experiments. The single global feature and local feature are respectively used as the baseline models for comparison, named O-G and O-L. The model proposed by the present invention is named D-S. Table 2 shows the classification accuracy, kappa value, and standard deviation of all subjects. Figure 6The Precision, Recall, and F1-score metrics of three models are given.

[0103] Table 2 Results table of classification accuracy, kappa value, and standard deviation for all subjects

[0104]

[0105]

[0106] From Table 2 and Figure 6 It can be seen that the dual-scale fusion features extracted by the present invention capture and make full use of the differences and connections between global features and local features, realizing the mutual complementation of global features and local features. As a result, the classification accuracy, kappa value, standard deviation, Precision, Recall, and F1-score metrics are improved. The experimental results show that the proposed method for motor imagery classification of dual-scale brain region features of the present invention has good classification performance and strong generalization ability.

[0107] The experiments show that the present invention realizes the accurate recognition of electroencephalogram motor imagery, improves the classification performance and reduces the standard deviation, which can provide a reference for the development of the electroencephalogram motor imagery field.

[0108] Those skilled in the art should understand that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0109] The present application is described with reference to the flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowcharts and / or block diagrams, and the combination of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to the processors of general-purpose computers, special-purpose computers, embedded processors, or other programmable data processing devices to generate a machine, such that the instructions executed by the processors of the computer or other programmable data processing devices generate means for implementing the specified functions in Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks.

[0110] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to work in a particular manner, such that the instructions stored in the computer-readable memory produce a manufacture including an instruction device that implements the functions specified in one process Figure 1 one process or a plurality of processes and / or blocks Figure 1 specified in a block or a plurality of blocks.

[0111] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus, such that a series of operational steps are performed on the computer or other programmable apparatus to produce a computer-implemented process, so that the instructions executed on the computer or other programmable apparatus provide steps for implementing the functions specified in one process Figure 1 one process or a plurality of processes and / or blocks Figure 1 specified in a block or a plurality of blocks.

[0112] Obviously, the above embodiments are only examples for clear illustration and are not limitations on the implementation manners. For those of ordinary skill in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to exhaustively list all the implementation manners here. The obvious changes or modifications derived therefrom are still within the protection scope of the present invention.

Claims

1. A motor imagery classification method based on dual-scale brain region features, characterized in that, Including: Obtain electroencephalogram (EEG) signal data and divide it into a training set and a test set. According to the correspondence between EEG electrodes and brain regions, obtain the information representations of the entire acquisition region and specific brain regions of the brain in the training set and the test set from both the global and local scales; Construct a classification model, train the classification model using the training set, and input the test set into the trained classification model for classification prediction of the motor imagery task; The processes of training and prediction include: Extract the temporal and spatial features of the entire acquisition region of the brain from the global-scale information as global-scale features, and establish an LSTM-MHSA module. The LSTM-MHSA module includes a long short-term memory network, 2D convolution, and a multi-head self-attention mechanism; adaptively enhance the temporal expression of the global-scale features based on a time recurrent neural network to obtain global features, specifically by inputting the global-scale features into the LSTM-MHSA module to obtain the global features; extract the temporal and spatial features of specific brain regions from the local-scale information respectively and perform feature fusion as local-scale features, and adaptively enhance the temporal expression of the fused local-scale features based on a time recurrent neural network to obtain local features, specifically by inputting the fused local-scale features into the LSTM-MHSA module to obtain the local features; Extract the differences and connections between the global features and the local features based on the multi-head attention mechanism to obtain dual-scale fusion features, and use a fully connected layer for classification of the motor imagery task, including: using the global features as the Q value of the multi-head self-attention mechanism, using the local features as the K value and V value of the multi-head self-attention mechanism, and performing calculations using the multi-head self-attention mechanism; adding the global features as a residual to the calculation result of the multi-head self-attention mechanism to obtain dual-scale fusion features, and inputting the dual-scale fusion features into the fully connected layer for classification of the motor imagery task.

2. The motor imagery classification method based on dual-scale brain region features according to claim 1, characterized in that: According to the correspondence between EEG electrodes and brain regions, obtain the information representations of the entire acquisition region and specific brain regions of the brain in the training set and the test set from both the global and local scales, including: Preprocess the EEG signal data and select all sequences of the motor imagery task for normalization processing; Divide the electrodes corresponding to specific brain regions and the electrodes of the entire region of the brain, and combine the time window length of the sample to construct the information representations of the entire acquisition region and specific brain regions of the brain.

3. The method for classifying motor imagery based on dual-scale brain region features according to claim 2, wherein: The construction of the information representations of the entire acquisition region and specific brain regions of the brain by combining the time window length of the sample is specifically: Establish the brain region electrodes of a specific brain region as a matrix of C i ×T, where C i is the number of brain region electrodes in this brain region, T is the time window length of the samples corresponding to the brain region electrodes, and i is the serial number of the brain region; When \(i\in[1,N]\), it is the information representation of a specific brain region, where \(N\) is the total number of brain regions; When \(i = 0\), it is the information representation of the entire acquisition region of the brain.

4. The motor imagery classification method based on dual-scale brain region features according to claim 1, wherein: When extracting the global-scale features and the local-scale features, extract them through a time-space feature extraction module, specifically: Establish a time-space feature extraction module. The time-space feature extraction module includes 2D convolution, depth convolution, and separable convolution. An activation function and a batch normalization layer are provided after the depth convolution and the separable convolution, and a batch normalization layer is provided after the 2D convolution; Adjust the vector sizes of the information representations of the entire acquisition area of the brain and the information representation of a specific brain region, and input them into the time-space feature extraction module respectively to obtain the global-scale features and local-scale features.

5. The motor imagery classification method based on dual-scale brain region features according to claim 1, characterized in that: The multi-head self-attention mechanism combines residual connections and layer normalization to enrich features.

6. The motor imagery classification method based on dual-scale brain region features according to claim 1, characterized in that: Take the calculation result of the multi-head self-attention mechanism as the first part, take the global feature as the second part and connect it to the first part residually, and then perform layer normalization on the first part and the second part after residual connection to obtain the dual-scale fusion feature.

7. The motor imagery classification method based on dual-scale brain region features according to any one of claims 1-6, characterized in that: When training the classification model, use the cross-entropy loss function to minimize the difference between the classification result and the corresponding true label, and use the Adam optimizer to optimize the loss function and update the learning rate.

8. A motor imagery classification system based on dual-scale brain region features, characterized in that, It includes: A data acquisition module for acquiring electroencephalogram signal data and dividing it into a training set and a test set, and obtaining the information representations of the entire acquisition area of the brain and a specific brain region in the training set and the test set from two scales of global and local according to the correspondence between the electroencephalogram electrodes and the brain regions; A model construction module for constructing a classification model; A training module for training the classification model using the training set; A test module for inputting the test set into the trained classification model to perform classification prediction of the motor imagery task; The processes of training and prediction include: Extract the temporal and spatial features of the entire acquisition area of the brain from the global-scale information as global-scale features, and establish an LSTM-MHSA module, which includes a long short-term memory network, 2D convolution, and a multi-head self-attention mechanism; adaptively enhance the temporal expression of the global-scale features based on the time recurrent neural network to obtain global features, specifically input the global-scale features into the LSTM-MHSA module to obtain the global features; extract the temporal and spatial features of a specific brain region from the local-scale information respectively and perform feature fusion as local-scale features, and adaptively enhance the temporal expression of the local-scale features after feature fusion based on the time recurrent neural network to obtain local features, specifically input the local-scale features after feature fusion into the LSTM-MHSA module to obtain the local features; Extract the differences and connections between the global features and the local features based on the multi-head attention mechanism to obtain dual-scale fusion features, and use a fully connected layer to perform classification of the motor imagery task, including: taking the global features as the Q values of the multi-head self-attention mechanism, taking the local features as the K values and V values of the multi-head self-attention mechanism, and performing calculations using the multi-head self-attention mechanism; adding the global features as residuals to the calculation result of the multi-head self-attention mechanism to obtain dual-scale fusion features, and inputting the dual-scale fusion features into the fully connected layer to perform classification of the motor imagery task.

Images