A motor imagery classification method using multi-branch feature extraction

Abstract

The application provides a motor imagery classification method using multi-branch feature extraction, comprising the following steps: S1: obtaining a raw EEG signal; S2: pre-processing the EEG signal; S3: extracting a spatial feature; S4: extracting a frequency feature; S5: extracting a time feature; S6: weighting and fusing the features extracted in the steps S3, S4 and S5; using the fused features to perform EEG signal classification by using softmax; according to the size of each parameter output by the softmax, selecting the category corresponding to the largest parameter, and the category is the final classification result; the application solves the problem of low EEG signal classification accuracy in the prior art.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of electroencephalogram signal processing, and particularly relates to a motor imagery classification method using multi-branch feature extraction. BACKGROUND

[0002] As the central hub of the human body, the brain controls a series of activities such as thoughts, actions, and emotions of a person, and all behaviors of a person are completed by the brain issuing instructions. The brain is indeed the "command center" of the human body. However, in recent years, due to various accidents, there are more and more people suffering from diseases such as epilepsy and amyotrophic lateral sclerosis. The brains of these patients can normally operate, but they have lost the ability of limb movement and cannot correctly express the signals transmitted by the brain. The great physicist Hawking is one of these patients. He suffered from amyotrophic lateral sclerosis at the age of 21, was completely paralyzed, and could not speak or complete normal behavior activities.

[0003] Amyotrophic lateral sclerosis has become a serious disease that endangers human life and health. The disease belongs to a rare disease in the world, and it is currently incurable, and the survival time of the patient is often less than 5 years. The brain is awake, but the limbs cannot be moved, which is very cruel to the patient. In recent years, with the development of science and technology and medical level, how to help these people with normal brain and abnormal limbs to communicate with the outside world has also become a problem that more and more researchers pay attention to. They turn their eyes to the field of brain science and try to help these patients build a channel to successfully express their thoughts in their brains. Brain-computer interface (BCI) technology also emerged in such a background.

[0004] Brain-computer interface is a kind of human-computer interaction technology, which can establish a direct connection path between human brain and external equipment, and through decoding the activity of brain neurons, the decoded command is transmitted to the external machine, so as to allow people to operate external machine equipment or communicate with the outside world through the brain.

[0005] Since the late 1920s, German physiologist and psychiatrist Hans Berger first collected electroencephalogram (EEG) from the scalp of human brain, and found the Alpha wave of 8-12 Hz, which laid a solid foundation for the future development of brain-computer interface technology. In 1973, Jacques Vidal published the first article on brain-computer interface based on EEG, which elaborated the use platform of brain-computer interface and the method of processing EEG. Since then, the research of brain-computer interface has been booming. With the improvement of computer technology in recent decades, various artificial intelligence algorithms such as machine learning and deep learning have emerged in an endless stream, greatly improving the accuracy of EEG decoding and making the brain-computer interface technology develop rapidly.

[0006] With the extensive research of brain-computer interface technology, it has gradually been divided into different application research directions such as motor imagery, emotion recognition, sleep staging, etc. Among them, motor imagery, as one of the important applications of brain-computer interface technology, has attracted great attention in society. Motor imagery refers to the realization of specific behaviors related to human movement in the brain, but without actual physical movement.

[0007] With the rapid progress of artificial intelligence algorithms and the rapid development of intelligent medical care, the research of brain-computer interface has been paid more and more attention. As an extremely important application in real life, the research on the classification model of brain-computer interface based on motor imagery has very high theoretical value and practical significance. Although the brain-computer interface technology has made very significant progress since its inception and has made many amazing achievements, many studies are still in the experimental stage, and only a few can be used in real life. Brain-computer interface needs to go a long way to achieve commercialization. The main reason hindering the development of brain-computer interface is the low classification accuracy of EEG, and the improvement of EEG classification accuracy is restricted by many problems:

[0008] (1) Small sample size of data set

[0009] Although non-invasive EEG is more convenient than invasive EEG, it still needs to be collected according to certain specifications and using professional equipment. At present, most of the data sets used in research are from the data sets of BCI competition in previous years. Although the data set quality is good, the sample size is often small. For deep learning, deep neural networks need a large amount of data to train to achieve better results. The sample size of the current data set cannot meet the needs. Therefore, when more and more researchers choose to use deep learning to process EEG, how to solve the problem of insufficient sample size needs further research.

[0010] (2) Loss of spatial feature information

[0011] EEG signal data is essentially a time series data, and most current researches can effectively extract the time characteristic information of the data itself. However, due to the regular spatial distribution of EEG signal collection electrodes and the clear positional relationship between each electrode channel, the EEG signal itself also has a large amount of spatial characteristic information, and more and more researches have proved that these spatial characteristic information will also affect the final classification result of the signal. The EEG signals of different motor imagery have significant differences in time domain, frequency domain and spatial domain, and the information carried by the EEG signals in each dimension is not equally important. While extracting characteristic information, it is a problem to extract appropriate time domain, frequency domain and spatial domain characteristic information using appropriate algorithms.

[0012] (3) Large amount of calculation and time consumption

[0013] The model of motor imagery needs to be operated in real time in practice, and the multi-channel motor imagery classification needs a continuous time segment signal, which further increases the amount of operation of the model. Therefore, they are not equally important. In addition, the classification model based on deep network needs a large amount of memory, calculation and time consumption when identifying EEG signals, and cannot be deployed to mobile medical devices with limited computing power. SUMMARY

[0014] The purpose of the embodiment of the present application is to provide a motor imagery classification method using multi-branch feature extraction, to solve the problem of low EEG signal classification accuracy in the prior art.

[0015] To solve the above technical problems, the technical solution adopted by the present application is a motor imagery classification method using multi-branch feature extraction, comprising the following steps:

[0016] S1: Obtain the original EEG signal, collect the original EEG signal from the public motor imagery database;

[0017] S2: EEG signal preprocessing: divide the original EEG signal into a training set, a validation set and a test set; perform equivalent segmentation processing on the EEG signal of the training set, and splice the segmented EEG signal to expand it to make the EEG signal from class imbalance to class balance;

[0018] S3: Perform feature extraction on the EEG signal expanded by sliding in the S2 process to obtain spatial features;

[0019] S4: After the EEG signal expanded by sliding in the S2 process is segmented, input a three-layer frequency feature extraction branch model to extract the frequency features contained in the EEG signal;

[0020] S5: extracting time features from the EEG signal amplified by sliding in the S2 process;

[0021] S6: weighting and fusing the features extracted in the S3, S4 and S5 processes, using the fused features to perform EEG signal classification by using softmax, and selecting the category corresponding to the largest parameter according to the size of each parameter output by the softmax, which is the final classification result;

[0022] S7: inputting the verification set into the model, taking the loss of the verification set as the standard for updating the parameters of the model, and updating the parameters of the model when the loss of the verification set decreases; finally, using the accuracy of the test set as the evaluation.

[0023] Further, the sliding amplification in S2 is specifically: cutting each channel of each category of EEG signal data into three equal parts, labeled a, b and c respectively; then randomly shuffling by category, and concatenating a, b and c in order to make the data amplified by three times.

[0024] Further, the S3 is specifically:

[0025] S31: first, according to the position arrangement of the EEG signal acquisition electrode, a graph G suitable for the EEG signal is constructed: the position of the EEG signal acquisition electrode is projected in two dimensions to obtain 8 naturally adjacent nodes for each node, including up, down, left, right, left up, right up, left down and right down, and all acquisition electrode nodes are connected with their naturally adjacent nodes to form the graph G.

[0026] S32: the original EEG signal is input into the spatial attention mechanism, and the output feature signal is combined with the graph G to perform graph convolution neural network graph embedding representation, and the position feature of the spatial arrangement of the acquisition electrode is added to the feature signal, and the two-dimensional embedding representation is performed on each channel node in the spatial feature map of each time point to obtain the spatial attention mechanism feature with graph embedding.

[0027] Further, the S4 is specifically:

[0028] S41: determining the sampling frequency of the EEG signal and determining the time of one cycle of motor imagination, using P length data points to determine the result of one motor imagination, using Q channels for each data of the EEG signal, i.e. each data has P x Q data points, and all data are used as the input of the frequency feature extraction branch model;

[0029] S42: the first layer is input into the frequency feature extraction branch model, the first layer is a general convolutional neural network (CNN), the second layer is a channel-by-channel convolutional layer, the convolution operation is performed in a two-dimensional plane, and the number of convolution kernels is the same as the number of channels of the first layer; the third layer is a point-by-point convolutional layer, the convolution kernel size is fixed as 1*1, the point-by-point convolution operation combines the feature maps output by the second layer in the depth direction, and finally the result obtained after the EEG signal sequentially passes through the three layers is the frequency feature.

[0030] Further, the S5 is specifically:

[0031] S51: data segmentation is performed on each channel of the EEG signal after the sliding amplification in S2, the EEG signal is divided into X segments, and then input into the first layer of the time feature extraction branch model;

[0032] S52: the first layer is composed of a general convolutional neural network, is used for mapping the signal, and extracting the frequency information in the EEG signal, and then input into the second layer;

[0033] S53: the second layer is composed of an attention mechanism and a fully connected network, and automatically extracts the long-distance time feature in the EEG signal.

[0034] The beneficial effects of the present application are:

[0035] 1. A multi-branch feature extraction idea is introduced in the field of motor imagery classification, a time-frequency feature extraction method is proposed, and the long-distance time-frequency feature of the electroencephalogram signal is extracted.

[0036] 2. The spatial information of the electrode position is used for motor imagery electroencephalogram signal classification for the first time, the problem of missing spatial information in the process of motor imagery electroencephalogram classification is solved, and the accuracy of motor imagery classification is effectively improved. BRIEF DESCRIPTION OF DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0038] Figure 1 is a classification method flow chart of the embodiment of the present application;

[0039] Figure 2 is a model diagram of the frequency feature extraction branch of the embodiment of the present application;

[0040] Figure 3A model diagram of a time-frequency feature extraction branch of an embodiment of the present application;

[0041] Figure 4 A multi-branch feature fusion model diagram of an embodiment of the present application. DETAILED DESCRIPTION

[0042] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative labor fall within the protection scope of the present application.

[0043] The present application proposes a method of a multi-branch feature extraction convolutional network for motor imagery electroencephalogram classification. First, according to the non-Euclidean characteristics of the electroencephalogram electrode distribution, a graph structure suitable for electroencephalogram signals is designed to fully consider the correlation between the electroencephalogram electrodes, and at the same time, the information of the original electroencephalogram signals in the time domain is mapped to the graph structure to form spatial domain feature extraction. Then a deep separable convolution is used to convolve each channel of the data to extract multi-channel frequency feature information. Finally, an improved seq-Transformer is used to extract the time domain features to form time, frequency and space three-domain feature extraction. Then the features of the three branches are fused to adaptively capture the most valuable information in the electroencephalogram time, frequency and space dimensions.

[0044] The specific process is as follows:

[0045] S1: Obtain the original EEG signal, collect the original EEG signal from the public motor imagery database (EEG Motor Movement / Imagery Dataset, BCI Competition IV-2b, etc.);

[0046] S2: EEG signal preprocessing: divide the original EEG signal into a training set, a validation set and a test set. Perform simple equal segmentation processing on the EEG signal of the training set, and splice the segmented EEG signal to expand it to make the EEG signal from class imbalance to class balance;

[0047] S3: According to the distribution of the electroencephalogram signal acquisition electrode position, a graph G structure is established. After the segmented processing of the training set obtained in the process of S2, the data of each channel is embedded and represented by using the structure of the graph G, mainly learning the graph structure characteristics of it, introducing the spatial structure information into the data, performing feature extraction, and obtaining the spatial features;

[0048] S4: The training set obtained in the S2 process is segmented and input into a three-layer special stacked convolution module for extracting frequency features contained in the EEG signal.

[0049] S5: The data obtained in the S2 process is used to extract time feature information in the sequence signal by using a seq-Transformer variant of a Transformer. In time series data, more attention is generally paid to the features between the signals before and after the sequence, especially some long-distance features, so the structure is used to extract time features.

[0050] S6: The features extracted in the S3, S4 and S5 processes are fused and weighted. The fused features are used for EEG signal classification by using softmax. According to the size of the five parameters output by the softmax, the class corresponding to the largest parameter is selected as the final classification result.

[0051] S7: The validation set is input into the model, and the loss of the validation set is used as the standard for updating the parameters of the model. Only when the loss of the validation set decreases, the parameters of the model are updated. Finally, the accuracy of the test set is used to measure the performance of the entire model as an evaluation.

[0052] Further, the specific process of S3 is as follows:

[0053] S31: First, according to the arrangement of the collected electrode positions, a graph G suitable for the EEG signal is constructed. In a natural situation, the electrode distribution is not a standard two-dimensional grid or three-dimensional grid, but a non-Euclidean space. In order to better observe the distribution of all the electrodes, the method is projected in two dimensions. Since the voltage value of each electrode node is most affected by the voltage values of its surrounding electrodes, the present application assumes that each node has eight natural adjacent nodes, namely up, down, left, right, left up, right up, left down and right down, and assumes that each node is naturally connected to itself. This method connects all electrode nodes to their natural adjacent nodes (including themselves), thus forming a graph G that can fully consider the correlation between electrodes.

[0054] S32: The feature signal output by the spatial attention mechanism is combined with the graph G to perform graph embedding representation in the graph convolutional neural network, and the position feature of the electrode arrangement in the space is added to the feature signal. The spatial feature map of each time point is doubly embedded. The spatial attention mechanism with graph embedding is obtained.

[0055] Further, the process of frequency feature extraction in S4 includes the following steps:

[0056] S41: In the EEG signal data set, the sampling frequency used when collecting the EEG is 160 Hz, and 4 s is used as a cycle for evaluating motor imagery. The length of 640 data points determines the result of one motor imagery, and 64 channels are used for each data, i.e. each data has 640x64 data points. These data are used as the input of the frequency feature extraction branch.

[0057] S42: These are input into the first layer of the frequency extraction branch model. The first layer of the model is a common convolutional neural network (CNN), the second layer is a channel-by-channel convolution operation, which is a convolution operation completely in the two-dimensional plane. The number of convolution kernels is the same as the number of channels of the previous layer. The third layer is a point-by-point convolution operation, and the convolution kernel size is fixed at 1x1. The convolution operation here will weight and combine the feature map of the previous step in the depth direction.

[0058] Further, the process of time-frequency feature extraction in S5 specifically includes the following steps:

[0059] S51: The EEG signal data after sliding amplification in S2, i.e. 640x64 data, is segmented for each channel, divided into 32 segments, and then input into the first layer of the time-frequency feature extraction branch.

[0060] S52: The first layer of the model is composed of a common convolutional neural network, which mainly maps the signal and extracts the frequency information in the original signal. Then input the second layer for long-distance time feature extraction.

[0061] S53: The second layer is composed of an attention mechanism and a full connection. The long-distance time features in the EEG signal are automatically extracted.

[0062] Further, the process of multi-branch feature fusion in S6 specifically includes the following steps:

[0063] The multi-branch extracted features are weighted, and then the self-attention mechanism is performed, which extracts the most important features for classification. These signal features are used to train the classifier, input into the full connection layer, and output the final prediction result through the softmax.

[0064] The following mainly explains the specific implementation of the present application:

[0065] 1. EEG signal collection

[0066] The present invention uses the benchmark dataset commonly used in motor imagery, the EEG Motor Movement / Imagery Dataset. These recordings were obtained from 109 subjects. The subjects performed different motor / imaging tasks while 64-channel electroencephalogram was recorded using the BCI2000 system. Each subject performed 14 experiments: two one-minute baseline motor (one with eyes open, one with eyes closed), and 3 times of each of the following four tasks, each for two minutes:

[0067] The target appears on the left or right side of the screen. The subject opens and closes the corresponding fist until the target disappears. Then the subject relaxes.

[0068] The target appears on the left or right side of the screen. The subject imagines opening and closing the corresponding fist until the target disappears. Then the subject relaxes.

[0069] The target appears on the top or bottom of the screen. The subject opens and closes both fists (if the target is on the top) or both feet (if the target is on the bottom) until the target disappears. Then the subject relaxes.

[0070] The target appears on the top or bottom of the screen. The subject imagines opening and closing both fists (if the target is on the top) or both feet (if the target is on the bottom) until the target disappears. Then the subject relaxes.

[0071] The data is provided here in EDF+ format (contains 64 EEG signals, each sampled at 160 samples per second and one annotation channel).

[0072] In these data sets, the recordings are divided into 5 periods according to the classification, namely B, R, RL, L, F. B represents baseline motor, R represents imagining opening the left fist, L represents imagining opening the right fist, RL represents imagining opening both hands simultaneously, and F represents imagining opening both feet. As shown in Table 1, the number of data of each motor imagery category of the data set is counted, and it can be seen that the data is unbalanced. In the present invention, 64 channels of data are used to evaluate the method of the present invention.

[0073] Table 1. Distribution table of each category of motor imagery data set

[0074] Dateset B R RL L F Number 44496 11035 11217 11074 11030

[0075] 2. Data preprocessing

[0076] For EEG signals, except for B category, sliding augmentation data is used: each channel of each category data is divided into three equal parts, labeled a, b, and c respectively. Then randomly shuffle by category, and splice a, b, and c in a certain order. Such as bca, cba, acb, etc. Make the data three times larger.

[0077] 3. Motor imagery classification method

[0078] As shown in Figure 1 , a flow chart of the method of the present application is shown. The model of the present application adopts three branches to extract features in different domains. Because each branch focuses on different points, more effective features for classification can be extracted more efficiently. As shown in Figure 2 , the first branch is an effective frequency feature extraction convolutional network, which adopts three layers of stacking, in order: ordinary convolutional layer, channel convolutional layer and point-by-point convolutional layer. The ordinary convolutional layer has a 3x3 convolutional kernel. Channel convolution is a convolution operation entirely in the two-dimensional plane. The number of convolution kernels is the same as the number of channels of the previous layer (ordinary convolutional layer). The channels and convolution kernels correspond one by one, so the number of output feature map channels remains unchanged. The convolution kernel size of the point-by-point convolution is fixed at 1x1. Assuming its output channel is N, then its convolution kernel shape is 1x1xMxN, M is the number of channels of the previous layer (channel convolutional layer), so the convolution operation here will combine the feature map of the previous step in the depth direction. The result obtained after the three layers of convolution of the electroencephalogram signal is the frequency feature obtained. Figure 3 As shown in Figure 4 , a feature fusion structure network is shown. The time features, frequency features extracted by the previous branches and the spatial features obtained from the graph G are input into the Flatten layer for alignment, then input into the concatenate layer for splicing to obtain the result, and finally input into the SelfAttention layer for similarity calculation to obtain the final aggregated features. This structure is specially used for multi-branch feature fusion, which can automatically select the features that contribute most to classification and perform weighted fusion to obtain the final result.

[0079] 4. Data set division

[0080] The present application adopts a k-fold cross-validation strategy to evaluate the model. When dividing the training set and the test set, each object in the data set is independent. In the experiment of the present application, the k of the data set is set to 10. Specifically, the present application uses N-(N / k) data records for training, and the remaining N / k data records for testing, where N is the total number of data in the data set. In the training process, (N / k) data in the training set are randomly selected as the validation set, which is used to select the appropriate training iteration number and the standard for stopping training in advance. The model of the present application needs to be trained and tested 10 times on the motor imagery data set, and then the motor imagery category prediction results of each test data are combined to calculate the evaluation index.

[0081] 5. Result evaluation

[0082] The present application uses a plurality of evaluation indexes to evaluate the method of the present application, which are accuracy, recall rate, precision, and F1 value (in order to evaluate the advantages and disadvantages of different algorithms, the concept of F1 value is proposed on the basis of accuracy and recall rate to evaluate the accuracy and recall rate as a whole. The definition of F1 is as follows: F1 value = accuracy * recall rate * 2 / (accuracy + recall rate)).

[0083] Firstly, the present application uses different feature extraction branches to predict the motor imagery data set, and the overall accuracy ACC and F1 value obtained are as shown in Table 2 below, which is the ablation comparison data of the results of the classification experiment using different branches:

[0084] Table 2. Ablation comparison data of the results of the classification experiment using different branches

[0085]

[0086] From Table 2, it can be seen that the overall accuracy ACC and F1 value of the Base-balance method compared with the Base-no_balance method has certain improvement, which proves that the data balancing method of the training set is useful for improving the accuracy of the model. The Base-balance method, the Seq-transformer method and the SG-net method represent using only the frequency feature extraction branch to do classification, using the time-frequency feature extraction branch to do classification and using the space feature extraction branch to do classification respectively. The Two-branch net method represents fusing the frequency feature extraction branch and the time-frequency feature extraction branch together to do the classification model. From the overall accuracy ACC, 70% is obviously higher than 63% and 60%, which proves that the multi-branch fusion idea is feasible. The method of the application, i.e. the final three-branch fusion classification model method, has an overall accuracy ACC value of 97.59%, which achieves good performance, and the F1 value as an overall evaluation index also reaches 97.39%. Therefore, the method of the application shows good classification performance on the motor imagery classification task. Compared with the experimental results of using only a single branch, using multi-branch classification can greatly improve the accuracy of the experimental results, which proves that the features extracted by each branch can complement each other for the final classification of the experimental results.

[0087] Further, the application uses the multi-branch model to predict on the EEG Motor Movement / Imagery Dataset data set, and the obtained confusion matrix is shown in Table 3, wherein PR represents the precision, and RE represents the recall:

[0088] Table 3. Confusion matrix of the multi-branch feature extraction classification model for predicting the MI data set

[0089]

[0090] In order to better reflect the advantages of the application, the overall result evaluation result is compared with other mainstream methods:

[0091] Table 4. Comparison of the method proposed in the application with the current method

[0092]

[0093]

[0094] From the above table 4, it can be seen that the overall accuracy of the multi-branch feature extraction model of the present application on the MI data set reaches 97.59%, which has reached the benchmark of the existing method. Through table 2, it can be seen that the multi-branch feature extraction only focuses on the features in the fixed domain. The time-frequency feature extraction branch mainly focuses on the time-frequency features of long distance, the frequency feature extraction branch uses CNN to extract frequency features. And the spatial domain feature extraction branch introduces the spatial information of electrode position. It mainly focuses on the extraction of spatial feature information. Especially, the electroencephalogram signal belongs to a sequence signal, and in many sequence transduction tasks, learning long-range dependencies is a key challenge. A key factor affecting the ability to learn such dependencies is the path length that forward and backward signals must travel in the network. The shorter these paths between any combination of positions in the input and output sequences, the easier it is to learn long-range dependencies. In the CNN type model, the number of operations required to calculate the association between two positions through convolution increases with distance. While the number of operations required to calculate the association between two positions in the self-attention of the Transformer is independent of distance. Therefore, long-distance features can be effectively extracted. The combination of features improves the classification accuracy of the model, and the classification result is more advantageous.

[0095] Finally, in order to compare the time performance of the multi-branch feature extraction electroencephalogram motor imagination classification model proposed by the present application, the present application carries out experiments on a computer with a memory of 128G, a CPU of Intel(R) Xeon(R) Silver 4210 CPU@2.20GHz, and a graphics card of NVIDIA Tesla V100. The training time of the multi-branch feature extraction model proposed by the present application is only about 10 minutes, and the test time is only 13 milliseconds. While the training time of most current models is about 1 hour, which proves the time advantage of the representation learning model proposed by the present application.

[0096] Each embodiment in the specification is described in a related manner, and the same or similar parts between each embodiment can be referred to each other, and each embodiment mainly explains the difference from other embodiments. Especially, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the related parts can be referred to the part of the method embodiment.

[0097] The above only describes the preferred embodiments of the present application, and is not used to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A motor imagery classification method using multi-branch feature extraction, characterized by, Comprise the following steps: S1: original EEG signal acquisition, collect the original EEG signal from the public motor imagination database; S2: EEG signal preprocessing: divide the original EEG signal into a training set, a validation set and a test set; the EEG signal of the training set is subjected to equivalent segmentation processing, and the segmented EEG signal is spliced and amplified in sliding manner, so that the EEG signal is balanced from class imbalance to class balance; S3: the EEG signal amplified in sliding manner in the S2 process is subjected to feature extraction, and spatial features are obtained; S4: after the EEG signal amplified in sliding manner in the S2 process is subjected to segmentation processing, input a three-layer frequency feature extraction branch model for extracting the frequency features contained in the EEG signal; S5: the EEG signal amplified in sliding manner in the S2 process is subjected to time feature extraction; S6: the features extracted in the S3, S4 and S5 processes are weighted and fused, the fused features are subjected to EEG signal classification by using softmax, according to the size of each parameter output by the softmax, the class corresponding to the largest parameter is selected, and the class is the final classification result; S7: input the validation set into the model, take the loss of the validation set as the standard for updating the model parameters, update the parameters of the model when the loss of the validation set decreases, and finally take the accuracy of the test set as the evaluation; The S3 is specifically: S31: first, according to the position arrangement of the EEG signal acquisition electrode, a graph G suitable for the EEG signal is constructed: the position of the EEG signal acquisition electrode is projected in two dimensions, so that each node has 8 natural adjacent nodes, including up, down, left, right, left up, right up, left down and right down, all the nodes of the acquisition electrode are connected with their natural adjacent nodes to form the graph G; S32: the feature signal output by the spatial attention mechanism is combined with the graph G to do graph convolution neural network in the graph embedding representation, the position feature of the spatial arrangement of the acquisition electrode is added to the feature signal, and the double embedding representation of each channel node in the spatial feature map of each time point is obtained to obtain the feature of the spatial attention mechanism with graph embedding; The S4 is specifically: S41: determine the sampling frequency of the EEG signal, and determine the time of evaluating a period of motor imagination, determine the result of a motor imagination with data points with a length of P, use Q channels for each data of the EEG signal, that is, each data has P*Q data points, and all the data are used as the input of the frequency feature extraction branch model; S42: the above data are input into the first layer of the frequency feature extraction branch model, the first layer is a common convolutional neural network CNN; The second layer is a channel-by-channel convolutional layer, the convolution operation is performed in the two-dimensional plane, and the number of convolution kernels is the same as the number of channels of the first layer; the third layer is a point-by-point convolutional layer, the convolution kernel size is fixed at 1*1, and the point-by-point convolution operation combines the feature maps output by the second layer in the depth direction, and the result obtained after the EEG signal sequentially passes through the three layers is the frequency feature obtained; The S5 is specifically: S51: data segmentation is performed on each channel of the EEG signal after sliding expansion in S2, divided into X segments, and then input into the first layer of the time feature extraction branch model; S52: the first layer is composed of a general convolutional neural network, which is used for mapping and extracting frequency information in the EEG signal, and then input into the second layer; S53: the second layer is composed of an attention mechanism and a fully connected network, which automatically extracts long-distance time features in the EEG signal. 2.The motor imagery classification method using multi-branch feature extraction of claim 1, wherein, The sliding expansion in S2 is specifically: each channel of each category data of the EEG signal is evenly divided into three parts, labeled a, b, and c respectively; then randomly shuffled by category, and a, b, and c are spliced in order to make the data expanded by three times.

Images