Feature fusion-based motor imagery electroencephalogram classification method and device

Through the feature fusion method of convolutional neural network and long-term memory network, local and global attention mechanisms are used to extract EEG signal characteristics, solving the problem of poor classification of EEG signal in the existing technology, achieving higher classification accuracy and stability.

CN118918356APending Publication Date: 2024-11-08WUHAN UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410904539.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-08
Publication Date
2024-11-08

Smart Images

  • Figure CN118918356A_ABST
    Figure CN118918356A_ABST
Patent Text Reader

Abstract

The invention provides a motor imagery electroencephalogram classification method and device based on feature fusion, and the method comprises the steps: inputting a preprocessed electroencephalogram signal into a first extraction module constructed based on a convolutional neural network, and obtaining a first feature map which is outputted by the first extraction module and is guided by local attention; inputting the preprocessed electroencephalogram signal into a second extraction module constructed based on a long-short term memory network to obtain a second feature map which is output by the second extraction module and is guided by global self-attention; and carrying out feature fusion on the first feature map and the second feature map and then carrying out classification to obtain a classification result of the motor imagery electroencephalogram. According to the method, the electroencephalogram signal features extracted based on the CNN and the LSTM are classified after being fused, the advantages of different networks are considered, the influence of redundant information on classification is reduced, and the stability in a cross-subject scene and the accuracy of a classification result are improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of brain-computer interface technology, and in particular to a method and device for classifying motor imagery electroencephalograms based on feature fusion. Background Art

[0002] In the motor imagery brain-computer interface system, artificial intelligence can only automatically identify and distinguish the categories of EEG signals generated when the user performs motor imagery, decode the user's motor intention, and provide users with movement capabilities based on brain control rather than muscle control in scenarios such as movement disorders, sports rehabilitation, military, and entertainment. In the study of motor imagery classification based on EEG signals, given the inherent complexity and heterogeneity of EEG signals, traditional machine learning methods usually require artificial feature design based on prior knowledge in the field of neuroscience. With the rapid development of deep learning, researchers have begun to apply deep neural networks to motor imagery classification tasks, hoping to automatically extract and learn the potential features contained in EEG signals. However, due to factors such as the size of the data set, the quality of EEG data, and real-time response performance, the application effect of deep learning in this field has not yet reached an ideal level. Summary of the invention

[0003] The present invention provides a method and device for classifying motor imagery electroencephalogram (EEG) based on feature fusion, so as to solve the defect of poor classification effect of EEG signals based on deep learning in the prior art, and realize an EEG signal classification method with higher classification accuracy.

[0004] The present invention provides a motor imagery electroencephalogram classification method based on feature fusion, comprising: Inputting the preprocessed electroencephalogram signal into a first extraction module constructed based on a convolutional neural network to obtain a first feature map output by the first extraction module and guided by local attention; Inputting the preprocessed EEG signal into a second extraction module based on a long short-term memory network to obtain a second feature map output by the second extraction module guided by global self-attention; The first feature map and the second feature map are subjected to feature fusion and then classified to obtain a classification result of the motor imagery electroencephalogram.

[0005] According to a motor imagery EEG classification method based on feature fusion provided by the present invention, the step of inputting the preprocessed EEG signal into a first extraction module constructed based on a convolutional neural network to obtain a first feature map output by the first extraction module after local attention guidance specifically includes: After the preprocessed electroencephalogram signal is deeply expanded, multi-scale temporal feature extraction is performed through a multi-scale dense connection module in the first extraction module, and feature map splicing is performed in the depth dimension; The local mixed attention module in the first extraction module performs time and space mixed attention weighting on the spliced ​​feature map to obtain the first feature map.

[0006] According to a motor imagery EEG classification method based on feature fusion provided by the present invention, before the step of performing multi-scale temporal feature extraction by the multi-scale dense connection module in the first extraction module, the method further includes: A multi-scale dense connection module is constructed through five time-dimension convolution branches and one time-dimension average pooling branch.

[0007] According to a method for classifying motor imagery EEG based on feature fusion provided by the present invention, before the step of weighting the spliced ​​feature map with temporal and spatial mixed attention by the local mixed attention module in the first extraction module, the method further includes: In the attention module of the channel dimension, the global maximum pooling is used instead of the global average pooling operation; In the attention module of the spatial dimension, global average pooling and global variance in the depth dimension are used to replace the compression operation; feature extraction submodules in the time dimension and the spatial dimension are constructed to obtain axial attention; The local mixed attention module is constructed based on the attention module of the channel dimension and the attention module of the spatial dimension.

[0008] According to a motor imagery EEG classification method based on feature fusion provided by the present invention, the step of inputting the preprocessed EEG signal into a second extraction module constructed based on a long short-term memory network to obtain a second feature map output by the second extraction module guided by global self-attention specifically includes: Inputting the preprocessed EEG signal into the hidden layer of the long short-term memory network in the second extraction module; Inputting the feature map output by the hidden layer into the global self-attention module obtained by the second extraction module to obtain the spatiotemporal self-attention guided weighted feature map output by the global self-attention module; The output of the last global self-attention module is used as the second feature map.

[0009] According to a motor imagery EEG classification method based on feature fusion provided by the present invention, the feature map output by the hidden layer is input into the second extraction module to obtain a global self-attention module, and the step of obtaining the spatiotemporal domain self-attention guided weighted feature map output by the global self-attention module specifically includes: Inputting the feature map output by the hidden layer into the second extraction module to obtain a global self-attention module; The global self-attention module calculates the spatial self-attention of the preprocessed EEG signal in the spatial domain, uses the spatial self-attention to weight the feature map output by the hidden layer, and then calculates the weighted feature map guided by the spatiotemporal self-attention based on the weighted result.

[0010] According to a method for classifying motor imagery electroencephalogram based on feature fusion provided by the present invention, the step of classifying the first feature map and the second feature map after feature fusion to obtain the classification result of the motor imagery electroencephalogram specifically includes: aggregating and compressing the first feature map and the second feature map in the depth dimension and then extracting spatial features; The spatiotemporal weights and depth weights of the extracted spatial features are further calibrated, and the calibrated spatial features are classified to obtain the classification results of the motor imagery electroencephalogram.

[0011] The present invention also provides a motor imagery EEG classification device based on feature fusion, comprising: A first extraction module, configured to receive the preprocessed EEG signal and output a first feature map guided by local attention; A second extraction module is used to receive the preprocessed EEG signal and output a second feature map guided by global self-attention; The classification module performs feature fusion on the first feature map and the second feature map and then classifies them to obtain a classification result of the motor imagery electroencephalogram.

[0012] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the motor imagery electroencephalogram classification method based on feature fusion as described above is implemented.

[0013] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-described motor imagery EEG classification methods based on feature fusion.

[0014] The motor imagery EEG classification method and device based on feature fusion provided by the present invention obtain a first feature map representing the local feature details of the EEG signal by extracting a first extraction module based on CNN, and obtains a second feature map representing the global dependency information of the EEG signal by extracting a second extraction module based on LSTM. The first feature map and the second feature map are classified after feature fusion, taking into account the advantages of different networks, so that the obtained EEG classification results increase the focus on important features, reduce the influence of redundant information on classification, and improve the stability in cross-subject scenarios and the accuracy of classification results. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0016] Figure 1 It is one of the flow charts of the motor imagery EEG classification method based on feature fusion provided by the present invention; Figure 2 It is a structural diagram mainly used to display the first extraction module in the motor imagery EEG classification method based on feature fusion provided by the present invention; Figure 3 It is a structural diagram mainly used to display the local mixed attention module in the motor imagery EEG classification method based on feature fusion provided by the present invention; Figure 4 It is a structural diagram mainly used to display the second extraction module in the motor imagery EEG classification method based on feature fusion provided by the present invention; Figure 5 It is a structural diagram mainly used to display the global self-attention module in the motor imagery EEG classification method based on feature fusion provided by the present invention; Figure 6 This is the second overall flow chart of the motor imagery EEG classification method based on feature fusion provided by the present invention; Figure 7 This is the third flow chart of the motor imagery EEG classification method based on feature fusion provided by the present invention; Figure 8 It is a structural schematic diagram of a motor imagery EEG classification device based on feature fusion provided by the present invention; Fig. 9 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0017] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0018] Combine the following Figures 1 to 7 The present invention introduces the motor imagery EEG classification method based on feature fusion, such as Figure 1 As shown, including: Step 101, inputting the preprocessed EEG signal into a first extraction module constructed based on a convolutional neural network, and obtaining a first feature map output by the first extraction module after being guided by local attention; Electroencephalogram (EEG) signals are formed by the synchronous sum of the postsynaptic potentials of a large number of neurons in the cortex, and are the result of the joint activity of many neurons. EEG signals have low signal intensity, broadband characteristics, low signal-to-noise ratio, non-stationarity, subject specificity, nonlinearity, and multi-channel characteristics, and have characteristics in three dimensions: time domain, space domain, and frequency domain. At the time domain level, EEG signals reveal the dynamic changes of neural activity over time; at the spatial domain level, EEG signals reveal the differences in the intensity distribution of neural activity at different scalp electrode locations, reflecting the functional activities of different brain regions; and at the frequency domain level, EEG signals reflect the different EEG rhythms contained in the signal.

[0019] Since EEG signals are easily affected by a variety of noise sources, including physiological artifacts such as eye movements, electromyography, and electrocardiogram artifacts, and non-physiological artifacts such as power frequency interference, EEG signals need to be preprocessed before classification.

[0020] The EEG signal to be classified in this application is a motor imagery EEG signal, which represents a motor intention, that is, a psychological process in which an individual imagines performing a specific action in the brain without actually performing the physical action. The study of motor imagery originated from the cognitive neuroscience exploration of the functional areas of the brain. The study found that when people imagine performing an action, even if the body does not actually perform the action, specific brain areas in the brain will still be activated. This physiological phenomenon makes the brain-computer interface system based on motor imagery possible. By identifying and decoding EEG signals related to motor imagery, non-invasive control of auxiliary equipment can be achieved based on the user's autonomous consciousness.

[0021] Specifically, the EEG signals of the subjects were collected based on the EEG acquisition equipment, and amplified and digitized. Subsequently, the EEG signals were preprocessed, the EOG (electro-oculography) channel data irrelevant to the motor imagery classification task was removed, the missing values ​​were replaced with the mean of the channel, the EEG signals were sliced, and the data in the time period related to motor imagery were extracted. Finally, the extracted data was Z-Score standardized to complete the preprocessing of the EEG signals.

[0022] The EEG signal is two-dimensional data. For the i-th subject who has performed n motor imagery, his EEG signal can be expressed as: ; Where, represents the EEG data record of the j-th motor imagery: ; Among them, C is the number of channels and T is the number of sampling points. That is to say, the preprocessed EEG data has spatial features corresponding to the number of channels and temporal features corresponding to the number of sampling points.

[0023] On this basis, a first extraction module is pre-built based on a convolutional neural network (CNN) to extract local features of the pre-processed EEG signal.

[0024] The structure of CNN mainly includes input layer, convolution layer, pooling layer, activation function, fully connected layer, output layer, and may also include Dropout layer, normalization layer, etc.

[0025] Optionally, a local mixed attention module is set in the first extraction module constructed based on CNN, which is used to guide the feature map extracted based on CNN based on the attention mechanism, so that it can focus on the key information in the input EEG signal and weaken the influence of noise, thereby extracting the first feature map of the EEG signal.

[0026] Step 102, inputting the preprocessed EEG signal into a second extraction module constructed based on a long short-term memory network, and obtaining a second feature map output by the second extraction module and guided by global self-attention; The second extraction module is pre-built based on the Long Short-Term Memory (LSTM) network to extract the long-term and short-term time dependency information of the EEG signal.

[0027] Optionally, a global self-attention module is set in the second extraction module constructed based on LSTM, which is used to guide the feature map extracted based on LSTM based on global attention, so that it can capture the global spatiotemporal dependencies in the EEG signal more finely and comprehensively, thereby extracting the second feature map of the EEG signal.

[0028] Step 103, classifying the first feature map and the second feature map after feature fusion to obtain the classification result of the motor imagery electroencephalogram.

[0029] On this basis, the first feature map extracted by the first extraction module represents the local feature details of the EEG signal, and the second feature map extracted by the second extraction module represents the global dependency information of the EEG signal.

[0030] The extracted first feature map and the second feature map are subjected to feature fusion to obtain a richer and more complete feature expression, and the fused features are classified to improve the accuracy of EEG signal classification.

[0031] The present invention obtains a first feature map characterizing local feature details of the electroencephalogram signal by extracting a first extraction module based on CNN, obtains a second feature map characterizing global dependency information of the electroencephalogram signal by extracting a second extraction module based on LSTM, and classifies the first feature map and the second feature map after feature fusion, taking into account the advantages of different networks, so that the obtained electroencephalogram classification result increases the attention to important features, reduces the influence of redundant information on classification, and improves the stability in cross-subject scenarios and the accuracy of classification results.

[0032] At the same time, the present invention dynamically classifies the collected motor imagery EEG signals without relying on prior knowledge of neuroscience, specific range filtering, and high spatial resolution, thereby improving the classification efficiency and automation level. The motor imagery brain-computer interface system is applicable in a variety of scenarios and can provide support for applications such as motor function restoration and military automation.

[0033] In the motor imagery EEG classification method based on feature fusion of the present invention, the step of inputting the preprocessed EEG signal into a first extraction module constructed based on a convolutional neural network to obtain a first feature map output by the first extraction module after local attention guidance specifically includes: After the preprocessed electroencephalogram signal is deeply expanded, multi-scale temporal feature extraction is performed through a multi-scale dense connection module in the first extraction module, and feature map splicing is performed in the depth dimension; A first extraction module is constructed to extract a first feature map.

[0034] like Figure 2As shown, the first extraction module first includes an inverted bottleneck layer, for the input preprocessed EEG signal , by inverting the bottleneck layer to expand the depth, that is, for the j-th motor imagery EEG data recording , which is expanded to Three-dimensional data. Where D is the data depth.

[0035] The first extraction module then includes a pre-built multi-scale densely connected module to perform multi-scale temporal feature extraction on the deeply expanded EEG signal.

[0036] In other feasible implementations, the extraction of temporal features may also be accomplished by arranging five separable time dimension axial convolution kernels in parallel and one time dimension axial pooling kernel.

[0037] It should be noted that there are two groups of multi-scale dense connection modules in this application. After the feature extraction results of the first group of multi-scale dense connection modules are spliced, they are input into the second group of multi-scale dense connection modules after the transition layer. After the second group of multi-scale dense connection modules extract the time features, the feature maps are spliced ​​in the depth dimension, and the size of the spliced ​​feature maps is .

[0038] The local mixed attention module in the first extraction module performs time and space mixed attention weighting on the spliced ​​feature map to obtain the first feature map.

[0039] Continue to refer to Figure 2 The first extraction module also includes a pre-built local mixed attention module. The local mixed attention module is based on the attention mechanism. It weights the spliced ​​feature map by calculating the weight map in the time dimension and the space dimension to obtain the first feature map, so that the first feature map can represent the local feature details of the EEG signal.

[0040] In the motor imagery EEG classification method based on feature fusion of the present invention, before the step of extracting multi-scale temporal features through the multi-scale dense connection module in the first extraction module, the method further includes: A multi-scale dense connection module is constructed through five time-dimension convolution branches and one time-dimension average pooling branch.

[0041] like Figure 2 As shown in Figure 1, a multi-scale dense connection module is pre-constructed through five time dimension convolution branches and one time dimension average pooling branch.

[0042] In the convolution branch, features are extracted through dense connections, and the densely connected nonlinear transformation composite function of the i-th branch is The steps involved are: ; In the formula, BatchNorm is batch normalization, GELU is the activation function, It is related to the sampling frequency and is calculated as follows: ; Through dense connections, the present invention enables each layer in the network to access and integrate the feature information extracted by all previous layers, thereby making full use of the low-level feature details and high-level semantic information in the EEG signal, introducing dense connections into multi-scale branches, and constructing a multi-scale dense connection module, which can retain more complete feature information while extracting features of different scales.

[0043] In the motor imagery EEG classification method based on feature fusion of the present invention, before the step of weighting the spliced ​​feature map with temporal and spatial mixed attention by the local mixed attention module in the first extraction module, the method further includes: In the attention module of the channel dimension, the global maximum pooling is used instead of the global average pooling operation; A local hybrid attention module (svSE module) is pre-built so that the attention mechanism can focus on the key information in the input data and weaken the impact of noise by dynamically assigning weights to the feature maps extracted by the multi-scale feature module.

[0044] like Figure 3 As shown, the svSE module in this embodiment is improved based on the attention mechanism module (scSE module). The scSE module includes a channel dimension attention module (cSE module) and a spatial dimension attention module (sSE module).

[0045] For the cSE module, the global maximum pooling is used to replace the global average pooling operation to highlight the significant features and obtain the weight map , its shape is .

[0046] In the attention module of the spatial dimension, global average pooling and global variance in the depth dimension are used to replace the compression operation; feature extraction submodules in the temporal and spatial dimensions are constructed to obtain axial attention; The local mixed attention module is constructed based on the attention module of the channel dimension and the attention module of the spatial dimension.

[0047] For the sSE module, improvements are made in two ways, and the weights obtained in the two ways are combined to obtain the final output.

[0048] First, for the form The global average pooling and global variance calculation operations in the depth dimension are used to replace the compression operation in the scSE module, and the obtained The output of , then the feature map is aggregated in the depth dimension through 1×1 convolution, and the shape is obtained The weight graph , so as to better characterize the temporal variation characteristics of EEG signals.

[0049] The second is to construct feature extraction sub-modules in the time dimension and space dimension to extract features from the time dimension and space dimension respectively to obtain axial attention.

[0050] In a feasible implementation, for a The input, in the spatial dimension, first uses 1×1 convolution to perform a deep compression operation, and the shape is The output of is then used to obtain two feature maps through average pooling and maximum pooling in the time dimension, and the resulting shape is The two feature maps are fused through 1×1 convolution to obtain the output of Output.

[0051] In the time dimension, a convolution operation is performed on the spatial dimension to obtain the shape The timing weight of .

[0052] Finally, the spatial weight and the temporal weight are multiplied by the Kronecker product to restore the dimension and obtain the final weight map. , its shape is .

[0053] On this basis, the formula of the svSE module provided in this embodiment is as follows: ; ; ; ; Among them, X is the input of the svSE module, that is, the feature map extracted by the multi-scale feature module; is the weighted output, which is the first feature map; is element-wise addition, It is element-by-element multiplication and Expand is a broadcast operation.

[0054] In the motor imagery EEG classification method based on feature fusion of the present invention, the step of inputting the preprocessed EEG signal into a second extraction module constructed based on a long short-term memory network to obtain a second feature map output by the second extraction module and guided by global self-attention specifically includes: Inputting the preprocessed EEG signal into the hidden layer of the long short-term memory network in the second extraction module; The second extraction module is constructed based on LSTM, such as Figure 4 As shown, for the shape The input of the EEG signal is regarded as the channel dimension as the feature of the data at the corresponding time step (sampling point), so as to align it with the input of LSTM, and its output is where H is the number of hidden layers.

[0055] In the hidden layer, LSTM cells are used to build a single-layer recurrent neural network, and the activation function uses the tanh function. The LSTM cell contains an input gate , Forget Gate , cell status , output gate The subscript t indicates the time t. And the output at time t-1 Splice and input into the cell to calculate the output at time t The forward propagation calculation formula is as follows: ; ; ; ; ; ; Where W and b are the corresponding weight coefficient matrix and bias term, respectively. is the sigmoid activation function.

[0056] Inputting the feature map output by the hidden layer into the global self-attention module obtained by the second extraction module to obtain the spatiotemporal self-attention guided weighted feature map output by the global self-attention module; The output of the last global self-attention module is used as the second feature map.

[0057] Further, such as Figure 4 As shown in the figure, a global self-attention module (SCoT) is constructed and added to the hidden layer of LSTM. The global spatiotemporal self-attention is calculated through the SCoT module. The feature map of H is still regarded as the feature of the channel dimension to integrate the features from the LSTM output, thereby improving the network's ability to capture the global spatiotemporal dependencies of the input EEG signal.

[0058] like Figure 5 As shown in FIG, the output of the last SCoT module is used as the second feature map for classification after fusion with the first feature map.

[0059] In the motor imagery EEG classification method based on feature fusion of the present invention, the feature map output by the hidden layer is input into the second extraction module to obtain a global self-attention module, and the step of obtaining the spatiotemporal domain self-attention guided weighted feature map output by the global self-attention module specifically includes: Input the feature map output by the hidden layer into the second extraction module to obtain a global self-attention module; The global self-attention module calculates the spatial self-attention of the preprocessed EEG signal in the spatial domain, uses the spatial self-attention to weight the feature map output by the hidden layer, and then calculates the weighted feature map guided by the spatiotemporal self-attention based on the weighted result; like Figure 6 As shown in the figure, a global self-attention module is pre-built. The SCoT module calculates the global spatiotemporal attention of the EEG signal in two processes. First, the spatial self-attention in the spatial domain is calculated by the spatial self-attention module. After the input is weighted by the spatial self-attention, the spatiotemporal self-attention module is used to calculate the spatiotemporal self-attention based on the weighted data.

[0060] in, Figure 6 The part within the yellow dotted border represents the spatial self-attention module, and the part within the blue dotted border represents the spatiotemporal self-attention module.

[0061] Specifically, for the form The overall process of the SCoT module calculating attention for the input X is shown in the formula: ; In the formula, is the spatial self-attention module, It is the spatiotemporal self-attention module.

[0062] In the spatial self-attention module In , a mixed pooling operation is first performed to transform the input, as shown in the formula: ; In the formula, is a pooling representation, such as , then three sets of 1×1 convolutions are used to obtain the key, query, and value matrices K, Q, and V. K is transposed and multiplied with Q to extract its similarity features, and normalized using softmax to obtain the weight matrix A. A is then multiplied with V to weight the feature map, and expanded to the same size as the input feature map by broadcasting, and summed with the input feature map X to obtain the output of spatial self-attention.

[0063] The output of spatial self-attention is used as the input Y of the spatiotemporal self-attention module to utilize the enhanced spatial features in the spatiotemporal domain. First, the local context features in the temporal domain are extracted through 1×kernel convolution, so that the generated key matrix K has the static context information of the input feature map. The query matrix Q is not transformed, and the value matrix V is obtained using a 1×1 matrix. Among them, The calculation of is as follows: ; In the formula, is the sampling rate. Then, K and Q are aggregated in the depth dimension, their similarity features are extracted, and fused with the V matrix to obtain the dynamic context information L of the feature map. The calculation process is as follows: ; Where Fusion is a local matrix multiplication operation, which is used to measure the pairwise relationship between each query and the corresponding key in the local grid. It uses axial convolution to perform local matrix multiplication operations in the time domain and spatial domain, respectively, followed by batch normalization and Softmax normalization.

[0064] Finally, Y and L are fused to obtain the output of the spatiotemporal self-attention module [Att]_st. Y and L are aggregated and fused by convolution to achieve feature calibration. Finally, the calibrated weighted output is obtained by element-by-element multiplication, which is the feature map guided by spatiotemporal self-attention weighting. The calculation formula is as follows: ; The present invention allocates more attention to important electrodes (channels) through spatial self-attention, highlighting the relative importance of different channels when local correlation is weak. Spatiotemporal self-attention can make full use of the local correlation in time series data and the spatial feature information that has been weighted and strengthened by spatial self-attention, so as to capture the global spatiotemporal dependency in EEG signals more finely and comprehensively, and at the same time calibrate the attention obtained in the previous step.

[0065] In the method for classifying motor imagery electroencephalogram based on feature fusion of the present invention, the step of classifying the first feature map and the second feature map after feature fusion to obtain the classification result of the motor imagery electroencephalogram specifically includes: aggregating and compressing the first feature map and the second feature map in the depth dimension and then extracting spatial features; The spatiotemporal weights and depth weights of the extracted spatial features are further calibrated, and the calibrated spatial features are classified to obtain the classification results of the motor imagery electroencephalogram.

[0066] like Figure 5 As shown in the figure, the first feature map extracted based on CNN and the second feature map extracted based on LSTM are aggregated in the depth dimension and compressed in the depth dimension through 1×1 point convolution. Subsequently, in the spatial convolution layer, spatial features are extracted through axial convolution in the spatial dimension, and dimensionality reduction is performed in the depth dimension.

[0067] Furthermore, in conjunction with reference Figure 6 and Figure 2 ,There is also an svSE module connected after the spatial convolution layer. The spatiotemporal weights and depth weights of the feature map are further calibrated through the scSE attention module to achieve more accurate results.

[0068] Finally, the output of the svSE module is averaged and pooled in the time dimension, and the classification prediction results of the EEG signal are output through the fully connected layer and softmax. The complete process is as follows: Figure 7 As shown. Among them, Figure 7 The DIS-Net model in represents the first extraction module, the LS-Net module represents the second extraction module, and HA-FuseNet is a pre-built model for classifying EEG signals.

[0069] The following is a description of the motor imagery EEG classification device based on feature fusion provided by the present invention. The motor imagery EEG classification device based on feature fusion described below and the motor imagery EEG classification method based on feature fusion described above can be referenced to each other.

[0070] like Figure 8 As shown, the motor imagery EEG classification device based on feature fusion of the present invention includes a first extraction module 801, a second extraction module 802 and a classification module 803: A first extraction module 801 is used to receive the preprocessed EEG signal and output a first feature map guided by local attention; A first extraction module is pre-built based on a convolutional neural network (CNN) to extract local features of the pre-processed EEG signal.

[0071] The structure of CNN mainly includes input layer, convolution layer, pooling layer, activation function, fully connected layer, output layer, and may also include Dropout layer, normalization layer, etc.

[0072] Optionally, a local mixed attention module is set in the first extraction module constructed based on CNN, which is used to guide the feature map extracted based on CNN based on the attention mechanism, so that it can focus on the key information in the input EEG signal and weaken the influence of noise, thereby extracting the first feature map of the EEG signal.

[0073] A second extraction module 802 is used to receive the preprocessed EEG signal and output a second feature map guided by global self-attention; The second extraction module is pre-built based on the Long Short-Term Memory (LSTM) network to extract the long-term and short-term time dependency information of the EEG signal.

[0074] Optionally, a global self-attention module is set in the second extraction module constructed based on LSTM, which is used to guide the feature map extracted based on LSTM based on global attention, so that it can capture the global spatiotemporal dependencies in the EEG signal more finely and comprehensively, thereby extracting the second feature map of the EEG signal.

[0075] The classification module 803 performs feature fusion on the first feature map and the second feature map and then classifies them to obtain a classification result of the motor imagery EEG.

[0076] On this basis, the first feature map extracted by the first extraction module represents the local feature details of the EEG signal, and the second feature map extracted by the second extraction module represents the global dependency information of the EEG signal.

[0077] The extracted first feature map and the second feature map are subjected to feature fusion to obtain a richer and more complete feature expression, and the fused features are classified to improve the accuracy of EEG signal classification.

[0078] The present invention obtains a first feature map characterizing local feature details of the electroencephalogram signal by extracting a first extraction module based on CNN, obtains a second feature map characterizing global dependency information of the electroencephalogram signal by extracting a second extraction module based on LSTM, and classifies the first feature map and the second feature map after feature fusion, taking into account the advantages of different networks, so that the obtained electroencephalogram classification result increases the attention to important features, reduces the influence of redundant information on classification, and improves the stability in cross-subject scenarios and the accuracy of classification results.

[0079] At the same time, the present invention dynamically classifies the collected motor imagery EEG signals without relying on prior knowledge of neuroscience, specific range filtering, and high spatial resolution, thereby improving the classification efficiency and automation level. The motor imagery brain-computer interface system is applicable in a variety of scenarios and can provide support for applications such as motor function restoration and military automation.

[0080] Fig. 9 An example of a physical structure diagram of an electronic device is shown in FIG. Fig. 9 As shown, the electronic device may include: a processor 910, a communication interface 920, a memory 930 and a communication bus 940, wherein the processor 910, the communication interface 920 and the memory 930 communicate with each other through the communication bus 940. The processor 910 may call the logic instructions in the memory 930 to execute the motor imagery EEG classification method based on feature fusion, the method comprising: inputting the preprocessed EEG signal into a first extraction module constructed based on a convolutional neural network to obtain a first feature map output by the first extraction module after local attention guidance; inputting the preprocessed EEG signal into a second extraction module constructed based on a long short-term memory network to obtain a second feature map output by the second extraction module after global self-attention guidance; and performing feature fusion on the first feature map and the second feature map and then classifying to obtain the classification result of the motor imagery EEG.

[0081] In addition, the logic instructions in the above-mentioned memory 930 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0082] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the motor imagery EEG classification method based on feature fusion provided by the above-mentioned methods, the method comprising: inputting the preprocessed EEG signal into a first extraction module constructed based on a convolutional neural network, and obtaining a first feature map output by the first extraction module after local attention guidance; inputting the preprocessed EEG signal into a second extraction module constructed based on a long short-term memory network, and obtaining a second feature map output by the second extraction module after global self-attention guidance; and classifying the first feature map and the second feature map after feature fusion to obtain the classification result of the motor imagery EEG.

[0083] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0084] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0085] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A motor imagery EEG classification method based on feature fusion, characterized in that: include: Inputting the preprocessed electroencephalogram signal into a first extraction module constructed based on a convolutional neural network to obtain a first feature map output by the first extraction module and guided by local attention; Inputting the preprocessed EEG signal into a second extraction module based on a long short-term memory network to obtain a second feature map output by the second extraction module guided by global self-attention; The first feature map and the second feature map are subjected to feature fusion and then classified to obtain a classification result of the motor imagery electroencephalogram.

2. The motor imagery EEG classification method based on feature fusion according to claim 1 is characterized in that: The step of inputting the preprocessed electroencephalogram signal into a first extraction module constructed based on a convolutional neural network to obtain a first feature map output by the first extraction module and guided by local attention specifically includes: After the preprocessed electroencephalogram signal is deeply expanded, multi-scale temporal feature extraction is performed through a multi-scale dense connection module in the first extraction module, and feature map splicing is performed in the depth dimension; The local mixed attention module in the first extraction module performs time and space mixed attention weighting on the spliced ​​feature map to obtain the first feature map.

3. The motor imagery EEG classification method based on feature fusion according to claim 2 is characterized in that: Before the step of extracting multi-scale temporal features through the multi-scale dense connection module in the first extraction module, the method further includes: A multi-scale dense connection module is constructed through five time-dimension convolution branches and one time-dimension average pooling branch.

4. The method for motor imagery EEG classification based on feature fusion according to claim 2, characterized in that: Before the step of weighting the spliced ​​feature map with time and space mixed attention by the local mixed attention module in the first extraction module, the method further includes: In the attention module of the channel dimension, the global maximum pooling is used instead of the global average pooling operation; In the attention module of the spatial dimension, global average pooling and global variance in the depth dimension are used to replace the compression operation; feature extraction submodules in the temporal and spatial dimensions are constructed to obtain axial attention; The local mixed attention module is constructed based on the attention module of the channel dimension and the attention module of the spatial dimension.

5. The motor imagery EEG classification method based on feature fusion according to claim 1, characterized in that: The step of inputting the preprocessed EEG signal into a second extraction module constructed based on a long short-term memory network to obtain a second feature map output by the second extraction module and guided by global self-attention specifically includes: Inputting the preprocessed EEG signal into the hidden layer of the long short-term memory network in the second extraction module; Inputting the feature map output by the hidden layer into the global self-attention module obtained by the second extraction module to obtain the spatiotemporal self-attention guided weighted feature map output by the global self-attention module; The output of the last global self-attention module is used as the second feature map.

6. The method for motor imagery EEG classification based on feature fusion according to claim 5, characterized in that: The step of inputting the feature map output by the hidden layer into the second extraction module to obtain a global self-attention module, and obtaining the spatiotemporal domain self-attention guided weighted feature map output by the global self-attention module specifically includes: Inputting the feature map output by the hidden layer into the second extraction module to obtain a global self-attention module; The global self-attention module calculates the spatial self-attention of the preprocessed EEG signal in the spatial domain, uses the spatial self-attention to weight the feature map output by the hidden layer, and then calculates the weighted feature map guided by the spatiotemporal self-attention based on the weighted result.

7. The method for motor imagery EEG classification based on feature fusion according to any one of claims 1 to 6, characterized in that: The step of fusing the first feature map and the second feature map and then classifying them to obtain the classification result of the motor imagery electroencephalogram specifically includes: aggregating and compressing the first feature map and the second feature map in the depth dimension and then extracting spatial features; The spatiotemporal weights and depth weights of the extracted spatial features are further calibrated, and the calibrated spatial features are classified to obtain the classification results of the motor imagery electroencephalogram.

8. A motor imagery EEG classification device based on feature fusion, characterized in that: include: A first extraction module, configured to receive the preprocessed EEG signal and output a first feature map guided by local attention; A second extraction module is used to receive the preprocessed EEG signal and output a second feature map guided by global self-attention; The classification module performs feature fusion on the first feature map and the second feature map and then classifies them to obtain a classification result of the motor imagery electroencephalogram.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the motor imagery EEG classification method based on feature fusion as described in any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the motor imagery EEG classification method based on feature fusion as described in any one of claims 1 to 7 is implemented.