A multi-branch motor imagery electroencephalogram signal feature fusion classification method based on data conversion
By using a multi-branch network structure and data transformation method, the problem of insufficient depth in deep learning networks for motor imagery EEG was solved, achieving higher classification accuracy and comprehensive feature extraction, reducing overfitting, and improving the effect of EEG signal classification.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- UNIV OF ELECTRONICS SCI & TECH OF CHINA
- Filing Date
- 2023-03-15
- Publication Date
- 2026-05-05
AI Technical Summary
Existing deep learning networks for motor imagery EEG are relatively shallow, leading to frequent overfitting, insufficient feature extraction, and inadequate classification accuracy.
A multi-branch network structure is adopted to convert EEG signals into different input formats, such as time-frequency images and Gram angle fields, through data transformation. Feature extraction is performed by combining depthwise separable convolution and VGG-13 network, and a multi-task constrained loss function is used for training to enhance the comprehensiveness and robustness of feature extraction.
It improves the classification accuracy of motor imagery EEG signals, enhances the robustness and comprehensiveness of feature extraction, reduces overfitting, and improves classification performance.
Smart Images
Figure CN116250846B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to brainwave signal feature extraction and classification technology, and particularly to data conversion and multi-branch feature fusion and classification technology in motor imagery brainwave classification. Background Technology
[0002] Brain-computer interfaces (BCIs) offer a new way for human-computer interaction by analyzing electrical signals generated by the brain and converting them into actual commands. Motor imagery EEG signals are widely used in BCI research, which helps control external devices. By correctly decoding motor imagery-related EEG signals, patients with motor disorders can correctly control exoskeletons, wheelchairs, and other devices; this can also be applied to the control of external robots and intelligent vehicles. Accurately decoding motor imagery EEG signals and improving the classification accuracy of these signals is significant.
[0003] EEG signals are affected by external noise and internal electromyographic noise, resulting in a very low signal-to-noise ratio. Accurate classification of EEG signals is a crucial component of Brain-Computed Interference (BCI) technology. Traditional methods primarily focus on processing signals in the time, frequency, and spatial domains. In the time domain, they rely on discovering the statistical properties of the signal waveform; in the frequency domain, they emphasize classifying the spectral characteristics of the signal. Combining time and frequency domain methods, techniques such as Local Feature Scale Decomposition (LCD), Discrete Wavelet Transform (DWT), and Flexible Analytical Wavelet Transform (FAWT) are employed. Spatial domain analysis utilizes methods such as Common Spatial Pattern (CSP) and Filter Bank Common Spatial Pattern (FBCSP). The main machine learning algorithms used include Linear Discriminant Analysis (LDA) and Support Vector Machines (SVM). However, these feature extraction and classification methods are discontinuous and require manual feature selection, necessitating prior knowledge. Therefore, the extracted features are often incomplete, resulting in a large workload and inaccurate classification.
[0004] With the development of deep learning, many researchers in the field of brainwave imaging (BCI) have been inspired to apply various deep learning methods to the field of EEG. Deep learning networks can adaptively select and extract features, reducing reliance on prior knowledge and manual screening. Currently, various neural network architectures are available for EEG feature extraction. These include recurrent neural networks such as EEGNet using deep separable convolutions, LSTM for processing time series data, and methods that convert motion-imagined EEG signals into spectral images and feed them into CNN networks for processing.
[0005] Due to the low signal-to-noise ratio and limited quantity of EEG data, overfitting is prone to occur when training deep learning networks, leading to the frequent use of shallow neural networks. Furthermore, different EEG feature extraction networks use different input formats, resulting in features that are biased towards a single direction. Currently, motor imagery EEG classification tasks typically categorize tasks involving multiple limbs, commonly including left hand, right hand, tongue, and both feet, or classify multiple different tasks involving a single limb, such as clenching a fist versus opening the palm, or swinging the palm left and right. Summary of the Invention
[0006] The technical problem to be solved by this invention is to provide a method for classifying motor imagery EEG signal features by adding network branches and network target tasks based on data transformation, which is a problem due to the shallow depth of existing deep learning networks for motor imagery EEG. By expanding the network structure in terms of width, it can combat overfitting to a certain extent and improve accuracy.
[0007] The technical solution adopted by this invention to solve the above-mentioned technical problems is a multi-branch motor imagery EEG signal feature fusion and classification method based on data conversion, comprising the following steps:
[0008] Preprocessing steps: The EEG signal is preprocessed to obtain a multi-channel EEG time sequence signal. The first, second and third branch processing steps are performed simultaneously in the three branch networks of the multi-channel EEG time sequence signal.
[0009] First branch processing step: Perform time-based depthwise separable convolution on the EEG temporal signals of each channel to generate the first branch feature spectrum, and then proceed to the fusion step;
[0010] The second branch processing steps are as follows: Wavelet transform is performed on the EEG time-series signals of each channel to obtain time-frequency images. The time-frequency images are superimposed as a multi-channel two-dimensional image. This two-dimensional image is fed into a VGG-13-based convolutional coding network to generate the second branch feature spectrum. Simultaneously with the fusion step, the second branch feature spectrum is input into a VGG-13 deconvolution-based convolutional decoding network that is symmetrical to the convolutional coding network to generate the corresponding time-frequency image. Then, the loss between the time-frequency image generated by the convolutional decoding network and the time-frequency image input to the convolutional coding network is calculated to constrain the training process of the second branch processing of the motor imagery EEG signal feature classification network.
[0011] The third branch processing step: construct Gram angle fields for the EEG time-series signals of each channel, superimpose the obtained Gram angle fields of each channel into a multi-channel two-dimensional image, and feed the two-dimensional image into a VGG-13-based convolutional coding network to generate the third branch feature spectrum. At the same time as entering the fusion step, the third branch feature spectrum is input into a VGG-13 deconvolution-based convolutional decoding network that is symmetrical to the convolutional coding network to generate the corresponding Gram angle field. Then, calculate the loss between the Gram angle field generated by the convolutional decoding network and the Gram angle field input to the convolutional coding network to constrain the training process of the third branch processing of the motor imagery EEG signal feature classification network.
[0012] Fusion steps: The generated first branch feature spectrum, second branch feature spectrum and third branch feature spectrum are respectively fed into the channel attention module to generate the first branch attention heatmap, the second branch attention heatmap and the third branch attention heatmap. Then, the attention heatmaps of the three branches are multiplied with the feature spectrum of their respective branches to obtain new feature spectra of the three branches. After the new feature spectra of the three branches are connected, they are simultaneously entered into the fine classification step and the large classification step.
[0013] Fine classification steps: After the feature spectrum after connection processing is fed into two fully connected layers and a Softmax layer for fine classification, the Softmax layer outputs the subdivided action class and calculates the fine classification loss to constrain the fine classification training process of the motor imagery EEG signal feature classification network.
[0014] Large classification step: After the feature spectrum after connection processing is fed into two fully connected layers and a Softmax layer for large classification, the Softmax layer outputs the classification results of two classes, either left-hand action class or right-hand action class, and calculates the large classification loss to constrain the large classification training process of the motor imagery EEG signal feature classification network.
[0015] Test steps: The motor imagery EEG signal feature classification network obtained after training is used for motor imagery EEG signal classification.
[0016] The objective of motor imagery EEG signal classification tasks is often to output the correct corresponding category. Adding task objectives helps deep learning networks extract more features, and features from different tasks can complement each other, improving classification performance. This invention aims to expand the network structure width, i.e., network branches, by converting EEG data into different input formats and using multiple branch networks for processing. This improves the completeness of feature extraction, ensuring that different network branches have different salient features, and the extracted features are complementary. Converting to different data formats facilitates training the network to learn different types of features. It also utilizes constraints from large and small tasks in classification tasks, as well as other tasks, i.e., using different task objectives of the network to achieve multiple constraints. Setting different task objectives when extracting features from EEG data ensures that the extracted features are not limited to the classification task, compensating for the shortcomings of insufficient feature extraction in classification tasks, and enabling the network to extract more general and comprehensive features.
[0017] The beneficial effects of this invention are that by using Gram angle field as the new transformed data format input to the network, it provides richer features compared to depthwise separable convolution and time-frequency maps, increasing the robustness of the extracted features; at the same time, it uses loss functions for extracting and reconstructing features from different data formats, as well as loss functions for classifying large and subdivided tasks simultaneously in specific motion classification tasks, achieving better classification results. Attached Figure Description
[0018] Figure 1 This is a flowchart of an example implementation. Detailed Implementation
[0019] When studying various tasks involving different limbs, classification is typically done only at the subdivided level. The self-test dataset used in this example includes six movements for both the left and right hands. These six movements are further divided into two main categories (left and right hands) and six subcategories, such as left hand clenching and relaxing, right hand clenching and relaxing, and left hand palm swinging left and right. The loss function used in this paper includes cross-entropy loss to constrain the classification categories for both the main and subcategories. The main classification loss also includes center loss using latent features of the main category. The loss function for image (Gram angle field, time-frequency map) reconstruction is the MSE loss.
[0020] The example uses three branches to extract EEG features.
[0021] The first branch uses a deep separable convolutional network based on EEGNet to directly extract features from the original input EEG signal;
[0022] The second branch uses wavelet transform to convert EEG signals into time-frequency domain images and uses a VGG-13-based network for feature extraction.
[0023] The third branch converts the EEG signals into Gram-angle field (GAF) images and uses a VGG-13-based network for feature extraction.
[0024] After converting EEG signals into different types of data, feature extraction was performed, and the original input was processed directly, allowing the network to learn from a more comprehensive signal representation.
[0025] The first branch uses EEGNet's depthwise separable convolution mode for feature extraction. First, it convolves the temporal signals of each channel simultaneously to obtain temporal information for each channel. Then, it uses depthwise convolution to connect to each feature map individually. Finally, it convolves all feature maps to fuse the information. The extracted feature spectrum at this point can be considered as a shallow representation of the EEG signal. This feature spectrum then needs to be processed by a channel attention module to generate an attention heatmap. Multiplying this attention heatmap by the feature spectrum yields a new feature spectrum.
[0026] The second branch uses wavelet transform to convert the EEG signal into a time-frequency domain image (WT). x This means performing wavelet transform on each channel of the input EEG signal; continuous wavelet transform is:
[0027]
[0028] Where a is the scale factor, τ reflects the displacement, and x(t) is the input EEG signal. For the basic wavelet function, This represents the complex conjugate of the basic wavelet function.
[0029] Decomposition was performed using discrete wavelet transform:
[0030]
[0031] Where h(n) and g(n) are a pair of complementary conjugate filters determined by the wavelet function, h(n) is a low-pass filter, g(n) is a high-pass filter, j is the scale of the wavelet decomposition, k is the step size of the wavelet filter, n is the position of the data in the sequence, Z is the kernel size of the wavelet filter, and C j and D j It represents the approximate and detailed portions of EEG data at scale j.
[0032] After performing wavelet transform on the EEG signals of m channels, m two-dimensional time-frequency images are obtained. The 3-40Hz portion of the corresponding image is extracted and the image size is readjusted to 224*224. Then, the m images are superimposed to form a 224*224*m image.
[0033] This image represents the data in a time-frequency domain combined form, compared to the original EEG signal format, and allows the use of an image feature extraction network to extract time-frequency domain features from the EEG signal.
[0034] The image is fed into a VGG-13-based convolutional coding network to obtain the corresponding feature spectrum, which is then passed through a channel attention module to obtain a new feature spectrum. Simultaneously, the original feature spectrum needs to be fed into a deconvolutional network that is nearly symmetrical to the convolutional coding network to generate the original time-frequency domain image. This constraint ensures that the intermediate generated feature spectrum comprehensively includes the important features of the time-frequency domain image.
[0035] The third branch converts the EEG signals into Gram angle GAF, x i It is the i-th value in the EEG signal of a certain channel. First, the EEG signal is normalized to the interval [-1,1] to obtain x. i Corresponding standard value Convert the normalized value to polar coordinates φ i :
[0036]
[0037] Find the Gram angle field corresponding to each channel:
[0038]
[0039] Where T represents the total number of data points in a single channel. After calculating the Gram angle field of the EEG signals from m channels, m two-dimensional images are obtained. The image size is readjusted to 224*224, and then the m images are superimposed as a 224*224*m image.
[0040] The advantage of this image is that it preserves the temporal and spatial information in the original EEG channel signals, and these features can be extracted using image convolution.
[0041] The image is fed into a VGG-13-based convolutional coding network to obtain the corresponding feature spectrum, which is then passed through a channel attention module to obtain a new feature spectrum. Simultaneously, the original feature spectrum is fed into a nearly symmetrical deconvolutional network to generate the original Gram corner field image. This constraint ensures that the intermediate generated feature spectrum comprehensively includes the important features of the Gram corner field image.
[0042] The new feature spectra obtained from the three branches are connected and then passed through two fully connected layers and a softmax layer for classification. The first group is classified into left and right hand action categories, and the second group is classified into more refined categories.
[0043] The implementation examples are built on the PyTorch framework, such as... Figure 1 The main steps include: performing depthwise separable convolution to extract the feature spectrum, performing wavelet transform to convert to a time-frequency graph and extracting the feature spectrum, performing Gram angle field transform and extracting the feature spectrum, fusing the feature spectra for classification tasks, and using a decoding network to decode and generate the original input signal from the feature spectrum.
[0044] The training process for the motor imagery EEG signal feature classification network involves the following steps:
[0045] (1) Input the preprocessed EEG signal into the three branch networks and perform steps (2), (3) and (4) simultaneously.
[0046] (2) Perform time-sequential depth-separable convolution on each time channel of the EEG signal to generate feature spectra, and proceed to step (5).
[0047] (3) Perform wavelet transform on each EEG time-series channel to obtain a time-frequency image. Superimpose the time-frequency images to form a two-dimensional image with m channels. Input the image into a VGG-13-based convolutional coding network to generate a feature spectrum. Simultaneously with step (5), input the feature spectrum into a VGG-13-based deconvolutional convolutional decoding network that is symmetrical to the convolutional coding network to generate the corresponding time-frequency image. Then, calculate the loss function between the time-frequency image generated by the convolutional decoding network and the time-frequency image input to the convolutional coding network. The loss function is the MSE loss function.
[0048]
[0049] Where u i and f(v) i ) represent the real value and generated value of the i-th pixel in the image, respectively, and o is the number of pixels.
[0050] (4) Construct Gram angle fields for each channel of EEG time sequence, and superimpose the obtained Gram angle fields of each channel as a two-dimensional image of m channels. Input the image into a VGG-13-based convolutional coding network to generate a feature spectrum. At the same time as entering step (5), input the feature spectrum into a VGG-13-based deconvolutional convolutional decoding network that is symmetrical with the convolutional coding network to generate the corresponding Gram angle fields. Then, calculate the loss function between the Gram angle fields generated by the convolutional decoding network and the Gram angle fields input to the convolutional coding network.
[0051] (5) The generated feature spectrum is sent to the channel attention module to generate an attention heatmap. The attention heatmap is multiplied with the feature spectrum to obtain a new feature spectrum. The new feature spectra of all branches are connected and then input to steps (6) and (7) respectively.
[0052] (6) After the feature spectrum after concatenation processing is fed into two fully connected layers and a Softmax layer for fine classification, the Softmax layer outputs the finely divided action classes, such as left hand palm moving up and down, right hand palm moving up and down, left hand clenching and relaxing, right hand clenching and relaxing, left hand palm swinging left and right through the wrist, and right hand palm swinging left and right through the wrist. The fine classification loss function is then calculated; this loss function is the cross-entropy loss function.
[0053]
[0054] Where S represents the number of subcategories of movement, c represents the specific subcategories of movement, i represents the i-th EEG data point, and p ic y represents the probability that the i-th EEG data point is predicted to be category c. ic The sign function is set to 1 when the actual category of the i-th EEG data is c, and 0 otherwise.
[0055] (7) After the concatenated feature spectrum is fed into two fully connected layers and a Softmax layer for large classification, the Softmax layer outputs the classification results of two classes, either left-hand action class or right-hand action class, and calculates the large classification loss function. The input features also need to be processed through a fully connected layer to calculate the center loss function.
[0056]
[0057] Where b is the batch size during network training, i is the i-th training data in the same batch, and fv i Let ci be the high-level feature vector of the i-th data after passing through a fully connected layer, and cv be the true class of the i-th input data. ei This represents the feature center vector of the true category of the i-th data.
[0058] (8) The training process is constrained by different loss functions for four tasks. The weights of different loss functions on the overall loss function are set differently according to different training tasks. The motor imagery EEG signal feature classification network obtained after training is used for motor imagery EEG signal classification.
Claims
1. A multi-branch motor imagery EEG signal feature fusion and classification method based on data transformation, characterized in that, Includes the following steps: Preprocessing steps: The EEG signal is preprocessed to obtain a multi-channel EEG time sequence signal. The first, second and third branch processing steps are performed simultaneously in the three branch networks of the multi-channel EEG time sequence signal. First branch processing step: Perform time-based depthwise separable convolution on the EEG temporal signals of each channel to generate the first branch feature spectrum, and then proceed to the fusion step; The second branch processing steps are as follows: Wavelet transform is performed on the EEG time-series signals of each channel to obtain time-frequency images. The time-frequency images are superimposed as a multi-channel two-dimensional image. This two-dimensional image is fed into a VGG-13-based convolutional coding network to generate the second branch feature spectrum. Simultaneously with the fusion step, the second branch feature spectrum is input into a VGG-13 deconvolution-based convolutional decoding network that is symmetrical to the convolutional coding network to generate the corresponding time-frequency image. Then, the loss between the time-frequency image generated by the convolutional decoding network and the time-frequency image input to the convolutional coding network is calculated to constrain the training process of the second branch processing of the motor imagery EEG signal feature classification network. The third branch processing step: construct Gram angle fields for the EEG time-series signals of each channel, superimpose the obtained Gram angle fields of each channel into a multi-channel two-dimensional image, and feed the two-dimensional image into a VGG-13-based convolutional coding network to generate the third branch feature spectrum. At the same time as entering the fusion step, the third branch feature spectrum is input into a VGG-13 deconvolution-based convolutional decoding network that is symmetrical to the convolutional coding network to generate the corresponding Gram angle field. Then, calculate the loss between the Gram angle field generated by the convolutional decoding network and the Gram angle field input to the convolutional coding network to constrain the training process of the third branch processing of the motor imagery EEG signal feature classification network. Fusion steps: The generated first branch feature spectrum, second branch feature spectrum and third branch feature spectrum are respectively fed into the channel attention module to generate the first branch attention heatmap, the second branch attention heatmap and the third branch attention heatmap. Then, the attention heatmaps of the three branches are multiplied with the feature spectrum of their respective branches to obtain new feature spectra of the three branches. After the new feature spectra of the three branches are connected, they are simultaneously entered into the fine classification step and the large classification step. Fine classification steps: After the feature spectrum after connection processing is fed into two fully connected layers and a Softmax layer for fine classification, the Softmax layer outputs the subdivided action class and calculates the fine classification loss to constrain the fine classification training process of the motor imagery EEG signal feature classification network. Large classification step: After the feature spectrum after connection processing is fed into two fully connected layers and a Softmax layer for large classification, the Softmax layer outputs the classification results of two classes, either left-hand action class or right-hand action class, and calculates the large classification loss to constrain the large classification training process of the motor imagery EEG signal feature classification network. Test steps: The motor imagery EEG signal feature classification network obtained after training is used for motor imagery EEG signal classification.