A multi-branch graph adaptive network for personalized motor imagery electroencephalogram signal classification and method thereof

Abstract

The application provides a multi-branch graph adaptive network for individualized motor imagery electroencephalogram signal classification and a method thereof. Through an adaptive matching technology, a graph convolutional neural network (GCN) is used to learn a mapping relationship between an EEG signal and an optimal time-frequency domain processing method, and an optimal processing method is matched for each individual. The multi-branch network comprises an original data branch, a CWT branch and an STFT branch, and corresponding feature extraction networks are respectively designed. A data enhancement method based on a super-resolution generative adversarial network is used to expand a training data set and improve the generalization ability of the model. Experimental results show that the method significantly improves the classification accuracy of individualized EEG signals and has practical application value.

Description

[0001] * This research was supported by the Hangzhou Municipal Major Science and Technology Innovation Project "Research and Application of AI-native Relational Data Intelligence System" (2022AIZD0056). Technical Field

[0002] This invention relates to the fields of brain-computer interfaces and deep learning, specifically to a multi-branch graph adaptive network and method for classifying personalized motor imagery electroencephalogram (EEG) signals. Background Technology

[0003] Brain-computer interfaces (BCIs) are systems that enable communication between the human brain and external devices, playing a vital role in neuroscience and clinical research. Motor imagery (MI) is an important paradigm of BCI, allowing users to control devices without actual muscle contractions, simply by imagining motor intentions.

[0004] In recent years, deep learning has made significant progress in the field of MI EEG signal classification. Compared with traditional machine learning methods, deep learning not only improves classification accuracy but also reduces the labor cost of manual feature extraction. However, existing models still have shortcomings in terms of individualized classification accuracy, namely, the sensitivity to EEG signals varies greatly among different individuals. Since MI EEG signals are mainly used in medical rehabilitation and intelligent control, these applications require high accuracy in signal recognition for each individual. Therefore, improving the model's classification accuracy for individualized EEG signals is of great significance.

[0005] Traditional methods typically apply a uniform time-frequency domain signal processing approach to all individuals, failing to consider individual differences. Research indicates that different individuals are suited to different time-frequency domain processing methods, such as Short-Time Fourier Transform (STFT) or Continuous Wavelet Transform (CWT). Therefore, adaptively selecting the most suitable time-frequency domain processing method for each individual is crucial for improving the accuracy of individualized classification. Summary of the Invention

[0006] The purpose of this invention is to provide a multi-branch graph adaptive network and method for personalized motor imagery EEG signal classification. By introducing adaptive matching technology, the most suitable time-frequency domain processing method is matched to each individual's EEG signal based on spatial-temporal domain characteristics. A multi-branch network structure is used to improve classification accuracy, thereby meeting the demand for high-precision recognition of individualized signals in practical applications. To achieve the above objective, this invention provides the following technical solution: 1. A multi-branch graph adaptive network for personalized motor imagery EEG signal classification, comprising: an adaptive matching module: used to learn the mapping relationship between EEG signals and time-frequency domain processing methods based on the spatial-temporal domain characteristics of the input EEG signal using a graph convolutional neural network (GCN), matching the most suitable time-frequency domain processing method to each individual's EEG signal. A multi-branch network module: including an original data branch, a CWT branch, and an STFT branch, each branch corresponding to a time-frequency domain processing method, performing feature extraction and classification on the EEG signal processed by the adaptive matching module. Data Augmentation Module: Employs data augmentation methods based on super-resolution generative adversarial networks (SRGAN and ESRGAN) to expand the time-frequency map dataset of EEG signals and improve the generalization ability of deep learning models.

[0007] 2. The specific implementation method of the above adaptive matching module includes: treating each channel of the EEG signal as a node in a graph, constructing the connection relationship between nodes using the Pearson correlation coefficient, and forming a graph structure representation of the EEG signal. Using GCN, features are extracted from the graph-structured EEG signal to learn the mapping relationship between the EEG signal and the optimal time-frequency domain processing method, generating a classifier. Using the trained classifier, new EEG signals are predicted, and the most suitable time-frequency domain processing method is matched.

[0008] 3. The specific implementation methods of the above-mentioned multi-branch network modules include: Original data branch: For EEG signals that have not undergone time-frequency domain transformation, a dual-branch convolutional structure is used for feature extraction to fully capture the spatial and temporal features of the signal. CWT branch: For the time-frequency map of the CWT-processed EEG signal, a deep convolutional neural network with residual connections and attention mechanisms is used to extract high-level features and improve classification performance. STFT branch: For the STFT-processed EEG signal, a convolutional neural network and a long short-term memory network (LSTM) are combined to simultaneously extract spatial and temporal features and perform feature fusion.

[0009] 4. The specific implementation methods of the above data augmentation module include: using SRGAN and ESRGAN to perform super-resolution reconstruction of the time-frequency map of EEG signals to generate high-quality augmented data and expand the training dataset. Through data augmentation, the training effect and generalization ability of the model are improved, and overfitting is reduced.

[0010] The beneficial effects of this invention are as follows: A multi-branch graph adaptive network based on adaptive matching technology is proposed, which can match the most suitable time-frequency domain processing method for each individual's EEG signal according to individual differences, thereby improving the accuracy of individualized classification. A multi-branch network structure is adopted, and corresponding feature extraction networks are designed for different time-frequency domain processing methods, fully exploiting the features of EEG signals and improving the model's classification performance. A data augmentation method based on super-resolution generative adversarial networks is introduced to expand the training dataset, improve the model's generalization ability, and reduce overfitting. Experimental results show that the method of this invention achieves excellent classification performance on multiple public datasets and has good practical application value. Attached Figure Description

[0011] Figure 1 This is a schematic diagram of the overall structure of the multi-branch graph adaptive network of the present invention.

[0012] Figure 2 This is a flowchart illustrating the data enhancement module of the present invention. Detailed Implementation

[0013] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. The present invention proposes a multi-branch graph adaptive network (MBGA-Net) for personalized motor imagery EEG signal classification, aiming to improve the accuracy of EEG signal classification for different individuals.

[0014] like Figure 1 As shown, the overall architecture of MBGA-Net includes four main modules: adaptive matching module, raw data branch, CWT branch, and STFT branch.

[0015] I. Adaptive Matching Module: The core of the adaptive matching module is to use a graph convolutional neural network (GCN) to learn the mapping relationship between the EEG signal and the most suitable time-frequency domain processing method.

[0016] 1. Graph Structure Construction: First, each channel of the EEG signal is considered as a node in a graph. The correlation between channels is calculated using the Pearson correlation coefficient. When the correlation coefficient between two channels exceeds a set threshold (e.g., 0.93), they are considered to be connected, thus constructing a graph structure representation of the EEG signal.

[0017] 2. Feature Extraction and Mapping Relationship Learning: A Geometric Network (GCN) is used to extract features from the graph structure described above. The GCN captures the spatial features of the EEG signal by aggregating the feature information of each node and its neighboring nodes. The GCN model is trained on a large amount of EEG data from different individuals to learn the mapping relationship between the EEG signal and the optimal time-frequency domain processing method (raw signal, CWT, or STFT), thus generating a classifier.

[0018] 3. Matching of New Signal Processing Methods: For new EEG signals, they are first converted into a graph structure representation and input into a trained GCN classifier. Based on the characteristics of the input signal, the classifier predicts the most suitable time-frequency domain processing method, providing a basis for subsequent feature extraction and classification.

[0019] II. Multi-branch network module: The multi-branch network module includes the original data branch, CWT branch and STFT branch. Each branch is designed with a corresponding feature extraction network for a time-frequency domain processing method to fully explore the features of EEG signals and improve classification performance.

[0020] 1. Original Data Branch

[0021] Structural Design: For EEG signals that have not undergone time-to-frequency domain conversion, a two-branch convolutional neural network structure is adopted. The left branch mainly extracts spatial features in the channel dimension, while the right branch extracts temporal features in the time dimension.

[0022] Feature extraction: The left branch first performs channel-dimensional convolution on the signal, then performs temporal convolution to capture spatial and temporal features. The right branch first performs temporal convolution, then uses pooling layers to compress the feature size to avoid overfitting.

[0023] Feature fusion and classification: The features from the left and right branches are fused along the channel dimension to form a comprehensive feature vector. Finally, the EEG signal is classified using a fully connected layer and a Softmax classifier.

[0024] 2. CWT branch

[0025] Targeted design: A deep convolutional neural network was designed for the time-frequency plot of the EEG signal after continuous wavelet transform (CWT) processing, making full use of the two-dimensional characteristics of the time-frequency plot.

[0026] Residual connections: Introducing residual connections alleviates the vanishing gradient problem in deep network training and enhances the network's learning ability.

[0027] Attention mechanism: By integrating channel attention and spatial attention mechanisms, the network's attention to important features is enhanced, thereby improving the effectiveness of feature extraction.

[0028] The classification process involves multiple convolutional layers, residual connections, and attention mechanisms to obtain high-level feature representations. Finally, classification is performed using fully connected layers and a Softmax classifier.

[0029] 3. STFT branch

[0030] Fusion of Convolutional and Recurrent Networks: For EEG signals processed by Short Time Fourier Transform (STFT), a convolutional neural network and a long short-term memory network (LSTM) are combined to extract both spatial and temporal features.

[0031] Feature extraction: First, a convolutional network is used to extract local features in the time and frequency dimensions. Then, an LSTM network is used to capture the long-term and short-term dependencies of the signal and extract temporal features.

[0032] Feature fusion and classification: Features extracted by convolutional networks and LSTM networks are fused to form a comprehensive feature vector. Finally, classification is achieved through fully connected layers and a Softmax classifier.

[0033] III. Data Augmentation Module: such as Figure 2 As shown, in order to address the problem of limited EEG data and improve the generalization ability of the model, a data augmentation method based on super-resolution generative adversarial networks (SRGAN and ESRGAN) was adopted.

[0034] 1. Generator Network Design: Two image generators based on SRGAN and ESRGAN are constructed respectively to perform super-resolution reconstruction of low-resolution time-frequency maps and generate high-quality augmented data.

[0035] 2. Discriminator Network Design: Design a discriminator network and train it adversarially against the generator network. The goal of the discriminator is to distinguish between generated high-resolution images and real high-resolution images.

[0036] 3. Adversarial Training and Data Generation: Through adversarial training, when the discriminator can no longer distinguish between generated and real images, the generator is considered successfully trained. At this point, the generated high-resolution time-frequency map can be added to the training set as new data samples.

[0037] 4. Dataset Expansion: The original dataset is expanded to three times its original size using augmented data generated by SRGAN and ESRGAN respectively. Richer training data helps the model learn more robust features, improving classification performance.

[0038] IV. Summary of Implementation Methods: This invention utilizes adaptive matching technology to automatically select the most suitable time-frequency domain processing method for the EEG signals of different individuals, thus addressing the problem of insufficient consideration of individual differences in existing methods. The multi-branch network structure design employs the optimal feature extraction network for different data types, improving classification accuracy. Simultaneously, the introduction of a data augmentation module solves the problem of insufficient EEG data volume. By generating high-quality augmented data, the model can learn richer features, improving generalization ability and reducing overfitting. The multi-branch graph adaptive network of this invention can be widely applied in fields requiring high-precision individualized EEG signal recognition, such as medical rehabilitation and intelligent control. By improving the classification accuracy of individual EEG signals, the reliability and user experience of brain-computer interface systems can be enhanced, promoting the development and application of related technologies.

Claims

1. A multi-branch graph adaptive network and method for classifying personalized motor imagery EEG signals, characterized in that... The network comprises an adaptive matching module, a multi-branch network module, and a data augmentation module. The adaptive matching module utilizes a graph convolutional neural network (GCN) to learn the mapping relationship between EEG signals and the optimal time-frequency domain processing method based on the spatial-temporal characteristics of the EEG signals, so as to adaptively match the most suitable time-frequency domain processing method for each individual. The multi-branch network module includes a raw data branch, a CWT branch, and an STFT branch. Each branch performs feature extraction for a time-frequency domain processing method to improve classification accuracy. The data augmentation module performs super-resolution reconstruction of the time-frequency map based on super-resolution generative adversarial networks (SRGAN and ESRGAN) to expand the EEG signal dataset and enhance the model's generalization ability.

2. The multi-branch graph adaptive network and method for personalized motor imagery EEG signal classification according to claim 1, characterized in that... The adaptive matching module treats each channel of the EEG signal as a node in a graph, uses the Pearson correlation coefficient to construct the connection relationship between the nodes, forms a graph structure representation of the EEG signal, and learns the mapping relationship of the optimal time-frequency domain processing method through GCN.

3. A multi-branch graph adaptive network and method for personalized motor imagery EEG signal classification according to claim 1 or 2, characterized in that... The multi-branch network module includes raw data branches for EEG signals that have not undergone time-frequency domain transformation, and uses a two-branch convolutional neural network structure to extract spatial and temporal features; The CWT branch extracts high-level features from the time-frequency plot of EEG signals processed by continuous wavelet transform using a deep convolutional network with residual connections and attention mechanisms; the STFT branch extracts spatial and temporal features from EEG signals processed by short-time Fourier transform by combining convolutional neural networks and long short-term memory networks (LSTM).

4. The multi-branch graph adaptive network and method for personalized motor imagery EEG signal classification according to claim 3, characterized in that... The data augmentation module uses SRGAN and ESRGAN generative adversarial networks to perform super-resolution reconstruction of time-frequency maps, thereby expanding the training dataset by generating high-resolution images and improving the model's generalization ability.

5. A multi-branch graph adaptive network and method for personalized motor imagery EEG signal classification according to any one of claims 1 to 4, characterized in that... The adaptive matching module trains a GCN classifier to predict new EEG signals and selects the most suitable time-frequency domain processing method to achieve personalized, high-precision signal classification.

6. A multi-branch graph adaptive network and method for classifying personalized motor imagery EEG signals according to any one of claims 1 to 5, characterized in that... The data augmentation module expands the original dataset to three times its original size by generating a high-resolution time-frequency plot, thereby improving the model's generalization ability on new data and reducing overfitting.

Images