Electroencephalogram data augmentation method and system based on generative adversarial network model

Abstract

The application discloses an electroencephalogram data amplification method and system based on a generative adversarial network model, to solve the problems of existing electroencephalogram data in motor imagery classification based on a deep neural network and meet the requirement of improving the generalization ability of the motor imagery classification model. The method introduces the generative adversarial network model into the electroencephalogram data amplification problem, generates a new electroencephalogram data set with higher accuracy by reasonably designing a loss function and using an experimental data set and a standard data set to have the data characteristics of both, and reduces the working time of experimenters and experimental costs; the generator of the generative adversarial network model is a kind of encoder-decoder based on dilated convolution, which can extract electroencephalogram data features at different levels; and the discriminator of the generative adversarial network model is a kind of discriminator based on a parallel feature extraction structure of different dimensions, which can extract features of different properties.

Description

Technical Field

[0001] This invention relates to the field of electroencephalogram (EEG) data processing technology, and more specifically, to an EEG data augmentation method and system based on a generative adversarial network (GAN) model. Background Technology

[0002] Currently, the application of brain-computer interface (BCI) systems based on motor imagery is mainly in the medical field, providing a new control method for patients with movement disorders, helping them achieve self-care in daily life and even complete rehabilitation. These patients with movement disorders include those with brainstem injury, stroke, muscle atrophy, etc. They have intact brains but are unable to control their bodies due to damage to peripheral nerves. BCI systems based on motor imagery can enable control of external devices, replacing trunk movements. In addition, the combination of motor imagery and physical therapy is an effective method for stroke rehabilitation.

[0003] Deep Neural Networks (DNNs), as a subclass of machine learning, have made remarkable progress in computer vision and natural language processing. However, due to the unique characteristics of electroencephalogram (EEG) signals, research on DNNs in motor imagery classification based on EEG is limited. On the one hand, EEG data is typically limited, and since DNNs usually require more training data than other machine learning methods, they may encounter severe overfitting problems. On the other hand, EEG data exhibits significant subject variability; directly training DNNs with EEG data from multiple subjects results in poor generalization ability for new subjects.

[0004] Previous methods for classifying motor imagery based on DNNs typically required collecting a large number of samples from each subject, which was often too time-consuming and increased experimental costs. Although Generative Adversarial Networks (GANs) have been widely used in the field of data generation, due to the unique characteristics of EEG data, developing effective methods for EEG data augmentation based on GAN models remains a pressing technical problem. Summary of the Invention

[0005] The purpose of this invention is to address the problems existing in the classification of motor imagery based on DNN using existing EEG data and to meet the requirements for improving the generalization ability of motor imagery classification models, by providing a method and system for augmenting EEG data based on a generative adversarial network model.

[0006] To achieve the above objectives, on the one hand, the present invention provides a method for augmenting electroencephalogram (EEG) data based on a generative adversarial network (GAN) model, characterized by comprising the following steps:

[0007] Step S1: Collect EEG data from the subjects, generate an experimental dataset, preprocess the experimental dataset, and divide the experimental dataset into a training set and a test set according to the proportion of the standard dataset.

[0008] Step S2: Perform wavelet transform on the experimental dataset and the standard dataset to generate time-frequency diagrams of the experimental dataset and the standard dataset;

[0009] Step S3: Based on the requirements for amplification of EEG data, design a generative adversarial network model for EEG data amplification and design the corresponding loss function;

[0010] Step S4: Design the generator and discriminator of the generative adversarial network model based on the characteristics of the EEG data;

[0011] Step S5: Using the designed generative adversarial network model, the features of the experimental dataset and the standard dataset are fused to generate a new dataset;

[0012] Step S6: Use the standard dataset, experimental dataset, and new dataset generated by the generative adversarial network model to train the classic deep neural network model for motion image classification.

[0013] Step S7: Use the trained classic deep neural network model to classify motor imagery and analyze whether the new dataset improves the accuracy of the EEG-based motor imagery classification model. If the accuracy does not meet the data application requirements, return to step S3 and continue to improve the model design until it meets the data application requirements.

[0014] Preferably, in step S3, the generative adversarial network model is a cycleGAN model; the loss function includes three types of loss: GAN loss L GAN cycle loss L cyc and identify loss L id It can be described by the following formula:

[0015] L total (G,F,D X D Y )=-L GAN (G,F,D X D Y )+λ cyc L cycle (G,F)+λ id L identify (G,F);

[0016] Among them, GAN loss L GAN The formula is as follows:

[0017]

[0018] cycle loss L cyc The formula is as follows:

[0019]

[0020] identify loss L id The formula is as follows:

[0021] In this feature, the introduction of the identify loss can prevent the target domain image from undergoing unnecessary distortion due to GAN loss.

[0022] Furthermore, in step S4, the generator comprises two generators, which together employ a residual structure. The feature extraction part uses a deencoder design and extracts more features through dilated convolutions with different coefficients. This feature allows the generator to extract EEG data features at different levels, thereby improving the generator's generation capability.

[0023] Furthermore, modules with an expansion coefficient of 1 change accordingly when the expansion coefficient of the deencoder changes, while modules with an expansion coefficient of 0 do not change accordingly. This feature fully integrates the characteristics of EEG signals from different feature extraction stages, enabling the generator to generate more universal EEG signals.

[0024] Preferably, in step S4, the discriminator structure includes a two-dimensional convolutional kernel, a one-dimensional convolutional kernel spanning frequency channels, and a one-dimensional convolutional kernel spanning time. This feature overcomes the problem that a single two-dimensional convolutional kernel is insufficient to capture features from an EEG representation image.

[0025] On the other hand, the present invention provides an EEG data augmentation system based on a generative adversarial network model, characterized in that, for implementing the EEG data augmentation method described above, it includes:

[0026] The acquisition module is used to acquire experimental datasets of electroencephalograms (EEGs).

[0027] The data augmentation module is used to augment the experimental dataset using a pre-designed generative adversarial network model to obtain a new dataset.

[0028] A training module is used to train the new dataset using a deep neural network model;

[0029] The application module uses a trained deep neural network model to classify motor imagery and analyzes whether the new dataset improves the accuracy of the EEG-based motor imagery classification model.

[0030] In another aspect, the present invention provides a computer device including a memory and a processor, wherein the processor and the memory communicate with each other, and the memory stores program instructions that can be executed by the processor, characterized in that the processor calls the program instructions to execute the electroencephalogram data amplification method as described above.

[0031] Finally, the present invention provides a computer-readable storage medium, characterized in that it stores a computer program, which, when executed by a processor, implements the electroencephalogram data amplification method as described above.

[0032] Compared with the prior art, the above invention has the following advantages or beneficial effects:

[0033] (1) Introducing the GAN model into the EEG data augmentation problem, by reasonably designing the loss function, a new EEG dataset with higher accuracy with the data characteristics of both experimental datasets and standard datasets obtained in the laboratory was generated, which reduced the working time and experimental cost of the experimenters.

[0034] (2) The generator of the GAN model is an encoder-decoder based on dilated convolution. The generator can extract EEG data features at different levels, thereby improving the generator's generation capability.

[0035] (3) The discriminator of the GAN model is a discriminator based on a parallel feature extraction structure of different dimensions. The discriminator can extract features of different properties, thereby improving the discriminator's discrimination ability. Attached Figure Description

[0036] The invention, its features, shape, and advantages will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings. Like reference numerals denote like parts throughout the drawings. The drawings are not drawn to scale; their focus is on illustrating the gist of the invention.

[0037] Figure 1 This is a flowchart of an EEG data augmentation method based on a GAN model in one embodiment of the present invention;

[0038] Figure 2 This is a structural diagram of a GAN model according to an embodiment of the present invention;

[0039] Figure 3 This is a generator structure diagram of a GAN model in one embodiment of the present invention;

[0040] Figure 4This is a diagram illustrating the generator structure, encoder, and residual block structure of a GAN model in one embodiment of the present invention.

[0041] Figure 5 This is an explanatory diagram of the three convolutional kernels of the GAN model in one embodiment of the present invention;

[0042] Figure 6 This is a diagram of the discriminator structure of a GAN model in one embodiment of the present invention;

[0043] Figure 7 This is a schematic diagram illustrating the generation of new data using a trained generator in one embodiment of the present invention. Detailed Implementation

[0044] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. It should be understood that all of these exemplary embodiments described are merely some embodiments and examples of the present invention, and not all of them. Rather, these exemplary embodiments are provided so that those skilled in the art can more thoroughly understand the present disclosure and to more completely convey the technical content of the present disclosure to those skilled in the art.

[0045] Example 1

[0046] See Figure 1 To address the serious overfitting and individual variability problems inherent in existing DNN methods for EEG-based motion image classification, this embodiment provides a GAN-based EEG data augmentation method, comprising the following steps:

[0047] Step S1: Collect EEG data from the subjects to generate an experimental dataset. Preprocess the experimental dataset by dividing it into a training set and a test set according to the proportions of the standard dataset. As a preferred embodiment, the method for collecting the experimental dataset is exactly the same as the standard dataset BCI Competition IV-2a. The dataset uses 22 Ag / AgCl electrodes to record two treatments for each subject, for a total of 9 subjects. For each subject, the two experiments were conducted over two days. During the experiment, the subject sat comfortably in front of the screen. At the start of each trial, a cross with a short warning tone appeared on the black screen. After 2 seconds, an arrow pointing left, right, down, or up (corresponding to left hand, right hand, foot, or tongue) appeared, indicating the execution of the motor imagery task. Subjects were asked to perform the MI task from t=2s to t=6s. Each subject contained 72 trials, for a total of 288 trials obtained from each subject per subject. The EEG signal used in each trial can be represented as X∈R C×T Where C=22 is the number of channels and T=1000 is the number of data points.

[0048] Step S2: Perform wavelet transform on the experimental dataset and the standard dataset to generate time-frequency plots of the experimental dataset and the standard dataset. In a preferred embodiment, the experimental dataset is first preprocessed, and then the time-frequency plots of the experimental dataset and the standard dataset are generated using continuous wavelet transform (CWT). Specifically, this includes the following steps:

[0049] Step S21: Preprocess the experimental dataset. The preprocessing method for the experimental dataset is exactly the same as that for the standard dataset. The experimental dataset samples the signal at 250Hz, performs bandpass filtering between 0.5Hz and 100Hz, and then uses a 50Hz notch filter to suppress noise. Finally, the experimental dataset is divided into training and test sets according to the proportions of the standard dataset.

[0050] Step S22: Generate time-frequency plots for the experimental and standard datasets using CWT. CWT inherits and develops the localization concept of Short-Time Fourier Transform (STFT), while overcoming the drawbacks such as the window size not changing with frequency. The time-frequency plots generated by CWT will have more details, facilitating subsequent model training and improving model performance.

[0051] The process of generating a time-frequency plot using CWT is as follows. After selecting the mother wavelet ψ(t), this process can be mathematically represented as follows:

[0052]

[0053] Where CWT is the time-frequency representation of x(t), ψ(t) is the scaled and transformed wavelet, and ψ * (t) is the complex conjugate ψ(t). The mother wavelet ψ(t) must satisfy three allowable conditions, including

[0054]

[0055] In the frequency domain, CWT can be represented as:

[0056]

[0057] in and are x(t) and ψ * The Fourier transform of (t). The wavelet coefficients in the wavelet transform accurately reflect the energy intensity and distribution related to scale a and time t. The larger the wavelet coefficients, the better the match between the signal and the wavelet. In particular, the scale a can be adjusted for frequency using the following formula:

[0058] F a =F c / a

[0059] Where F c The center frequency ψ(t) of the mother wavelet, F a This corresponds to the pseudo-frequency at scale a. However, most features of the EEG signal are redundant, so only the μ and β bands highly correlated with MI are extracted. To ensure balance between different bands, the μ and β bands can be resized to the same size before being combined into the represented image.

[0060] Step S3: Based on the requirements for EEG data augmentation, a generative adversarial network (GAN) model for EEG data augmentation is designed, along with a corresponding loss function. In this embodiment, the model structure borrows from the CycleGAN model, leveraging CycleGAN's excellent image transfer capabilities to fuse features from EEG signals in two-domain datasets to generate more universally applicable EEG signals. See also... Figure 2 Signals from the X domain (standard dataset) and the Y domain (experimental dataset) are processed by continuous wavelet transform to convert them into time-frequency maps, which are then input into CycleGAN for further processing. This yields new EEG signals with the same characteristics as the standard dataset, thus regenerating the EEG signals. The model is trained by solving a min-max problem between the generator and discriminator networks. The generator is trained to minimize the loss function, while the discriminator is trained to maximize the loss function. This can be represented by the following optimization problem:

[0061] minmaxL total (G,F,D X D Y )

[0062] Our defined loss function includes three types of losses: GAN loss L... GAN cycle loss L cyc and identify loss L id :

[0063] L total (G,F,D X D Y )=-L GAN (G,F,D X D Y )+λ cyc L cycle (G,F)+λ id L identify (G,F)

[0064] GAN loss L GAN The formula is as follows:

[0065]

[0066] cycle loss L cyc The formula is as follows:

[0067]

[0068] identify loss L id The formula is as follows:

[0069]

[0070] The identification loss is not shown in the figure above. This loss can prevent the target domain image from being unnecessarily distorted by the GAN loss.

[0071] Step S4: Design the generator and discriminator of the generative adversarial network model based on the characteristics of the EEG data. In this embodiment, the generator design is described in [reference needed]. Figure 3 To extract as many time-frequency features as possible from the EEG signal, the generator adopts a residual structure, and the feature extraction part uses a deencoder design, further extracting more features through dilated convolutions with different coefficients. The generator input is the time-frequency map obtained from wavelet transform, and the output is then input into another generator. The two generators have identical structures.

[0072] For the design of the decoder and residual block (expansion factor equal to 1), please refer to [reference needed]. Figure 4 In this design, modules with an expansion coefficient of 1 change accordingly when the expansion coefficient of the deencoder changes, while modules with an expansion coefficient of 0 do not change accordingly. This deencoder design fully integrates the characteristics of EEG signals from different feature extraction stages, enabling the generator to generate more universal EEG signals.

[0073] The discriminator structure of the GAN model used for EEG data augmentation does not employ a single 2D convolutional kernel. This is because the wavelet-transformed EEG data time-frequency plot uses rows representing time and columns representing data frequency, which is insufficient to capture enough features from the represented image. Therefore, in addition to the usual 2D kernel, one-dimensional convolutional kernels spanning frequency channels and time are added to the proposed discriminator structure. For an explanation of these three types of convolutional kernels, see [link to relevant documentation]. Figure 5 .

[0074] Step S5: Using the designed generative adversarial network model, fuse the features of the experimental dataset and the standard dataset to generate a new dataset. (See also...) Figure 7 As shown, the EEG signals from the two datasets are converted into time-frequency graphs by wavelet transform, and then new samples are generated by two generators. After inverse wavelet transform, they are converted into a new EEG dataset, thus completing the sample generation task.

[0075] Step S6: Train the classic deep neural network model for motion image classification using the standard dataset, experimental dataset, and a new dataset generated by the generative adversarial network model. The classic DNN model used for validation on the new dataset in this patent is EEGNet. The deep learning architecture used for training is PyTorch, and the data preprocessing procedures for the standard dataset, experimental dataset, and new dataset are identical.

[0076] Step S7: Use the trained classic deep neural network model to classify motor imagery and analyze whether the new dataset improves the accuracy of the EEG-based motor imagery classification model. If the accuracy does not meet the data application requirements, return to step S3 and continue to improve the model design until it meets the data application requirements. Experiments show that the new dataset generated using the data augmentation scheme of this application, compared to the standard dataset and experimental dataset, shows that the classification accuracy of the new dataset is nearly 10% higher than both the latter two. Therefore, the new dataset generated by the data augmentation scheme of this application improves the accuracy of the EEG-based motor imagery classification model.

[0077] Example 2

[0078] This embodiment provides an EEG data augmentation system based on a generative adversarial network model, used to implement the EEG data augmentation method described in Embodiment 1, including:

[0079] The acquisition module is used to acquire experimental datasets of electroencephalograms (EEGs).

[0080] The data augmentation module is used to augment the experimental dataset using a pre-designed generative adversarial network model to obtain a new dataset.

[0081] A training module is used to train the new dataset using a deep neural network model;

[0082] The application module uses a trained deep neural network model to classify motor imagery and analyzes whether the new dataset improves the accuracy of the EEG-based motor imagery classification model.

[0083] Example 3

[0084] This embodiment provides a computer device, including a memory and a processor, wherein the processor and the memory communicate with each other, the memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to execute the electroencephalogram data amplification method as described in Embodiment 1.

[0085] Example 4

[0086] This embodiment provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the electroencephalogram (EEG) data amplification method as described in Embodiment 1.

[0087] In summary, this invention provides a method and system for EEG data augmentation based on a generative adversarial network (GAN) model, addressing existing problems in the classification of motor imagery using deep neural networks and meeting the requirement to improve the generalization ability of motor imagery classification models. The method introduces a GAN model into the EEG data augmentation problem. By rationally designing a loss function, it generates a new, more accurate EEG dataset with the characteristics of both experimental and standard datasets, reducing the workload and cost for researchers. The generator of the GAN model is an encoder-decoder based on dilated convolution, capable of extracting features from EEG data at different levels. The discriminator of the GAN model is a discriminator based on a parallel feature extraction structure with different dimensions, capable of extracting features of different properties.

[0088] Those skilled in the art should understand that variations can be implemented by combining existing technology with the above embodiments, which will not be elaborated here. Such variations do not affect the essence of the present invention, and will not be elaborated here either.

[0089] The preferred embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and the devices and structures not described in detail should be understood as being implemented in a conventional manner in the art. Any person skilled in the art can make many possible variations and modifications to the technical solutions of the present invention using the methods and techniques disclosed above, or modify them into equivalent embodiments with equivalent changes, without departing from the scope of the present invention. This does not affect the essential content of the present invention. Therefore, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the present invention's technical solutions still fall within the protection scope of the present invention.

[0090] Those skilled in the art will recognize that the units, i.e., algorithm steps, of the various examples described in connection with this embodiment can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

Claims

1. A method for augmenting electroencephalogram (EEG) data based on a generative adversarial network (GAN) model, characterized in that, Includes the following steps: Step S1: Collect EEG data from the subjects, generate an experimental dataset, preprocess the experimental dataset, and divide the experimental dataset into a training set and a test set according to the proportion of the standard dataset. Step S2: Perform wavelet transform on the experimental dataset and the standard dataset to generate time-frequency diagrams of the experimental dataset and the standard dataset; Step S3: Based on the requirements for amplification of EEG data, design a generative adversarial network model for EEG data amplification and design the corresponding loss function; Step S4: Design the generator and discriminator of the generative adversarial network model based on the characteristics of the EEG data; the generator includes two generators, the whole adopts a residual structure, the feature extraction part adopts a deencoder design, and more features are extracted by dilated convolution with different coefficients. The module with an expansion coefficient of 1 changes accordingly when the expansion coefficient of the decoder changes, while the module with an expansion coefficient of 0 does not change accordingly when the expansion coefficient of the decoder changes; the discriminator structure includes a two-dimensional convolutional kernel, a one-dimensional convolutional kernel across frequency channels, and a one-dimensional convolutional kernel across time. Step S5: Using the designed generative adversarial network model, the features of the experimental dataset and the standard dataset are fused to generate a new dataset; Step S6: Use the standard dataset, experimental dataset, and new dataset generated by the generative adversarial network model to train the classic deep neural network model for motion image classification. Step S7: Use the trained classic deep neural network model to classify motor imagery and analyze whether the new dataset improves the accuracy of the EEG-based motor imagery classification model. If the accuracy does not meet the data application requirements, return to step S3 and continue to improve the model design until it meets the data application requirements.

2. The method for amplifying electroencephalogram (EEG) data according to claim 1, characterized in that, In step S3, the generative adversarial network model adopts the cycleGAN model; the loss function includes three types of loss: GAN loss. cycle loss and identify loss It can be described by the following formula: ； Among them, GAN loss The formula is as follows: ； cycle loss The formula is as follows: ； identify loss The formula is as follows: 。 3. A system for augmenting electroencephalogram (EEG) data based on a generative adversarial network (GAN) model, characterized in that, A method for implementing electroencephalogram (EEG) data amplification as described in claim 1 or 2 includes: The acquisition module is used to acquire experimental datasets of electroencephalograms (EEGs). The data augmentation module is used to augment the experimental dataset using a pre-designed generative adversarial network model to obtain a new dataset. A training module is used to train the new dataset using a deep neural network model; The application module uses a trained deep neural network model to classify motor imagery and analyzes whether the new dataset improves the accuracy of the EEG-based motor imagery classification model.

4. A computer device comprising a memory and a processor, the processor and the memory communicating with each other, the memory storing program instructions executable by the processor, characterized in that, The processor invokes the program instructions to execute the electroencephalogram data amplification method as described in claim 1 or 2.

5. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the electroencephalogram (EEG) data amplification method as described in claim 1 or 2.

Images