A cross-subject cognitive load electroencephalogram signal data enhancement method based on a generative adversarial network

Abstract

The application discloses a cross-subject cognitive load electroencephalogram signal data enhancement method based on a generative adversarial network, and source domain data and target domain calibration data are input into the SIE-GAN data enhancement method of the application. The method adds an encoder module for extracting the features of original cognitive load electroencephalogram signals. At the same time, the data enhancement method discards the setting of random noise as input in the traditional GAN method and instead uses source domain data and target domain calibration data as input. Since the source domain data and the target domain calibration data can provide more context information, the model can better adapt to the differences between different subjects, thereby improving the cross-subject generalization ability of the model. The application has been verified by multiple deep learning models on a cognitive load electroencephalogram signal self-sampling data set, and the results show that the classification accuracy of cross-subject cognitive load recognition is effectively improved.

Description

Technical Field

[0001] This invention belongs to the interdisciplinary field of computer science and biomedical engineering. This field combines multiple disciplines such as engineering, medicine, and biology, aiming to apply engineering principles and technologies to solve medical problems, such as developing medical devices, studying biosignal mechanisms, control, and simulation. Among these, electroencephalograms (EEGs) are one of the most common signal types in biosignal processing; therefore, cognitive load assessment methods based on EEGs have broad application prospects in the field of biomedical engineering. Background Technology

[0002] Cognitive load is generally defined as the level of energy and resource consumption required by the brain. When the human brain performs cognitive tasks, neurons frequently transmit and process information, which consumes energy and resources. Load reflects the level of brain activity and resource utilization during cognitive tasks.

[0003] In recent years, with the development of intelligent and automated auxiliary technologies, equipment, and human-machine collaborative practices, the monitoring of human psychological states such as fatigue, mood, and cognitive load has been widely applied in fields such as human-computer interaction and brain-computer interfaces. By measuring cognitive load, people can assess the difficulty and complexity of cognitive tasks. For example, for professions requiring high workloads, such as pilots, who need to process large amounts of information and coordinate multiple tasks in a short period of time, their cognitive load directly affects task performance. At the same time, high workload is a direct factor leading to unsafe operations. Workers in certain occupational groups are prone to physical fatigue under prolonged high workloads, lack awareness of the risks of being in a high-load state for extended periods, leading to a numbness to safety and a strong sense of insecurity. Therefore, effectively assessing and reducing operator workload is extremely important for preventing unsafe behaviors and reducing the occurrence of dangerous accidents.

[0004] Identifying pilot workload based on EEG signals has been an active research area. However, collecting a large number of labeled training samples of cognitive workload is often challenging due to complex experimental setups, sensory discomfort from prolonged use of EEG acquisition devices, and difficulties in inducing and accurately labeling fatigue states. Data augmentation methods can alleviate these problems by expanding the samples from the original signals. Typical data augmentation methods include geometric transformations, such as cropping, flipping, and scaling image data. Unlike images, EEG signals often exhibit high spatiotemporal correlation. Therefore, data augmentation techniques using geometric transformations commonly used for image data are not suitable for EEG signals. While some feasible methods exist for EEG signal data augmentation, including adding Gaussian noise, segmentation, and generative models, these methods cannot meet the needs of multi-channel EEG signal generation due to redundant noise or information loss.

[0005] Generative adversarial networks (GANs), as a type of deep generative model, can learn to generate samples from data distributions and significantly improve the quality of generated samples. However, because EEG signals are subject-dependent, most existing methods for data augmentation on EEG data from specific subjects only achieve good classification results on data from the same subject; their classification performance on data from other subjects remains unsatisfactory.

[0006] Therefore, this invention proposes a cross-subject data augmentation method based on generative adversarial networks, which can generate high-quality, subject-independent EEG signals, reduce the impact of individual differences on experimental results, and enable the model to have better generalization ability among different subjects. Summary of the Invention

[0007] This invention proposes a cross-subject data augmentation method based on generative adversarial networks (GANs), namely Subject-Invariant Encoder Generative Adversarial Network (SIE-GAN). The main idea of ​​this method is to change the input of traditional GAN ​​models—raw data and random noise—to using source domain data and target domain calibration data as input for adversarial learning. This generates subject-independent EEG samples, effectively improving the problem of EEG signal samples depending on specific subjects in cross-subject experiments and enhancing the model's generalization ability. Specifically, this invention incorporates an encoder module to convert high-dimensional EEG data into low-dimensional feature vectors, reducing the input dimensionality of the generator and discriminator, thus improving training efficiency and model performance. Simultaneously, the encoder extracts feature vectors that effectively capture information relevant to the cognitive load task while ignoring irrelevant noise and subject variability, supporting the generation of cognitive load EEG signals unrelated to the subject.

[0008] The specific contents of the invention include:

[0009] First, the original EEG dataset is preprocessed. Then, the dataset is divided into three parts: training set, validation set, and test set. The SIE-GAN method proposed in this invention is used to augment the EEG signal samples in the training set. Finally, the training set samples (original samples + generated samples) and validation set samples are trained in multiple deep learning network models. The model classification results are obtained on the test set, and the classification results are evaluated to verify the accuracy and generalization ability of the model.

[0010] The specific steps of the technical solution of this invention are as follows:

[0011] Step 1. Data preprocessing and segmentation of cognitive load EEG signals: The raw cognitive load EEG signals were bandpass filtered using a Butterworth bandpass filter, and then the filtered signals were standardized by exponential moving mean. The preprocessed data from multiple subjects were divided into source and target domains, with 10% of the target domain subjects having known labels (a common approach in cross-subject classification experiments). The source domain subjects and the 10% target domain subjects with known labels were divided into a training set (20% of the training set was used as the validation set), and the remaining 90% of the target domain subjects were used as the test set.

[0012] Step 2. Data Augmentation of EEG Signals under Cognitive Load: The source domain data and target domain calibration data from the training set are input into the SIE-GAN data augmentation method of this invention. The encoder module extracts features from the EEG signals, and then the generator generates the EEG data. During adversarial training, a loss function guides the generator to produce EEG signals that are more consistent with the irrelevance of the subjects.

[0013] Step 3. Train the classification model based on cognitive load EEG signals. Input the training set (including the EEG signals generated in Step 2) and validation set from Step 1 into the deep learning model for training, optimize model parameters, and improve the model's classification accuracy and generalization ability;

[0014] Step 4. Identify the states corresponding to cognitive load EEG signals based on the trained classification model. Input the test set from Step 1 into the trained model from Step 3 for classification and evaluate the accuracy of the classification.

[0015] The present invention has the following advantages:

[0016] 1. EEG signals are highly individualized, with significant differences among different subjects. Effectively generating subject-independent EEG signals within generative adversarial networks (GANs) is a major challenge. This data augmentation method abandons the traditional GAN ​​data augmentation approach that uses random noise and raw signals as input. Instead, it inputs source domain data and target domain calibration data into the GAN, allowing the source and target domain data to provide more contextual information. This enables the model to better adapt to the differences among subjects, effectively generating subject-independent EEG signals and improving the generalization ability of the model in cross-subject experiments.

[0017] 2. The training process of GANs is prone to instability issues, such as mode collapse and gradient vanishing. Stable training of generative adversarial networks to converge to an ideal state is a significant technical challenge. This method uses input data containing more useful information, which can reduce mode collapse and instability during GAN training, thereby improving training efficiency and model performance.

[0018] 3. A major challenge is how to reduce the number of parameters in the training process of generative adversarial networks while ensuring the effectiveness of feature extraction. This invention adds an encoder module to extract features from the original EEG signal, so that the EEG signal with extracted features is input to the generator and discriminator. This not only effectively reduces the number of parameters, but also better captures information related to the workload, thus improving the training efficiency of the GAN model. Attached Figure Description

[0019] Figure 1 This is the overall flowchart of the method of the present invention.

[0020] Figure 2 This is a flowchart of the data partitioning method of the present invention.

[0021] Figure 3 This is a network structure diagram of the method of the present invention.

[0022] Figure 4 This is a network structure diagram of the encoder part of the method of the present invention. Detailed Implementation

[0023] This invention proposes a method for augmenting EEG signal data across subject cognitive load based on generative adversarial networks. This method uses source domain data and target domain calibration data as inputs, and uses the encoder layer to extract features from the EEG signals, which facilitates learning information related to the load task during training. While generating subject-irrelevant samples, it also reduces the number of parameters and improves the model training efficiency and performance.

[0024] Figure 1 The overall flowchart of the method of this invention can be broken down into the following steps:

[0025] Step 1. Data preprocessing and segmentation of cognitive load EEG signals;

[0026] Step 2. Perform data augmentation of cognitive load EEG signals using the SIE-GAN method proposed in this invention;

[0027] Step 3. Train the deep learning model based on the training and validation sets of cognitive load EEG signals;

[0028] Step 4. Test the model's performance and evaluate the accuracy of identifying cognitive load EEG signal states across subjects.

[0029] The specific details of each step are explained below:

[0030] Step 1:

[0031] (1) The original cognitive load EEG signal samples were bandpass filtered using a third-order 4-40Hz Butterworth bandpass filter to filter out the signals in the required frequency band.

[0032] (2) The filtered cognitive load EEG signal was standardized by exponential moving mean, with the attenuation factor set to 0.999 to reduce the impact of numerical differences on the model performance.

[0033] (3) Before starting training, the preprocessed EEG dataset is divided; Figure 2 The flowchart below illustrates the cross-subject experimental data partitioning process of this invention. This invention employs the leave-one-subject-out (LOSO) method, commonly used across subject domains, where data from one subject is used as the target domain subject, and data from other subjects are used as the source domain subject. Following common practice in other literature, the labels of 10% of the data from the target domain subject are used as target domain calibration data. The source domain subject data and the 10% target domain calibration data are used as training samples, and the remaining 90% of the target domain data are used as test samples. Furthermore, 80% of the data from the training samples is randomly selected as the training set, and the remaining 20% ​​is used as the validation set.

[0034] Step 2:

[0035] To address the issue that raw EEG signals are dependent on specific subjects, this invention proposes a cross-subject data augmentation method based on generative adversarial networks (SIE-GAN). The network structure of this method is as follows: Figure 3 As shown, it mainly consists of three parts: the encoder, the generator, and the discriminator. Each part will be described in detail below:

[0036] (1) Encoder

[0037] The encoder of this invention is responsible for extracting spatiotemporal information from electroencephalogram (EEG) signals to obtain feature vectors. The network structure of the encoder is as follows: Figure 4 As shown, the encoder, drawing inspiration from the classic shallow convolutional neural network ShallowConvNet, comprises three core structures: temporal convolution, spatial convolution, and average pooling. Temporal convolution extracts temporal information from each channel of the EEG signal through one-dimensional convolution along the time axis (kernel length equal to 25). Spatial convolution performs one-dimensional convolution along the channel axis of the EEG signal (kernel length equal to the number of leads), assigning weights to the signals of each lead to extract spatial information from the EEG signal. The average pooling layer performs average convolution along the time axis with a kernel length of 75 and a stride of 15, compressing temporal information and providing downsampling. The encoder offers a solution combining temporal and spatial convolution layers, extracting information from the time and spatial axes of the input signal respectively, replacing the traditional two-dimensional convolution structure. This not only effectively extracts temporal and spatial information but also significantly reduces the number of parameters.

[0038] (2) Generator

[0039] The generator is responsible for reconstructing the input feature vector (after passing through the encoder), and then deconvolving it to the original signal dimension to obtain the generated EEG signal sample.

[0040] (3) Discriminator

[0041] The discriminator is responsible for judging the probability that the input feature vector (after passing through the encoder) belongs to the target domain. For EEG signal samples generated by the generator, they need to pass through an encoder before being input into the discriminator. Ideally, the discriminator obtains a value of 0.5, indicating that the discriminator cannot determine whether the input feature vector comes from the source domain EEG signal or the target domain EEG signal.

[0042] The above are the three main parts of the network module of this invention. In order for the generator to generate subject-independent EEG signals, the loss function of the GAN training process needs to be introduced here:

[0043] During GAN training, the generator and discriminator are updated alternately, using data inputs from the source domain and target domain as real and fake samples, respectively. The loss function of the discriminator is calculated as follows:

[0044]

[0045] Where m represents the batch size, which is the number of samples used each time the parameters are updated;

[0046] This means maximizing the output probability of the discriminator for the source domain data, so that it can identify the source domain data as real data; This represents minimizing the output probability of the discriminator for the target domain data, enabling it to identify the target domain data as fake data; This means maximizing the output probability of the discriminator for the data generated from the source domain samples, so that it can identify the data generated by the generator as source domain data.

[0047] The loss function of the generator is calculated as follows:

[0048]

[0049] in By minimizing the probability that the generated sample is identified as data in the target domain, the generator produces data that makes it difficult for the discriminator to distinguish between the source domain and the target domain.

[0050] Training parameters: Batch size is set to 64, generator and discriminator are trained for 200 rounds, with the generator learning rate being 1e-4 and the discriminator learning rate being 8e-5.

[0051] Step 3:

[0052] The cognitive load EEG signal data of the training set and validation set are input into the deep learning model for training; the number of iterations in the training phase is set to 500, the batch size of batch training is set to 64, and the Adam method is used as the optimizer; the model is saved when the validation set loss function reaches the minimum; the model with the lowest validation set loss function during the training process is recorded, and it is used to predict the test set samples, and the test set accuracy is recorded.

[0053] Step 4:

[0054] The test set from step 1 is input into the pre-trained model from step 3 for classification and recognition, and the accuracy of classification is evaluated. The dataset and experimental results used in this invention are described below:

[0055] (1) Dataset

[0056] This invention uses a self-collected dataset containing EEG signals from 7 subjects during cognitive loads during flight missions. These EEG signals have a total of 69 channels and a sampling rate of 1000Hz. During the data loading phase, 9 useless channels are filtered out, leaving 60 channels, and the sampling rate of the EEG signals is downsampled to 250Hz.

[0057] (2) Experimental Results

[0058] This experiment employed the leave-one-subject-out (LOSO) method, a common technique in cross-subject experiments. Seven participants were rotated among the target domain participants, while the remaining six were used as the source domain participants. 90% of the data from the target domain was used as the test set, and the remaining data, along with the source domain data, served as the training set. The average classification accuracy across the seven target domain participants under both high and low loads was used as the performance metric.

[0059] Table 1 shows the LOSO experimental results under different models and settings. As can be seen from the table, the cross-subject classification accuracy of the SIE-GAN data augmentation method proposed in this invention is higher than that without data augmentation method on multiple deep learning models.

[0060] Table 1 LOSO Comparison Experiment

[0061]

Claims

1. A method for enhancing EEG signal data across subject cognitive load based on generative adversarial networks, characterized in that, Includes the following steps: Step 1. Preprocessing of cognitive load EEG signal data: The original cognitive load EEG signal samples were bandpass filtered using a Butterworth bandpass filter, and then the filtered signals were normalized by moving mean. The preprocessed data from multiple subjects were divided into source domain subjects and target domain subjects. The source domain subjects and 10% of the target domain calibration data were used as the training set, and the remaining 90% of the target domain subjects' data were used as the test set. Step 2. Cross-subject data augmentation of cognitive load EEG signals: The source domain data and target domain calibration data in the training set are fed into the SIE-GAN data augmentation method. The EEG signals are feature extracted through the Encoder module. During adversarial training, the generator is guided by the loss function to generate EEG signals that are more consistent with subject irrelevance. Step 3. Verify the improvement of the model's generalization ability by cross-subject data augmentation methods: Input the training set and validation set from Step 1 into the deep learning model for training, and identify the corresponding state of the cognitive load EEG signals from the test set in Step 1 based on the trained classification model. Step 2 is as follows: A cross-subject data augmentation method SIE-GAN based on generative adversarial networks is proposed. The network structure of this method consists of three parts: encoder, generator and discriminator. (1) Encoder; The encoder extracts spatiotemporal information from the cognitive load EEG signal to obtain a feature vector. Borrowing from the classic shallow convolutional neural network ShallowConvNet, the encoder comprises three core structures: temporal convolution, spatial convolution, and average pooling. Temporal convolution extracts temporal information from each channel of the cognitive load EEG signal through one-dimensional convolution along the EEG signal's time axis. Spatial convolution performs one-dimensional convolution along the EEG signal's channel axis, assigning weights to each channel's signal to extract spatial information from the cognitive load EEG signal. The average pooling layer performs average convolution along the time axis to compress temporal information and downsample. The encoder combines temporal and spatial convolutional layers to extract information along the time and spatial axes of the input EEG signal, replacing the two-dimensional convolution structure, thus extracting both temporal and spatial information and reducing the number of parameters. (2) Generator; The generator reconstructs the input feature vector and deconvolves it to the original signal dimension to obtain the generated EEG signal sample. (3) Discriminator; The discriminator is responsible for judging the input feature vector and obtaining the probability that the feature vector is from the target domain. For the cognitive load EEG signal sample generated by the generator, it needs to be passed through an encoder before being input into the discriminator. The discriminator gets a value of 0.5, which means that the discriminator cannot determine whether the input feature vector comes from the source domain EEG signal or the target domain EEG signal. During GAN training, the generator and discriminator are updated alternately, using data inputs from the source domain and target domain as real and fake samples, respectively. The loss function of the discriminator is calculated as follows: Where m represents the batch size, i.e., the number of samples used each time the parameters are updated; i is the batch number; This means maximizing the output probability of the discriminator for the source domain data, so that it can identify the source domain data as real data; This represents minimizing the output probability of the discriminator for the target domain data, enabling it to identify the target domain data as fake data; This means maximizing the output probability of the discriminator for the data generated from the source domain samples, so that it can identify the data generated by the generator as source domain data. The loss function of the generator is calculated as follows: in By minimizing the probability that the generated sample is identified as data in the target domain, the generator produces data that makes it difficult for the discriminator to distinguish between the source domain and the target domain. Training parameters: Batch size is set to 64, generator and discriminator are trained for 200 rounds, with the generator learning rate being 1e-4 and the discriminator learning rate being 8e-5.

2. The cross-subject data augmentation method based on generative adversarial networks according to claim 1, characterized in that: Step 1 is as follows: (1) The original cognitive load EEG signal samples were bandpass filtered using a third-order 4-40Hz Butterworth bandpass filter to filter out the signals in the required frequency band. (2) The filtered signal is normalized by exponential moving average, with the attenuation factor set to 0.999 to reduce the impact of numerical differences on the model performance. (3) Before training begins, the preprocessed EEG dataset is divided; one subject's data is used as the target domain subject, and the data of other subjects are used as the source domain subjects. The labels of 10% of the data in the target domain subjects are known; the source domain subjects and 10% of the known target domain subjects are used as training samples, and the remaining 90% of the target domain subjects are used as test samples. 80% of the data in the training samples are randomly selected as the training set, and the remaining 20% ​​of the data are used as the validation set.

3. The cross-subject data augmentation method based on generative adversarial networks according to claim 1, characterized in that: Step 3 specifically involves: The training and validation sets of cognitive load EEG signals are input into the deep learning model for training; the number of iterations in the training phase is set to 500, the batch size of batch training is set to 64, and the Adam method is used as the optimizer; the model is saved when the validation set loss function reaches its minimum; the model with the lowest validation set loss function during training is recorded and used to predict test set samples; the test set from step 1 is input into the trained model for classification and recognition, and the accuracy of classification is evaluated.

4. The cross-subject data augmentation method based on generative adversarial networks according to claim 3, characterized in that: The experiment used the leave-one-out method, in which 7 participants took turns as the target domain and the remaining 6 participants as the source domain. 90% of the data in the target domain was used as the test set, and the remaining data and the source domain data were used together as the training set.

Images