Cross-subject cognitive load electroencephalogram signal data enhancement method based on generative adversarial network

Abstract

The invention discloses a cross-subject cognitive load electroencephalogram signal data enhancement method based on a generative adversarial network. Source domain data and target domain calibration data are input into the SIE-GAN data enhancement method. According to the method, an encoder module is added and used for extracting features of original cognitive load electroencephalogram signals. Meanwhile, according to the data enhancement method, the setting that random noise serves as input in a traditional GAN method is abandoned, source domain data and target domain calibration data serve as input, and due to the fact that the source domain data and the target domain calibration data can provide more contextual information, the model can better adapt to the difference between different subjects, and the accuracy of data enhancement is improved. Therefore, the generalization ability of the model across subjects is improved. Verification is carried out on a cognitive load electroencephalogram signal self-acquisition data set through a plurality of deep learning models, and the result shows that the classification accuracy of cross-subject cognitive load recognition is effectively improved.

Description

Technical Field

[0001] The present invention belongs to the intersection of computer and biomedical engineering. This field combines multiple disciplines such as engineering, medicine, biology, etc., and aims to apply engineering principles and technologies to solve medical problems, such as developing medical equipment, studying biological signal mechanisms, control and simulation, etc. Among them, electroencephalogram (EEG) is one of the most common signal types in biological signal processing, so the cognitive load assessment method based on EEG has broad application prospects in the field of biomedical engineering. Background Art

[0002] Cognitive load is usually defined as the degree of energy and resource consumption required by the brain. When the human brain performs cognitive tasks, neurons in the brain frequently transmit and process information, which requires energy and resources. Load can reflect the activity level and resource utilization of the brain when performing cognitive tasks.

[0003] In recent years, with the development of intelligent automation auxiliary technology, equipment and human-machine collaborative practice, human psychological state monitoring such as fatigue, emotion, cognitive load, etc. has been widely used in the fields of human-machine interaction and brain-computer interface. By measuring the cognitive load state, people can evaluate the difficulty and complexity of cognitive tasks. For example, for occupations that require high load, such as pilots, they need to process a large amount of information in a short period of time and complete multiple tasks in collaboration. Their cognitive load state has a direct impact on task performance. At the same time, high load is a direct factor that causes unsafe operations. Workers in special occupational groups are prone to physical fatigue under long-term high load, lack awareness of being in a high-load state for a long time, resulting in numbness to the safety state and a great sense of insecurity. Therefore, effectively evaluating and reducing the load of operators is extremely important for preventing unsafe behaviors and reducing the occurrence of dangerous accidents.

[0004] Identifying the load state of pilots based on EEG signals has always been an active research area. However, due to the complex experimental setup, the sensory discomfort caused by wearing EEG signal acquisition equipment for a long time, and the difficulty in inducing and correctly labeling fatigue states, collecting a large number of labeled training samples of cognitive load states is usually a big challenge. Data augmentation methods can alleviate the above problems by expanding samples through the original signal. Typical data augmentation methods include geometric transformations, such as cropping, flipping and scaling image data. Unlike images, EEG signals often have high spatiotemporal correlation. Therefore, the data augmentation precautions of geometric transformations commonly used for image data are not applicable to EEG signals. For data augmentation of EEG signals, although there are still some feasible methods, including adding Gaussian noise, segmentation and generative models, these methods cannot meet the needs of multi-channel EEG signal generation due to redundant noise or information loss.

[0005] As a deep generative model, the Generative Adversarial Network (GAN) can learn to generate samples from data distribution and significantly improve the quality of generated samples. At the same time, due to the problem that EEG signals are dependent on specific subjects, most existing methods use EEG data of specific subjects to enhance data. The models trained with these data can only achieve good classification results on the data of the same subjects, and the classification results for data of other subjects have not yet achieved satisfactory results.

[0006] Therefore, the present invention proposes a cross-subject data enhancement method based on a generative adversarial network, which can generate high-quality subject-independent EEG signals, reduce the impact of individual differences on experimental results, and enable the model to have better generalization ability between different subjects. Summary of the invention

[0007] The present invention aims to propose a cross-subject data enhancement method based on generative adversarial network, Subject-Invariant Encoder Generative Adversarial Network, referred to as SIE-GAN. The main idea of ​​this method is to change the input of the traditional GAN ​​model - raw data and random noise, and change it to: use the source domain data and the target domain calibration data as input for adversarial learning, so as to generate subject-independent EEG samples, which effectively improves the problem that the load EEG signal samples in the cross-subject experiment depend on specific subjects, and improves the generalization ability of the model. Specifically, the present invention adds an encoder module to convert high-dimensional EEG data into low-dimensional feature vectors to reduce the input dimensions of the generator and discriminator, and improve the training efficiency and model performance. At the same time, the encoder extracts the feature vector, effectively captures the information related to the cognitive load task, and ignores the irrelevant noise and the variability data of the subject, supporting the generation of cognitive load EEG signals that are irrelevant to the subject.

[0008] The specific invention contents include:

[0009] First, the original EEG data set is preprocessed; then the data set is divided into three parts: training set, validation set and test set, and the SIE-GAN method proposed in the present invention is used to enhance the data of the load EEG signal samples of the training set. Finally, the training set samples (original samples + generated samples) and the validation set samples are trained in multiple deep learning network models, and the model classification results are obtained on the test set, and the classification results are evaluated to verify the accuracy and generalization ability of the model;

[0010] The specific steps of the technical solution of the present invention are as follows:

[0011] Step 1. Data preprocessing and division of cognitive load EEG signals: Use a Butterworth bandpass filter to perform bandpass filtering on the original cognitive load EEG signals, and then perform exponential moving mean normalization on the filtered signals; divide the preprocessed data of multiple subjects into source domain and target domain, where 10% of the target domain subjects have known labels (a common solution for cross-subject classification experiments), divide the source domain subjects and 10% of the target domain subjects with known labels into a training set (take 20% of the training set as a validation set), and the remaining 90% of the target domain subjects as a test set;

[0012] Step 2. Data enhancement of cognitive load EEG signals: The source domain data and target domain calibration data in the training set are input into the SIE-GAN data enhancement method of the present invention, and the EEG signals are feature extracted through the encoder module, and then the EEG data is generated through the generator. During the adversarial training process, the generator will be guided to generate EEG signals that are more in line with the subject's independence through the loss function.

[0013] Step 3. Train the classification model based on the EEG signals of cognitive load. Input the training set in step 1 (including the EEG signals generated in step 2) and the validation set into the deep learning model for training, optimize the model parameters, and improve the classification accuracy and generalization ability of the model;

[0014] Step 4. Identify the state corresponding to the EEG signal of cognitive load based on the trained classification model. Input the test set in step 1 into the trained model in step 3 for classification, and evaluate the accuracy of classification.

[0015] The present invention has the following advantages:

[0016] 1. EEG signals are highly individualized, and there are significant differences between EEG signals of different subjects. How to effectively generate EEG signals that are independent of the subject in a generative adversarial network is a major difficulty. This data enhancement method abandons the setting of using random noise and original signals as input in the traditional GAN ​​data enhancement method, and instead inputs source domain data and target domain calibration data into the GAN, so that the source domain data and target domain calibration data can provide more contextual information, allowing the model to better adapt to the differences between different subjects, effectively generate EEG signals that are independent of the subject, and improve the generalization ability of the cross-subject experimental model;

[0017] 2. The training process of GAN is prone to instability, such as mode collapse and gradient vanishing. How to stably train the generative adversarial network so that it converges to an ideal state is an important technical difficulty. The input data of this method contains more useful information, which can reduce the problems of mode collapse and instability during GAN training, thereby improving training efficiency and model performance.

[0018] 3. How to reduce the number of parameters in the training process of the generative adversarial network while ensuring the effectiveness of feature extraction is a major difficulty. The present invention adds an encoder module to extract features of the original EEG signal, so that the EEG signal after feature extraction is input into the generator and discriminator. This can not only effectively reduce the number of parameters, but also better capture information related to the load task, thereby improving the efficiency of GAN model training. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is the overall flow chart of the method of the present invention.

[0020] Figure 2 It is a data partitioning flow chart of the method of the present invention.

[0021] Figure 3 It is a network structure diagram of the method of the present invention.

[0022] Figure 4 It is a network structure diagram of the encoder part of the method of the present invention. DETAILED DESCRIPTION

[0023] The present invention proposes a cross-subject cognitive load EEG signal data enhancement method based on a generative adversarial network. The method takes source domain data and target domain calibration data as input, and uses the encoder layer to extract features of the EEG signal, which is convenient for learning load task-related information during training. While generating subject-independent samples, it also reduces the number of parameters and improves the model training efficiency and performance.

[0024] Figure 1 The overall flow chart of the method of the present invention can be decomposed into the following steps:

[0025] Step 1. Data preprocessing and segmentation of cognitive load EEG signals;

[0026] Step 2. Perform data enhancement of cognitive load EEG signals using the SIE-GAN method proposed in the present invention;

[0027] Step 3. Train the deep learning model based on the training set and validation set of cognitive load EEG signals;

[0028] Step 4. Test the model effect and evaluate the accuracy of identifying the EEG signal state of cognitive load across subjects.

[0029] The following is a detailed description of each step:

[0030] Step 1:

[0031] (1) The original cognitive load EEG signal samples were band-pass filtered using a 3rd-order 4-40 Hz Butterworth band-pass filter to filter out the signals in the required frequency band;

[0032] (2) The filtered cognitive load EEG signal was normalized by exponential moving mean, with the attenuation factor set to 0.999 to reduce the impact of numerical differences on the model effect;

[0033] (3) Before starting training, the preprocessed load EEG data set is divided; Figure 2 This is the flow chart of the cross-subject experimental data partitioning of the present invention. The present invention adopts the leave-one-subject-out (LOSO) method commonly used in the cross-subject field to conduct experiments, that is, the data of one subject is used as the target domain subject, and the data of other subjects are used as the source domain subjects; according to the common practice in other literature, the labels of 10% of the data in the target domain subjects are used as the target domain calibration data. The source domain subjects and 10% of the target domain calibration data are used as training samples, and the remaining 90% of the target domain data are used as test samples. And 80% of the data in the training samples are randomly selected as the training set, and the remaining 20% ​​of the data are used as the validation set.

[0034] Step 2:

[0035] In order to solve the problem that the original EEG signal depends on specific subjects, this paper proposes a cross-subject data enhancement method based on generative adversarial network (Subject-Invariant Encoder Generative Adversarial Network, SIE-GAN). The network structure of this method is as follows: Figure 3 As shown, it can be summarized into three parts: encoder, generator and discriminator. Each part is described in detail below:

[0036] (1) Encoder

[0037] The encoder of the present invention is responsible for extracting the spatiotemporal information in the EEG signal to obtain the feature vector. The network structure of the encoder is as follows: Figure 4 As shown. The encoder draws on the idea of ​​the classic shallow convolutional neural network ShallowConvNet and contains three core structures: time domain convolution, spatial domain convolution and average pooling. Among them, the time domain convolution extracts the time domain information of each channel in the EEG signal through a one-dimensional convolution along the time axis (the convolution kernel length is 25); the spatial domain convolution performs a one-dimensional convolution along the EEG signal channel axis (the convolution kernel length is equal to the number of leads), assigns weights to the signals of each lead, and realizes the extraction of spatial domain information in the EEG signal; the average pooling layer performs an average convolution calculation with a kernel length of 75 and a step length of 15 along the time axis to achieve compression of time domain information and has the effect of downsampling. The encoder provides the idea of ​​combining the time domain convolution layer and the spatial domain convolution layer, that is, extracting the information on the time axis and the spatial axis of the input signal respectively, replacing the traditional two-dimensional convolution structure, which not only effectively extracts the time domain and spatial domain information, but also effectively reduces the number of parameters.

[0038] (2) Generator

[0039] The generator is responsible for reconstructing the input feature vector (after passing through the Encoder) and generating EEG signal samples by deconvolution to the original signal dimension.

[0040] (3) Discriminator

[0041] The discriminator is responsible for judging the input feature vector (after passing through the Encoder) and obtaining the probability that the feature vector is from the target domain. For the EEG signal samples generated by the generator, they need to pass through an Encoder before being input into the discriminator. Ideally, the value obtained by the discriminator is 0.5, which means that the discriminator cannot determine whether the incoming feature vector is from the source domain EEG signal or the target domain EEG signal.

[0042] The above are the three main parts of the network module of the present invention. In order to enable the generator to generate EEG signals that are independent of the subject, it is necessary to introduce the loss function of the GAN training process:

[0043] During the GAN training process, the generator and discriminator are updated alternately, using the data input of the source domain and the target domain as true samples and false samples respectively. The loss function calculation formula of the discriminator is as follows:

[0044]

[0045] Where m represents the batch size, that is, the number of samples used each time the parameters are updated;

[0046] It means maximizing the output probability of the discriminator for the source domain data, so that it can identify the source domain data as true data; It means minimizing the output probability of the discriminator for the target domain data, so that it can identify the target domain data as false data; It means maximizing the output probability of the discriminator for the source domain sample generated data, so that it can identify the data generated by the generator as source domain data.

[0047] The loss function calculation formula of the generator is as follows:

[0048]

[0049] in By minimizing the probability that the generated samples are identified as target domain data, the generator generates data that makes it difficult for the discriminator to distinguish between the source domain and the target domain.

[0050] Introduction to the parameters during training: The batch size is set to 64, the generator and discriminator are trained for 200 rounds, the learning rate of the generator is 1e-4, and the learning rate of the discriminator is 8e-5.

[0051] Step 3:

[0052] The cognitive load EEG signal data of the training set and the validation set were input into the deep learning model for training; the number of iterations in the training phase was set to 500, the batch size of the batch training was set to 64, and the Adam method was used as the optimizer; the model was saved when the validation set loss function reached the minimum; the model with the lowest validation set loss function during the training process was recorded, and used to predict the test set samples, and the test set accuracy was recorded;

[0053] Step 4:

[0054] The test set in step 1 is input into the trained model in step 3 for classification and recognition, and the accuracy of classification is evaluated; the data set and experimental results used in the method of the present invention are described as follows:

[0055] (1) Dataset

[0056] The present invention uses a self-collected data set, which contains EEG signals of cognitive load of 7 subjects performing flight tasks; these EEG signals have a total of 69 channels and an EEG signal sampling rate of 1000 Hz. During the data loading stage, 9 useless channels will be screened out, 60 channels will be retained, and the EEG signal sampling rate will be downsampled to 250 Hz.

[0057] (2) Experimental results

[0058] This experiment uses the leave-one-subject-out (LOSO) method, which is the most commonly used method in cross-subject experiments. In this experiment, these seven subjects are used as the target domain in turn, and the remaining six subjects are used as the source domain. 90% of the data in the target domain is used as the test set, and the remaining data and the source domain data are used as the training set. The average classification accuracy of the seven target domain subjects under high and low loads is used as the performance measurement indicator.

[0059] Table 1 shows the LOSO experimental results under different models and settings. It can be seen from the table that the cross-subject classification accuracy of the SIE-GAN data enhancement method proposed in the present invention on multiple deep learning models is higher than that without the data enhancement method.

[0060] Table 1 LOSO comparison experiment

[0061]

Claims

1. A method for enhancing EEG signal data under cognitive load across subjects based on generative adversarial networks, characterized in that: The steps include: Step 1. Preprocessing of cognitive load EEG signal data: Use a Butterworth bandpass filter to perform bandpass filtering on the original cognitive load EEG signal samples, and then perform moving mean normalization on the filtered signals; divide the preprocessed data of multiple subjects into source domain subjects and target domain subjects, divide the source domain subjects and 10% of the target domain calibration data into the training set, and the remaining 90% of the target domain subject data as the test set; Step 2. Cross-subject data enhancement of cognitive load EEG signals: The source domain data and target domain calibration data in the training set are put into the SIE-GAN data enhancement method of the present invention, and the EEG signals are feature extracted through the Encoder module. During the adversarial training process, the generator is guided by the loss function to generate EEG signals that are more in line with the subject independence. Step 3. Verify that the cross-subject data augmentation method improves the generalization ability of the model: Input the training set and validation set in step 1 into the deep learning model for training, and identify the corresponding state of the cognitive load EEG signal of the test set in step 1 based on the trained classification model.

2. The cross-subject data enhancement method based on a generative adversarial network according to claim 1, characterized in that: Step 1 is as follows: (1) The original cognitive load EEG signal samples were band-pass filtered using a 3rd-order 4-40 Hz Butterworth band-pass filter to filter out the signals in the required frequency band; (2) The filtered signal is normalized by exponential moving mean, where the attenuation factor is set to 0.999 to reduce the impact of numerical differences on the model effect; (3) Before starting training, the preprocessed EEG dataset is divided; the data of one subject is used as the target domain subject, and the data of other subjects are used as the source domain subjects; The labels of 10% of the target domain subjects are taken as known; the source domain subjects and 10% of the known target domain subjects are taken as training samples, and the remaining 90% of the target domain subjects are taken as test samples. 80% of the data are randomly selected from the training samples as the training set, and the remaining 20% ​​of the data are taken as the validation set.

3. The cross-subject data enhancement method based on generative adversarial network according to claim 1, characterized in that: Step 2 is as follows: A cross-subject data enhancement method SIE-GAN based on generative adversarial network is proposed. The network structure of this method consists of three parts: encoder, generator and discriminator. (1) Encoder; The encoder extracts the spatiotemporal information in the cognitive load EEG signal to obtain the feature vector. The encoder draws on the idea of ​​the classic shallow convolutional neural network ShallowConvNet and contains three core structures: time domain convolution, spatial domain convolution and average pooling. Among them, the time domain convolution extracts the time domain information of each channel in the cognitive load EEG signal through a one-dimensional convolution along the time axis of the EEG signal; the spatial domain convolution performs a one-dimensional convolution along the EEG signal channel axis to assign weights to the signals of each channel to realize the extraction of spatial domain information in the cognitive load EEG signal; the average pooling layer performs average convolution calculation along the time axis to realize the compression of time domain information and downsampling; the encoder combines the time domain convolution layer and the spatial domain convolution layer to extract the information on the time axis and space axis of the input EEG signal respectively, replacing the two-dimensional convolution structure, extracting time domain and spatial domain information and reducing the number of parameters; (2) Generator; The generator reconstructs the input feature vector and deconvolves it to the original signal dimension to generate EEG signal samples; (3) Discriminator; The discriminator is responsible for judging the input feature vector and obtaining the probability that the feature vector is from the target domain. For the cognitive load EEG signal samples generated by the generator, they need to pass through an Encoder and then be input into the discriminator. The value obtained by the discriminator is 0.5, which means that the discriminator cannot determine whether the incoming feature vector is from the source domain EEG signal or the target domain EEG signal.

4. The cross-subject data enhancement method based on generative adversarial network according to claim 3, characterized in that: During the GAN training process, the generator and discriminator are updated alternately, using the data input of the source domain and the target domain as true samples and false samples respectively. The loss function calculation formula of the discriminator is as follows: Where m represents the batch size, that is, the number of samples used each time the parameters are updated; i is the batch number; It means maximizing the output probability of the discriminator for the source domain data, so that it can identify the source domain data as true data; It means minimizing the output probability of the discriminator for the target domain data, so that it can identify the target domain data as false data; It means maximizing the output probability of the discriminator for the source domain sample generated data, so that it can identify the data generated by the generator as source domain data; The loss function calculation formula of the generator is as follows: in By minimizing the probability that the generated sample is identified as target domain data, the generator generates data that makes it difficult for the discriminator to distinguish between the source domain and the target domain; Parameters during training: The batch size is set to 64, the generator and discriminator are trained for 200 rounds, the learning rate of the generator is 1e-4, and the learning rate of the discriminator is 8e-5.

5. The cross-subject data enhancement method based on generative adversarial network according to claim 1, characterized in that: Step 3 is as follows: The training set and validation set of cognitive load EEG signals were input into the deep learning model for training; the number of iterations in the training phase was set to 500, the batch size of batch training was set to 64, and the Adam method was used as the optimizer; the model was saved when the validation set loss function reached the minimum; the model with the lowest validation set loss function during the training process was recorded and used to predict the test set samples; the test set in step 1 was input into the trained model for classification and recognition, and the classification accuracy was evaluated.

6. The cross-subject data enhancement method based on generative adversarial network according to claim 5, characterized in that: The experiment adopts the leave-one-subject method. In the experiment, 7 subjects are used as the target domain in turn, and the remaining 6 subjects are used as the source domain. 90% of the data in the target domain is used as the test set, and the remaining data and the source domain data are used as the training set.

Images