A method of EEG emotion recognition based on generative adversarial network data enhancement

Abstract

The present invention discloses an EEG emotion recognition method based on data enhancement of a generative adversarial network. First, the length of the EEG sample, the length of the slice within the sample and the slice window shift are determined, the EEG sample is sliced, and decomposed into four different frequency bands; next, the EEG signal features on each channel frequency band are calculated, all the calculated features are spliced ​​into a one-dimensional feature vector and input into a generative adversarial network for training. After the training is completed, the network generates a large number of high-quality EEG samples, and then the generated EEG feature vector and the original EEG feature vector are rearranged according to the position information of the electrode placement during EEG signal acquisition, and spliced ​​according to the frequency band to construct a three-dimensional EEG feature map, which is then sent to a continuous convolutional neural network for training, and finally a softmax classifier is used for emotion classification. The present invention can maintain good efficiency and robustness in the scenario of few samples of EEG emotion recognition.

Description

Technical Field

[0001] The present invention relates to the technical field of electroencephalogram emotion recognition, and in particular to an electroencephalogram emotion recognition method based on generative adversarial network data enhancement. Background Art

[0002] Emotions are fundamental in human daily life because they play an important role in human cognition, namely perception, rational decision-making, human interaction, and human intelligence. With the development of artificial intelligence technology and deep learning, emotion recognition has broad prospects in human-computer interaction and clinical treatment, which has been widely concerned by researchers. Human emotions can be recognized through speech, blinking, facial expressions, and physiological signals. But the first three methods are unstable and easily affected by subjectivity. Subjects may deliberately hide their emotions and cause recognition errors. Physiological signals such as electrooculogram (EOG), electroencephalogram (EEG), and blood pressure (BVP) are all spontaneously generated by the human body. Therefore, physiological signals can more accurately reflect people's emotional state. Among these physiological signals, electroencephalogram (EEG) is a holistic reflection of the electrophysiological activity of the cerebral cortex or scalp surface, indicating that changes in EEG signals can be used to characterize changes in human emotions.

[0003] The process of applying EEG signals to emotion recognition includes emotion induction, EEG signal acquisition, EEG signal preprocessing, feature extraction and selection, and emotion classification. In these processes, effective feature extraction and selection of the final emotion classifier are the two most important steps. Researchers are interested in EEG signals in four frequency bands, including Theta (about 4-7Hz), Alpha (about 8-12Hz), Beta (about 13-30Hz) and Gamma (about 31-100Hz). Basically, the EEG features on each frequency band can be time domain features, frequency domain features, or time-frequency domain features. Based on the obtained EEG features, a large number of emotion recognition models are constructed.

[0004] In recent years, many researchers have established emotion recognition models based on machine learning and deep learning models. Machine learning models generally use support vector machines, decision trees, random forests, etc.; deep learning models generally use convolutional neural networks, recurrent neural networks, etc. According to current research results, deep learning models have higher accuracy in emotion recognition tasks.

[0005] Based on the outstanding performance of deep learning in EEG emotion recognition tasks, deeper deep learning models are needed to promote the exploration of emotion recognition. However, deep learning models are data-driven and can only have good performance when there is more data, which requires a large number of labeled training samples for the training of deep learning models. However, due to the high cost, for most public EEG datasets, they only have a small number of samples, and these sample categories are extremely unbalanced. Insufficient sample size and class imbalance will lead to model overfitting, which seriously affects the recognition accuracy. Therefore, it is very important to obtain enough training samples and improve the inter-class balance of samples. Generative adversarial networks can use the existing small number of EEG samples to generate a large number of high-quality EEG samples with probability distribution similar to the probability distribution of the original data, which can solve the problems of sample class imbalance and small sample size, and will greatly improve the EEG emotion recognition rate and robustness. Summary of the invention

[0006] The purpose of the present invention is to provide an EEG emotion recognition method based on generative adversarial network data enhancement, which solves the problems of insufficient samples and imbalance between sample classes in the training process of EEG emotion recognition model, and improves the accuracy and recall rate of emotion model recognition. This method can enhance the feature description ability of different categories of emotion recognition model, so that the model has higher accuracy and better robustness.

[0007] The specific technical solution adopted by the present invention is as follows:

[0008] A method for EEG emotion recognition based on generative adversarial network data enhancement includes the following steps:

[0009] Step 1: Preprocessing of raw EEG signals;

[0010] Setting the EEG sample length, the slice length within the EEG sample, and the EEG sample slice window shift, slicing the EEG signal to obtain EEG sample slices;

[0011] For each lead in the EEG sample slice, the power spectral density features are calculated by dividing it into four different bands, and all the features calculated for each lead are concatenated into a one-dimensional vector.

[0012] Step 2: Construction and training of the adversarial generative model; establish a GAN generative adversarial network data enhancement model and input the preprocessed EEG signals into the model for training.

[0013] Step 3: EEG feature vector processing;

[0014] All sample data are z-score standardized to avoid the impact caused by the difference between the units and values ​​in the original data;

[0015] The normalized feature vectors are rearranged according to the position information of the electrodes when the EEG signals are collected, and spliced ​​according to the frequency bands to construct a three-dimensional EEG feature map;

[0016] Step 4: Emotion classification: Use the real data and generated data to form a new balanced and larger dataset, and use the continuous convolutional neural network as an EEG emotion classifier for classification.

[0017] Furthermore, the length of the EEG signal is set in the original EEG signal preprocessing performed in step 1, and the specific implementation method is as follows;

[0018] The original EEG signal is cut off to remove the baseline signal part of the EEG signal, and only the experimental signal part with emotional changes is retained.

[0019] Furthermore, for the slice length and window shift in the original EEG signal preprocessing in step 1, the generative adversarial network needs to be trained under a certain number of samples. Therefore, the present invention uses sample overlap to amplify the data with an overlap rate of 50%.

[0020] Furthermore, the GAN generative adversarial network model in step 2 mainly includes a generator G and a discriminator D;

[0021] The generator G inputs a set of one-dimensional randomly distributed noise and outputs a set of new samples that conform to the probability distribution of the original EEG samples;

[0022] The discriminator D is a binary classifier, and the input of the discriminator includes the original data and the data generated by the generator G.

[0023] The output is a probability value;

[0024] The adversarial principle of the GAN generative adversarial network data enhancement algorithm is as follows: GAN, as its name describes, consists of a generator and a discriminator. The generator receives a random noise signal (which can be uniformly distributed or Gaussian distributed), and then generates corresponding samples. The discriminator receives samples generated by the generator and samples from real samples. The main function of the discriminator is to judge the authenticity of the samples. It will give the real samples the greatest possible probability and the generated samples the smallest possible value (the greater the probability, the more likely it is a real sample). The generator continuously strengthens its ability, making the generated samples closer and closer to the real samples, that is, the discriminator can no longer distinguish whether the samples are real. By continuously iterating the above process, the discriminator cannot distinguish whether the received samples are from real samples or generated samples. Through adversarial training, the samples generated by the generator are closer to the original data samples, effectively improving the quality of data enhancement.

[0025] The GAN data enhancement model structure of the present invention is mainly divided into two parts: the generator and the discriminator, which are all composed of three layers of neural network, in which the neural network input layer, hidden layer, and output layer are all composed of linear layers, and the original data enhancement work that meets all situations is achieved through linear layer transformation and nonlinear transformation of activation function;

[0026] Furthermore, the generation principle of the GAN generative adversarial network data enhancement model is the maximum and minimum game, where the objective function is:

[0027]

[0028] Where E represents the expected value of the distribution function, x is the original sample data, and z is the noise data of the data generator G.

[0029] A further step 2 comprises the following steps:

[0030] Step 2.1: Initialize the parameters of the two neural networks, generator G and discriminator D;

[0031] Step 2.2: Extract n samples from the original data, and the generator generates n samples using the defined noise distribution, fix the generator G, and train the discriminator D so that the discriminator can distinguish between true and false;

[0032] Step 2.3: In the iterative training, the discriminator D and the generator G compete with each other. Ideally, the discriminator D cannot distinguish whether the data comes from the original data or the data generated by the generator G. The discrimination probability is 0.5 at this time, and the training is completed.

[0033] Noise data is input into the trained generator model. Through the transformation of the three-layer neural network of the generator G, the model generates sample data that is close to the distribution of the original data, thereby expanding the original data.

[0034] Furthermore, an activation function is added after each linear layer in the generator G neural network and the discriminator network. If there is no activation function, it will eventually be equivalent to a linear function. Therefore, the ReLU activation function is introduced in this algorithm, and its formula is as follows:

[0035] f(x)=max(0,x)

[0036] Furthermore, for the GAN generative adversarial network data enhancement algorithm, an Adam optimizer is used. Adam optimization comprehensively considers the first-order moment and the second-order moment, and has the advantages of simple implementation, high computational efficiency, and good effect.

[0037] Furthermore, in step 3, the generated data and the original data are standardized, and the formula is as follows:

[0038]

[0039] Where μ is the mean of all sample data, and σ is the standard deviation of all sample data. The result after transformation is the result of data standardization, and the data enhancement effect after standardization is better.

[0040] Furthermore, for step 3, the standardized feature vectors are rearranged according to the EEG emotion features of each electrode channel in different frequency bands according to the position of EEG signal acquisition, and then spliced ​​according to the frequency band to form a three-dimensional EEG feature map. In the present invention, the feature values ​​of all leads in the same band are placed in a 9*9 two-dimensional square matrix according to the distribution mode on the EEG acquisition device, and all lead feature matrices are stacked according to four bands to form a three-dimensional 4*9*9 matrix as the input of the emotion model.

[0041] The beneficial effects of the present invention are as follows:

[0042] 1. The present invention generates a large number of high-quality EEG samples through a generative adversarial network, which can solve the problems of imbalance between sample classes and insufficient training samples in the current field of EEG emotion recognition, thereby improving the accuracy of EEG-based emotion classification.

[0043] 2. Compared with upsampling and copying samples to expand data, the data generated by the GAN generative adversarial network data enhancement model can better reflect the characteristics of its original distribution, and the generated data is more convenient and simple, and the scale of the generated data can be freely determined.

[0044] 3. Use the generative adversarial training method to make the data generated by the noise data after passing through the generator G neural network closer to the original data, effectively improving the quality of enhancement. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 It is the overall flow chart of the present invention.

[0046] Figure 2 This is a schematic diagram of the GAN generation adversarial network data enhancement algorithm in the present invention.

[0047] Figure 3 This is a diagram of EEG preprocessing and feature vector processing in the present invention.

[0048] Figure 4 This is a continuous convolutional neural network diagram in the present invention. DETAILED DESCRIPTION

[0049] In order to deepen the understanding of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. The embodiments are only used to explain the present invention and do not limit the protection scope of the present invention.

[0050] Embodiment: A method for EEG emotion recognition based on generative adversarial network data enhancement solves the problems of insufficient samples and imbalance between sample classes in the field of EEG emotion recognition.

[0051] First, the EEG sample length, the slice length within the sample and the slice window shift are determined, the EEG sample is sliced ​​and decomposed into four different frequency bands; next, the EEG signal features on each channel frequency band are calculated, and all the calculated features are spliced ​​into a one-dimensional feature vector and input into the generative adversarial network for training. After the training is completed, the network generates a large number of high-quality EEG samples, and then the generated EEG feature vectors and the original EEG feature vectors are rearranged according to the position information of the electrodes during EEG signal acquisition, and spliced ​​according to the frequency band to construct a three-dimensional EEG feature map, which is then sent to the continuous convolutional neural network for training, and finally the softmax classifier is used for emotion classification.

[0052] Specific steps are as follows Figure 1 As shown:

[0053] Step 1: Preprocessing of raw EEG signals:

[0054] The original EEG signal is cut off to remove the baseline signal part of the EEG signal, and only the experimental signal part with emotional changes is retained.

[0055] The EEG sample length, the slice length within the EEG sample and the EEG sample slice window shift are set to slice the EEG signal to obtain EEG sample slices; each lead in the EEG sample slice is divided into four different bands to calculate the power spectral density features, and all the features calculated for each lead are spliced ​​into a one-dimensional vector.

[0056] This embodiment selects a suitable EEG emotion recognition length of 2 seconds, which is divided into windows of 1 second; then, for each lead in the sample, the power spectrum features are calculated for 4 different bands respectively; all the calculated eigenvalues ​​are organized into a one-dimensional vector of 4*9*9 eigenvalues ​​in the order of frequency bands as the input of the adversarial generative network.

[0057] Step 2: Building and training the adversarial generative model; establishing a GAN generative adversarial network data enhancement algorithm, and inputting the preprocessed one-dimensional vector of the original EEG into the model for training.

[0058] This embodiment selects a set of data sets of size [64,256] from the target data set, indicating that the set has 64 samples, each sample is 2 seconds long, and there are 256 sampling points in total. After processing in step 1, the size is obtained to be [64,128]. The size of all noise inputs of the generator G in the GAN generation adversarial network is set to 16, and the input size of the discriminator D is 128. The generator output size is set to 128, and the discriminator output size is set to 1, which is the probability value that the generated data conforms to the real data. The generation principle of the GAN generation adversarial network enhancement model is the maximum and minimum game, in which the objective function is:

[0059]

[0060] Where E represents the expected value of the distribution function, x is the original sample data, and z is the noise data of the data generator G.

[0061] Step 2.1: Initialize the parameters of the two neural networks: generator G and discriminator D;

[0062] Initialize the weights and biases in the generator G and discriminator D neural networks

[0063] Step 2.2: Extract n samples from the original data, and the generator uses the defined noise distribution to generate n samples. Fix the generator G and train the discriminator D so that the discriminator can distinguish between true and false.

[0064] 64 samples are extracted from the original data, and the number of hidden layers of the discriminator and generator is set to 64.

[0065] During each iteration:

[0066] The training sample size is [64, 128], where 64 means that 64 samples are used as a group for training each time. 128 means the combined result of the power spectrum features calculated for different frequency bands of all leads.

[0067] Select a set of random variables from a random distribution (Gaussian, Sentient, etc.), with the dimension set to 16;

[0068] 1. Send the one-dimensional vector to the generator G neural network, and finally obtain a set of generated data with a dimension of 128;

[0069] 2. Update the parameters of the discriminator D so that the discriminator judges the original sample as true and the generated sample as false, and the generator. Update the parameters of the generator so that the samples generated by the generator can be judged as true by the discriminator.

[0070] Step 2.3: In the iterative training, the discriminator and the generator compete with each other. In the rational state, the discriminator D cannot distinguish whether the data comes from the original data or the data from the generator G. The discrimination probability is 0.5 at this time, and the training is completed.

[0071] During each iteration:

[0072] 1. Fix the generator G and only update the parameters of the discriminator D. Input the data of the samples generated by the generator and the sample data in the original data into the discriminator D. The goal of D in the discriminator is that if the input comes from the real data set, that is, the larger the output value, the more real it represents. On the contrary, the smaller the value, the more false the input.

[0073] 2. Fix the parameters of the discriminator D, update the parameters of the generator G, input a set of noise vectors into the generator G, get a set of outputs, input its outputs into the discriminator D, and you will get a numerical value. At this stage, the parameters of the discriminator D are fixed, and the generator G needs to update its own parameters so that the value is as large as possible.

[0074] Step 3: EEG feature vector processing: standardize the generated data and the original data. The formula is as follows:

[0075]

[0076] Where μ is the mean of all sample data, and σ is the standard deviation of all sample data. The result after transformation is the result of data standardization, and the data enhancement effect after standardization is better.

[0077] The standardized feature vectors are rearranged according to the EEG emotion characteristics of each electrode channel in different frequency bands according to the location of EEG signal collection, and put into a 9*9 two-dimensional square matrix. All lead feature square matrices are stacked according to four bands to form a three-dimensional 4*9*9 matrix as the input of the emotion model.

[0078] Step 4: Sentiment classification; use a continuous convolutional neural network, in which all convolutional layers are stacked together continuously, with no pooling layer in the middle, and only the activation function and batch normalization layer are connected. This network does not have a pooling layer and does not lose information in the small-sized three-dimensional input matrix.

[0079] In this implementation example, the data distribution of the EEG signal after data enhancement is almost consistent with the target original data distribution. Adding the generated samples to the training set improves the effect by about 10% on average compared to using only the original data as the data set, indicating that the algorithm can be used for data enhancement in EEG emotion recognition, and effectively solves the problems of small data volume and imbalance between classes.

[0080] The present invention preprocesses the EEG samples. The preprocessing process is not complicated. It mainly includes removing the baseline signal, selecting the window shift and sample length for segmentation, and then calculating the power spectrum density eigenvalues ​​of different leads and different frequency bands through frequency band separation, and forming all eigenvalues ​​into a one-dimensional vector. The data is then sent to the generative adversarial network for data generation. The adversarial generative network data enhancement algorithm is mainly divided into two parts: the generator and the discriminator. The network structure is simple, mainly three linear layers, and each linear layer will have an activation function at the end, which greatly reduces the number of network parameters and reduces the difficulty of training. Using adversarial training, the noise data is reconstructed by the generator G to be closer to the original target data, effectively improving the EEG emotion recognition rate.

[0081] The above shows and describes the basic principles, main features and advantages of the present invention. It should be understood by those skilled in the art that the present invention is not limited to the above embodiments. The above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A method for EEG emotion recognition based on generative adversarial network data enhancement, characterized in that: The following steps are involved: Step 1: Preprocessing of raw EEG signals: Setting the EEG sample length, the slice length within the EEG sample, and the EEG sample slice window shift, slicing the EEG signal to obtain EEG sample slices; For each lead in the EEG sample slice, the power spectral density features are calculated in four different bands, and all the features calculated for each lead are concatenated into a one-dimensional vector; Step 2: Building and training the adversarial generative model: Building a GAN generative adversarial network data enhancement algorithm, and inputting the preprocessed EEG signals into the model for training; Step 3: EEG feature vector processing: All sample data are z-score normalized to avoid the impact caused by the difference between the unit and the value in the original data; the normalized feature vectors are rearranged according to the position information of the electrode placement during EEG signal acquisition, and spliced ​​according to the frequency band to construct a three-dimensional EEG feature map; Step 4: Emotion classification: Use the real data and generated data to form a new balanced and larger data set, and use the continuous convolutional neural network as an EEG emotion classifier for classification; The specific implementation process of step 1 is as follows: select the EEG emotion recognition length L, and divide it with L / 2 as the window shift; then for each lead in the sample, calculate the power spectrum features for 4 different bands respectively, and combine all the calculated eigenvalues ​​into a one-dimensional vector of 4*9*9 eigenvalues ​​in the order of frequency bands as the input of the adversarial generation network; The GAN generative adversarial network model in step 2 includes two parts: a generator model G and a discriminator model D; all of which are composed of a three-layer neural network, including a neural network input layer, a hidden layer, and an output layer. The three layers of the neural network are all composed of linear layers, which are transformed by linear layers and nonlinear transformations of activation functions; the GAN generative adversarial network principle is the maximum and minimum game between the generator G and the discriminator D, and its objective function is: Where E represents the expected value of the distribution function, x is the original sample data, and z is the noise data of the data generator G.

2. The method for EEG emotion recognition based on generative adversarial network data enhancement according to claim 1 is characterized in that: An activation function is added after each linear layer in the generator G neural network and the discriminator D neural network. The ReLU activation function is introduced in this algorithm, and its formula is as follows: f(x)=max(0,x).

3. The method for EEG emotion recognition based on generative adversarial network data enhancement according to claim 2 is characterized in that: The step 2 comprises the following steps: Step 2.1: Initialize the parameters of the two neural networks, generator G and discriminator D; Step 2.2: Extract n samples from the original data, and the generator generates n samples using the defined noise distribution, fix the generator G, and train the discriminator D so that the discriminator can distinguish between true and false; Step 2.3: In the iterative training, the discriminator D and the generator G compete with each other. Ideally, the discriminator D cannot distinguish whether the data comes from the original data or the data generated by the generator G. The discrimination probability is 0.5 at this time, and the training is completed.

4. The method for EEG emotion recognition based on generative adversarial network data enhancement according to claim 1 is characterized in that: The specific implementation process of step 3 is as follows: Step 3.1: Standardize all samples according to the z-score, the formula is as follows: Where μ is the mean of all sample data, σ is the standard deviation of all sample data; Step 3.2: The standardized feature vectors are rearranged according to the EEG emotion characteristics of each electrode channel in different frequency bands according to the location of EEG signal collection, and then spliced ​​according to the frequency band to form a three-dimensional EEG feature map.

5. The method for EEG emotion recognition based on generative adversarial network data enhancement according to claim 1 is characterized in that: In the step 4 continuous convolutional neural network, all convolutional layers are stacked together continuously, with no pooling layer connected in the middle, and only the activation function and batch normalization layer are connected.

Images