A CNN-based emotional EEG classification method

Abstract

The present invention discloses a method for classifying emotional EEG based on CNN. The present invention adopts the Russell emotional dimension model in the continuous emotional dimension model as a benchmark, uses the DEAP data set as a sample, removes the baseline of the emotional EEG therein, and then normalizes the data, extracts the Pearson coefficient of the three frequency bands of the EEG, converts it into a 2D image format, and uses SBS to screen experiments that are valuable for emotional EEG classification in units of experiments, and inputs the screened experimental data into CWAGAN‑GP for data enhancement to supplement the training set, and then inputs the integrated convolutional neural network in the form of frames. This method can effectively classify emotional EEG signals, provides considerable classification accuracy, and the operation of inputting the integrated convolutional neural network in the form of frames can effectively prevent the convolutional neural network from overfitting.

Description

Technical Field

[0001] The present invention relates to an emotional EEG classification method, and in particular to a method for classifying emotional EEG using CNN after optimizing the preprocessing. Background Art

[0002] Convolutional neural network (CNN) is a feedforward neural network that performs well in large-scale image processing and has been widely used in image classification, positioning and other fields. Compared with other neural network structures, convolutional neural network requires relatively fewer parameters, making it widely used.

[0003] Generative adversarial networks (GANs) produce fairly good outputs through the mutual game learning of the two modules in the framework: the generative model and the discriminative model. They can generate realistic new data based on the original data set. WGAN-GP was proposed to address the existing problems of WGAN. In real experiments, WGAN still has problems with training difficulties and slow convergence speed. Compared with traditional GANs, the experimental improvement is not very obvious. The problem with WGAN is that it directly uses clipping weights when dealing with continuity constraints. That is, every time the parameters of the discriminator are updated, it is checked whether the absolute values ​​of all the parameters of the discriminator exceed a threshold, such as 0.01. If so, these parameters are returned to the range of [-0.01, 0.01]. In this case, the best strategy is to make all parameters as extreme as possible, either taking the maximum value (such as 0.01) or the minimum value (such as -0.01). For deep neural networks, it cannot give full play to the fitting ability of deep neural networks. In addition, it is also found that forced clipping of weights can easily lead to gradient vanishing or gradient explosion. The reasons for gradient vanishing and gradient explosion are the selection of clipping range. If the selection is too small, the gradient will disappear. If it is set a little larger, the gradient will become a little larger after each layer of the network, and gradient explosion will occur after multiple layers. In order to solve this problem and find a suitable way to meet the continuity condition, WGAN-GP uses gradient penalty to meet this continuity condition. CWGAN-GP adds conditional constraints, namely labels, on the basis of WGAN-GP, so that it can generate data with specified labels.

[0004] Sequential backward selection (SBS) starts from the full feature set A, and each time a feature is removed from the feature set A, the evaluation function value reaches the optimal value.

[0005] Convolutional neural networks are very effective models for image classification tasks. For such models, this study attempts to convert DEAP data into 2D image format so that CNN models can learn how to classify them effectively. Summary of the invention

[0006] The present invention adopts the Russell emotion dimension model in the continuous emotion dimension model as a benchmark, uses the DEAP data set as a sample, performs preprocessing data normalization, extracts the Pearson coefficients of the three frequency bands most relevant to emotion recognition in the EEG channel, and converts them into a 2D image format. The SBS is used to screen experiments with value for emotion EEG classification in units of experiments, and the screened experimental data is input into CWAGAN-GP for data enhancement to supplement the training set, and finally input into the convolutional neural network classification. This method can effectively classify emotional EEG signals, screen data, and enhance valuable data, which can effectively improve the accuracy.

[0007] The present invention uses CNN to classify emotional EEG after preprocessing, which specifically includes the following steps:

[0008] Step 1. Establish an emotion dimension model represented by Russell, and label the evaluation index labels in the data set according to the required categories.

[0009] Step 2. Extract the baseline data of the first 3 seconds of each experiment, divide it into segments according to the selected time window, add it and then average it, then divide the remaining EEG data into segments according to the selected time window and subtract the processed baseline data to obtain the difference value of the EEG data, and then normalize the data. Extract the three frequency bands most relevant to emotion recognition from the difference value of the EEG data, calculate the Pearson coefficient of these three frequency bands, and convert them into image format.

[0010] Step 3. For each subject, take the experiment as the unit, convert the data of each subject into the number of experiments * number of segments * 32 * 32 * 3, set the remaining number after screening, send it to SBS to screen out the experimental combination with the highest accuracy, and send it to CWGAN-GP for data enhancement through the labeled generative adversarial network. After screening, some categories of emotions will be reduced, so as to supplement the data and balance the emotional data of each category.

[0011] Step 4. Allocate the processed data and labels in a ratio of 8:2 between training samples and test samples, and input them into the integrated convolutional neural network in the form of frames to perform classification tests on the processed samples.

[0012] In the step 1, when the Russell emotional dimension model is used as a reference, each emotional state is placed on a two-dimensional plane, and the arousal and pleasure constitute the two dimensions of the coordinate axis. Valence represents the subjective evaluation of the subject (person) on the emotion, which changes from negative emotion to positive emotion on the number axis; arousal represents the degree to which the subject (person) feels the emotion, which changes from calmness to excitement on the number axis. The values ​​of both dimensions vary from 1 to 9, and the magnitude of the values ​​is used to represent the degree of emotion. When the emotion binary classification is selected, the evaluation values ​​on valence and arousal are divided with the median 5 as the threshold, and those greater than 5 are marked as 1, representing high valence or high arousal (HV, HA); those less than or equal to 5 are marked as 0, representing low valence or low arousal (LV, LA); when the emotion four-category classification is selected, HV, HA, LV, and LA can be combined to generate four types of labels: HVHA, HVLA, LVHA, and LVLA.

[0013] In step 2, the EEG signal is converted into a 2D image format, and the specific steps are:

[0014] 2-2. For 8-12 Hz (α band), 12-30 Hz (β band), and 30-60 Hz (γ band), a fourth-order Butterworth filter is used to filter the signal.

[0015] 2-3. For each frequency band, calculate the Pearson coefficient between 32 EEG channels and each pair of channels, convert it into a 3*3*1 format, and then combine the results of the three frequency bands to form a 32*32*3 2D image format.

[0016] In step 4, the processed data and labels are distributed in a ratio of 8:2 between training samples and test samples, and are input into the integrated convolutional neural network in the form of frames to perform classification tests on the processed samples. Specifically:

[0017] The time window size is set to 1s, and a total of 40*60*32=76800 samples are segmented. For each experiment's 60 segments, every 3 segments, or samples every 3 seconds, are combined into a frame. There are no repeated samples between frames. One experiment can generate 20 frames, and the total number of samples becomes 40*20*32=25600. The sample format is 25600*3*32*32*3. Three pictures are input to three convolutional neural networks at the same time each time, and the average of the three output results is the final result of this input. All samples are divided into training samples and test samples at a ratio of 8:2.

[0018] The integrated convolutional neural network is composed of three convolutional neural networks with exactly the same structure, and the structure of each convolutional neural network is:

[0019] The C1 layer is a convolutional layer with RELU, with 32 filters, the convolution kernel is set to 3*3, and the stride is 1*1.

[0020] The S2 layer is an average pooling layer with a size of 2*2 and a stride of 2*2.

[0021] The B3 layer is a BN layer.

[0022] The C4 layer is also a convolutional layer with RELU, with 64 filters, the convolution kernel is set to 3*3, and the stride is 1*1.

[0023] The S5 layer is an average pooling layer with a size of 2*2 and a stride of 2*2.

[0024] The B6 layer is a BN layer.

[0025] The C7 layer is a convolutional layer with RELU, with 128 filters, the convolution kernel is set to 3*3, and the stride is 1*1.

[0026] The D8 layer is a fully connected layer with an output dimension of 128.

[0027] The DO9 layer is a Dropout layer, which randomly turns off some neurons with a probability of 0.5 during training.

[0028] Finally, the output dimension of the final connected Softmax layer is 2.

[0029] The beneficial effects of the present invention are as follows:

[0030] The present invention adopts the Russell emotion dimension model in the continuous emotion dimension model as a benchmark, uses the DEAP data set as a sample, removes the baseline of the emotional EEG, and then performs data normalization to extract the Pearson coefficient of the three frequency bands of the EEG, converts it into a 2D image format, and then uses SBS data screening and data enhancement, and then inputs it into the integrated convolutional neural network in the form of frames, which can effectively improve the accuracy of emotion classification and prevent the convolutional neural network from overfitting. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 It is the Russell emotion dimension model;

[0032] Figure 2 The data type of the DEAP dataset;

[0033] Figure 3 Schematic diagram of EEG data preprocessing;

[0034] Figure 4 This is the basic structure diagram of the CNN network;

[0035] Figure 5Flowchart for emotional EEG classification; DETAILED DESCRIPTION

[0036] The present invention is further described below in conjunction with specific embodiments. The following description is only for demonstration and explanation and does not limit the present invention in any form.

[0037] like Figure 5 As shown, a method for classifying emotional EEG based on CNN, the method specifically includes the following steps:

[0038] Step 1. Figure 1 As shown, the Russell emotion dimension model in the continuous emotion dimension model is established, and the evaluation indicator labels in the data set are labeled according to the required categories.

[0039] The Russell emotional dimension model is a continuous two-dimensional emotional space with arousal and valence as coordinate axes. Valence represents the subjective evaluation of emotions by the subject (person), ranging from negative emotions to positive emotions on the numerical axis; arousal represents the degree to which the subject (person) feels emotions, ranging from calm to excitement on the numerical axis. The values ​​of both dimensions vary from 1 to 9, and the magnitude of the values ​​represents the degree of emotion. When the emotion binary classification is selected, the evaluation values ​​of valence and arousal are divided with the median of 5 as the threshold. Values ​​greater than 5 are marked as 1, representing high valence or high arousal (HV, HA); values ​​less than or equal to 5 are marked as 0, representing low valence or low arousal (LV, LA); when the emotion quadrilateral classification is selected, HV, HA, LV, and LA can be combined to generate four types of labels: HVHA, HVLA, LVHA, and LVLA.

[0040] Step 2. Figure 2 , Figure 3As shown in the figure, this method only uses the EEG data in the DEAP data set as the data set. The original data format of each subject is 40*40*8064 (number of experiments × number of channels × data points), and the data of 8 peripheral physiological signals are deleted. The sampling points of each channel are 63*128=8064, which are divided into 3 seconds of baseline data and 60 seconds of test data. They are segmented into non-overlapping segments with a time window of 1 second. Each channel obtains 63 segmented samples. The baseline data samples of the first 3 seconds are added and averaged. It can be considered that the EEG data without emotional fluctuations under normal conditions is obtained. Then, the EEG data samples without emotional fluctuations are subtracted from the remaining EEG data samples to obtain the difference value of the EEG data. Then, the difference value of the EEG signal is z-score standardized. The number of samples obtained is 76800=32*40*60 (number of subjects × number of experiments × number of segments). The three frequency bands most relevant to emotion recognition are extracted from the difference values ​​of the EEG data, namely 8-12Hz (α band), 12-30Hz (β band), and 30-60Hz (γ band). A fourth-order Butterworth filter is used to filter each channel of the signal. Three frequency bands are extracted from each channel. For each frequency band, the Pearson coefficient (PCC) between the 32 EEG channels is calculated. The calculation formula is as follows:

[0041]

[0042] Where X and Y represent channels selected from 32 EEG channels.

[0043] By calculating the Pearson coefficient, it is converted into a 32*32*1 format, and then the results of the three frequency bands are combined to become a 32*32*3 2D image format.

[0044] Step 3. For each subject, take the experiment as the unit, convert the data of each subject into 40*60*32*32*3 (number of experiments × number of segments × pictures), set the number of remaining experiments to 30, and send them to SBS to screen out the experimental combination with the highest accuracy. Since some categories of emotions will be reduced after screening, the screened experimental data will be sent to CWGAN-GP for data enhancement through the labeled generative adversarial network to generate data with specified labels, so as to supplement the data and balance the emotional data of each category.

[0045] Step 4. For each of the 60 segments of each experiment, combine every 3 segments, i.e. every 3 seconds of samples, into one frame. There are no repeated samples between frames. One experiment can generate 20 frames, and the total number of samples becomes 40*20*32=25600. The sample format is 25600*3*32*32*3. All samples are divided into training samples and test samples at a ratio of 8:2. Input 3 pictures to 3 convolutional neural networks at the same time each time. The configuration of the convolutional neural networks is the same. The average of the 3 output results is the final result of this input. All samples are divided into training samples and test samples at a ratio of 8:2.

[0046] like Figure 4 As shown;

[0047] 4-1. The C1 layer is a convolutional layer with RELU, which has 32 filters, the convolution kernel is set to 3*3, and the step size is 1*1.

[0048] 4-2. The S2 layer is an average pooling layer with a size of 2*2 and a stride of 2*2.

[0049] 4-3. Layer B3 is a BN layer.

[0050] 4-4.C4 layer is also a convolutional layer with RELU, with 64 filters, the convolution kernel is set to 3*3, and the step size is 1*1.

[0051] 4-5. The S5 layer is an average pooling layer with a size of 2*2 and a stride of 2*2.

[0052] 4-6. Layer B6 is a BN layer.

[0053] 4-7.C7 layer is a convolution layer with RELU, with 128 filters, the convolution kernel is set to 3*3, and the step size is 1*1.

[0054] 4-8. Layer D8 is a fully connected layer with an output dimension of 128.

[0055] 4-9.DO9 layer is a Dropout layer, which randomly turns off some neurons with a probability of 0.5 during training.

[0056] 4-10. Finally, the output dimension of the final connected Softmax layer is 2.

[0057] Table 1 is a comparison of the binary classification effects of the traditional classification model based on the DEAP dataset and the Russell emotion dimension model:

[0058] Table 1 Comparison of classification accuracy of each classification model

[0059]

[0060] By using samples extracted in proportion as training samples and the remaining samples as test samples, the CNN model was input in the form of frames for binary classification based on the two criteria of pleasure and arousal. Remarkable results were obtained, with an average accuracy of 83.3% and 84.2%, which is a significant improvement compared to other classification algorithms.

Claims

1. A CNN-based emotional EEG classification method, characterized in that: The method specifically comprises the following steps: Step 1. Establish an emotional dimension model represented by Russell, and label the evaluation index labels in the data set according to the required categories; Step 2. Extract the baseline data of 3 seconds before each experiment, divide it into segments according to the selected time window, add and average it, then divide the remaining EEG data into segments according to the selected time window and subtract the processed baseline data to obtain the difference value of the EEG data, and then normalize the data; extract the three frequency bands most relevant to emotion recognition from the difference value of the EEG data, calculate the Pearson coefficient of these three frequency bands, and convert them into a 32*32*3 image format; Step 3. For each subject, take the experiment as the unit, convert the data of each subject into the number of experiments * number of segments * 32 * 32 * 3, set the remaining number after screening, and send it to the sequence backward selection to screen out the experimental combination with the highest accuracy. Since some categories of emotions will be reduced after screening, the screened experimental data will be sent to CWGAN-GP through the labeled generative adversarial network to perform data enhancement and generate data with specified labels to supplement the data and balance the emotional data of each category; Step 4. Distribute the processed data and labels according to the set ratio of training samples and test samples, and input them into the integrated convolutional neural network in the form of frames to perform classification tests on the processed samples; The data obtained in step 3 are input into the convolutional neural network in the form of frames. Specifically, for each experiment's 60 segments, every 3 segments, i.e., every 3 seconds of samples, are combined into one frame. There are no repeated samples between frames. One experiment generates 20 frames, and the total number of samples becomes: number of experiments * 20 * number of subjects. The sample format is: total number of samples * 3 * 32 * 32 * 3. All samples are input according to the set ratio of training samples and test samples. Three pictures are input to three convolutional neural networks at the same time each time. The configuration of the convolutional neural networks is the same. The average of the three output results is the final result of this input. The integrated convolutional neural network is composed of three convolutional neural networks with exactly the same structure, and the structure of each convolutional neural network is: The C1 layer is a convolutional layer with RELU, which has 32 filters, the convolution kernel is set to 3*3, and the step size is 1*1; The S2 layer is an average pooling layer with a size of 2*2 and a stride of 2*2; The B3 layer is a BN layer; The C4 layer is also a convolutional layer with RELU, with 64 filters, the convolution kernel is set to 3*3, and the step size is 1*1; The S5 layer is an average pooling layer with a size of 2*2 and a stride of 2*2; Layer B6 is a BN layer; The C7 layer is a convolutional layer with RELU, with 128 filters, the convolution kernel is set to 3*3, and the stride is 1*1; The D8 layer is a fully connected layer with an output dimension of 128; The DO9 layer is a Dropout layer, which randomly turns off some neurons with a probability of 0.5 during training; The output dimension of the final connected Softmax layer is 2.

2. The method for classifying emotional EEG based on CNN according to claim 1, characterized in that: In the step 1, the evaluation index labels in the data set are labeled according to the required categories, specifically: when the Russell emotional dimension model is used as a reference, each emotional state is placed on a two-dimensional plane, and the arousal and pleasure constitute the two dimensions of the coordinate axis. The valence represents the subjective evaluation of the subject on the emotion, which varies from negative emotion to positive emotion on the axis; the arousal represents the degree to which the subject feels the emotion, which varies from calmness to excitement on the axis; the values ​​of the two dimensions vary from 1 to 9, and the degree of emotion is represented by the size of the value; when the emotion binary classification is selected, the evaluation values ​​on the valence and arousal are divided with the median 5 as the threshold, and those greater than 5 are marked as 1, representing high valence HV or high arousal HA; those less than or equal to 5 are marked as 0, representing low valence LV or low arousal LA; when the emotion four-category classification is selected, HV, HA, LV, and LA are combined to generate four types of labels: HVHA, HVLA, LVHA, and LVLA.

Images