A method and system for EEG emotion recognition

Abstract

This invention relates to the field of data processing technology, specifically to a method and system for EEG emotion recognition. The method acquires EEG data to be recognized, preprocesses the EEG data to obtain baseline and stimulus signals, and inputs these signals into a trained emotion recognition model to obtain the EEG emotion recognition result. The emotion recognition model employs a two-stream neural network model, which is trained using the preprocessed EEG dataset. This method effectively avoids the risk of misjudgment or ignoring key emotional features, increasing the model's recognition accuracy.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and particularly relates to an electroencephalogram emotion recognition method and system. BACKGROUND

[0002] With the development of electroencephalogram technology, electroencephalogram emotion recognition has become an important field for understanding and predicting human emotional states. Electroencephalogram emotion recognition mainly refers to recognizing and classifying human emotional states through electroencephalogram signals. It uses electroencephalogram signals to reveal the electrical activity of the human brain in different emotional states. Therefore, electroencephalogram emotion recognition technology can provide support for the fields of mental health, medical diagnosis, and human-computer interaction. Electroencephalogram emotion recognition mainly involves selecting a number of samples from a set of EEG data, then comparing EEG features under different emotional states based on signal changes before and after emotional stimuli. If a certain emotional state is associated with significant features in EEG signals, it can be considered that the feature reflects a specific emotion. The key to EEG emotion recognition lies in processing and analyzing complex electroencephalogram signals. These electroencephalogram signals contain rich information, but are affected by electrical interference, muscle movement, and baseline shift, etc. Direct recognition of electroencephalogram signals will have problems such as low recognition accuracy. Therefore, data preprocessing is a basic step for emotion recognition. Data preprocessing usually involves signal slicing, baseline adjustment, and denoising, etc. to extract stable emotion-related features. There are still many key features that have not been fully revealed in the current EEG emotion recognition field, mainly caused by two reasons: first, in the construction of emotion classification models, many studies only consider EEG signal features under specific emotions, ignoring the correlation between the baseline and emotional stimulus signals. This limitation may lead to misjudgment or ignore key emotional features; second, the complexity of emotion recognition is due to the differences between different individuals and the combined effects of multiple factors. The changes in electroencephalogram signals are not only affected by individual emotions, cognition, and environment, but even measurement techniques can cause bias. This increases the difficulty of extracting and classifying emotional features.

[0003] To address the above problems, a common solution in the prior art is to use deep learning techniques such as convolutional neural networks (CNN) and recurrent neural networks (RNN) to process EEG signals. Deep learning networks can extract features from high-dimensional data and consider the temporal information of the signal. However, this method may require a large amount of training data to achieve high accuracy, and the risk of overfitting is high, especially in cases where the amount of data is small. In addition, due to the complexity of emotion recognition, there are differences in EEG signals between different individuals, which may limit the generalization ability of the model. Another solution is to use advanced signal processing techniques such as wavelet transform and Fourier transform to remove noise from EEG signals. This method can improve the quality of the signal and provide a more stable basis for emotion recognition. However, excessive signal processing may result in the loss of important features, affecting the accuracy of the model.

[0004] In summary, the problems in the prior art include: the emotion classification model ignores the correlation between the baseline and the stimulus signal, which may lead to misjudgment or omission of key features in emotion recognition. In addition, the complexity of emotion recognition is due to individual differences and the combined effects of multiple factors, increasing the difficulty of feature extraction and classification, and thus affecting the accuracy of the classification model. SUMMARY

[0005] The present application aims to solve the problem of low classification accuracy of emotion classification models by providing a method and system for EEG emotion recognition.

[0006] The purpose of the present application is achieved by the following technical solutions:

[0007] An EEG emotion recognition method, comprising the following steps:

[0008] Obtaining EEG data to be recognized, preprocessing the EEG data to obtain baseline signals and stimulus signals, inputting the baseline signals and stimulus signals into a trained emotion recognition model to obtain EEG emotion recognition results.

[0009] The emotion recognition model uses a dual-stream neural network model, and the emotion recognition model is a model trained using a preprocessed EEG data set, and the data set includes a plurality of labeled different types of EEG emotion signals.

[0010] As a further improvement of the present application, the dual-stream neural network model includes a feature extractor and a classifier, one end of the feature extractor is used to extract features from the input baseline signals and stimulus signals and calculate the difference between them, the other end is used to extract features from the input stimulus signals and fuse them with the difference to obtain fused features, and the classifier is used to classify the fused features.

[0011] As a further improvement of the present application, the feature extractor comprises a twin convolution module for extracting features of the baseline signal and the stimulus signal.

[0012] As a further improvement of the present application, the classifier comprises two connected fully connected layer modules, the first fully connected layer module is used for normalizing the input baseline signal and stimulus signal features, and the normalized features are processed by an activation function; the second fully connected layer module is used for dimension reduction of the features output by the first fully connected layer module, and the dimension-reduced features are calculated by a classification probability to output a probability distribution of classification.

[0013] As a further improvement of the present application, the preprocessing process comprises: segmenting the electroencephalogram emotional signal according to time period to obtain the baseline signal and the stimulus signal; and cutting the baseline signal and the stimulus signal with a set window, the cut stimulus signal is used for association with the emotional label, and the cut baseline signal is used for averaging processing, so that the processed signal is expanded to a dimension capable of matching the length of the stimulus signal.

[0014] As a further improvement of the present application, the emotional recognition model training process further comprises training by using a cross-entropy loss function and an optimizer, and adjusting the emotional recognition model weight based on the training result.

[0015] As a further improvement of the present application, the cross-entropy loss function is:

[0016]

[0017] In the formula, N represents the number of samples in the data set, y i,c is the true label of the tth sample in the cth class, p i,c is the classifier output probability of the tth sample in the cth class.

[0018] As a further improvement of the present application, an early stopping mechanism is added in the training process, which is used to stop the current training when the prediction result in the training process reaches a set stop threshold.

[0019] The present application also provides an electroencephalogram emotional recognition system, comprising a data preprocessing module and an emotional classification module, the data preprocessing module is used for preprocessing electroencephalogram data to obtain baseline signals and stimulus signals; the emotional classification module comprises an emotional recognition model, the baseline signals and the stimulus signals are input into the trained emotional recognition model to obtain an electroencephalogram emotional recognition result.

[0020] The emotional recognition model adopts a double-flow neural network model, the emotional recognition model is a model trained by using a preprocessed electroencephalogram data set, and the data set comprises a plurality of different types of electroencephalogram emotional signals processed by labeling.

[0021] As a further improvement of the present application, the double-flow neural network model comprises a feature extractor and a classifier, one end of the feature extractor is used for feature extraction on the input baseline signal and stimulus signal and calculation of the difference between the two, the other end is used for feature extraction on the input stimulus signal and fusion with the difference to obtain fused features, and the classifier is used for classification of the fused features.

[0022] The present application has the beneficial effects that the present application provides an electroencephalogram emotion recognition method and system, wherein the method of the present application fully considers the connection between the baseline signal and the stimulus signal, adopts a double-flow neural network model as an emotion recognition model, extracts emotion features by using the double-flow neural network model to enhance the stability of the features, effectively avoids the risk of misjudgment or neglect of key emotion features, and can also reduce the risk of overfitting, ensuring the stability of the model when dealing with complex emotion signals. BRIEF DESCRIPTION OF DRAWINGS

[0023] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiment or prior art description. Obviously, the drawings described below are some embodiments of the present application, and those skilled in the art can obtain other drawings according to these drawings without creative labor.

[0024] Figure 1 is the flow chart of the electroencephalogram emotion recognition method provided by the embodiment of the present application combined with baseline analysis and double-flow network;

[0025] Figure 2 is the implementation flow chart of the electroencephalogram emotion recognition method provided by the embodiment of the present application combined with baseline analysis and double-flow network;

[0026] Figure 3 is the double-flow network structure diagram used by the embodiment of the present application. DETAILED DESCRIPTION

[0027] In order to make the purpose and technical solutions of the present application more clear and convenient to understand. The following will further describe the present application combined with the drawings and embodiments, the specific embodiments described here are only used to explain the present application, and are not used to limit the present application.

[0028] The idea of the present application is to provide an electroencephalogram emotion recognition method and system, the method comprising: acquiring electroencephalogram data to be recognized, pre-processing the electroencephalogram data to obtain baseline signals and stimulus signals, inputting the baseline signals and stimulus signals into a trained emotion recognition model to obtain an electroencephalogram emotion recognition result;

[0029] The emotion recognition model adopts a double-flow neural network model, and the emotion recognition model is a model trained by using the preprocessed electroencephalogram data set.

[0030] The technical solutions of the present application will be described clearly and completely below in combination with the drawings and specific embodiments, wherein the described embodiments are only some of the embodiments of the present application, but not all.

[0031] The electroencephalogram emotion recognition method embodiment includes the following steps:

[0032] As shown in the electroencephalogram emotion recognition method shown in Figure 1 , Figure 2 , the electroencephalogram emotion recognition method includes the following steps:

[0033] First, the electroencephalogram data set is obtained, and the data in the electroencephalogram data set is preprocessed. The preprocessing step includes: the electroencephalogram emotion signal (i.e. electroencephalogram data) in the electroencephalogram data set is segmented according to the time period to obtain the baseline signal and the stimulus signal; the baseline signal and the stimulus signal are cut with a set window, the cut stimulus signal is used to associate with the emotion label, and the cut baseline signal is used for averaging processing, so that the processed signal is expanded to a dimension that can match the length of the stimulus signal. The electroencephalogram data set used in this embodiment is the DEAP data set.

[0034] Specifically, the electroencephalogram emotion signal in a set time period (for example, the first 3s of the electroencephalogram emotion signal) under the state of electroencephalogram activity without external stimulation is selected as the baseline signal, and the signal in a set time period (for example, 60s) under the state of emotional stimulation is selected as the stimulus signal. The two signals are cut with a set window (for example, a 1s window) to obtain smaller signal segments, and the cut electroencephalogram emotion signal of each window includes time and frequency information. The label processing includes dividing the label into 0 and 1, and one window of the stimulus signal corresponds to one label. The baseline signal is averaged. The baseline signal averaging step is: by processing each baseline signal group by loop, calculating the average value of the baseline signal in the group, and expanding it to a new dimension containing the average value to match the length of the stimulus signal.

[0035] Secondly, the emotion recognition model is constructed. The emotion recognition model in this embodiment adopts a double-flow neural network model. The double-flow neural network model includes a feature extractor and a classifier, as shown in Figure 3 The one end of the feature extractor is used for feature extraction of the input baseline signal and stimulus signal, and the difference between the two is calculated. The other end is used for feature extraction of the input stimulus signal and fusion with the difference to obtain the fusion feature.

[0036] The feature extractor includes a twin convolution module and a convolution module, and the twin convolution module is used to extract the features of the baseline signal and the stimulation signal. For example, in the embodiment, a one-dimensional convolution twin module with a kernel size of 9 is used in one stream, and a one-dimensional convolution block with a kernel size of 9 is used in the other stream.

[0037] The classifier is used to classify the fused features. The classifier includes two connected fully connected layer modules, i.e., a first connected layer module and a second connected layer module. The first connected layer module and the second connected layer module have different sizes. The first fully connected layer module is used to normalize the input baseline signal and stimulation signal features, and the normalized features are subjected to an activation function processing. The second fully connected layer module is used to reduce the dimension of the features output by the first fully connected layer module, and the reduced features are subjected to a classification probability calculation, and the output is a probability distribution of classification. For example, in the embodiment, the first fully connected layer module is a 128-dimensional fully connected layer, and the second fully connected layer module is a 2-dimensional fully connected layer. The function used for the separation probability calculation is a LogSoftmax function.

[0038] The above EEG dataset is used to train the constructed emotion recognition model. In the embodiment, a cross-entropy loss function and an optimizer are used for training, and the emotion recognition model weight is adjusted based on the training result. The cross-entropy loss function is as follows:

[0039]

[0040] In the formula, N represents the number of samples in the dataset, y i,c is the true label of the i-th sample in the c-th class, p i,c is the classifier output probability of the i-th sample in the c-th class.

[0041] In order to avoid overfitting, an early stopping mechanism is added in the training process. The early stopping mechanism is used to stop the current training when the prediction result in the training process reaches the set stopping threshold. That is, if the output result of the emotion recognition model does not change and exceeds the set threshold, the early stopping is triggered. For example, in the embodiment, the stopping threshold can be set to 5.

[0042] The embodiment also includes a step of testing using the EEG dataset. The ten-fold cross-validation method is mainly used for verification. That is, the data is divided into ten parts using the ten-fold cross-validation method. In the training and testing of the model, a cyclic method is used for verification. In each cycle, nine parts are used to train the model, and the remaining one part is used to test the model. This process is repeated ten times, and each time a different part is selected as the test set and the remaining part is selected as the training set. In the cycle, the threshold of the early stopping mechanism is set to 5. If the prediction result of the model after the forward propagation does not change for more than 5 times, the next cycle is entered.

[0043] Finally, the electroencephalogram data to be recognized is obtained, the electroencephalogram data to be recognized is preprocessed to obtain baseline signals and stimulation signals, and the baseline signals and the stimulation signals are input into the trained emotion recognition model to obtain an electroencephalogram emotion recognition result.

[0044] In order to better illustrate the performance of the recognition method mentioned in the embodiment, the embodiment is verified through a specific example, and the example is as follows:

[0045] The real electroencephalogram data selects DEAP data set, the DEAP data set provides 32 electroencephalogram channels, the original record frequency is 512Hz, but then it is down-sampled to 128Hz. After the data is down-sampled, it is processed by a 4.0-45.0Hz band-pass filter. In the experiment, each participant watched 40 pieces of emotional music videos, each video was 1 minute long. After watching, the participants scored the Valence label, Arousal label, Dominance label and Liking label according to the emotional experience in the range from 1 to 10. The specific implementation steps on the DEAP data set are as follows:

[0046] Pretreatment of initial data: the data dimension of each electroencephalogram emotion in the DEAP data set will be 40*63*32*128, and the dimension of each electroencephalogram emotion video corresponding to each subject is 63*32*128. For each 63s video, take the first 3s as the baseline signal and the last 60s as the stimulation signal. The two of the four labels in the DEAP data set, namely Arousal and Valence, are selected, and labels greater than 5 are regarded as 1, and labels less than or equal to 5 are regarded as 0. At this time, the baseline signal dimension is 3*32*128, the stimulation signal dimension is 60*32*128, and the label dimension is 60*2. Then the baseline signal is averaged, the baseline mean is repeated 60 times, and the length of the stimulation signal is consistent. Finally, the averaged baseline signal and the segmented stimulation signal will be input into the next step of the emotion recognition model.

[0047] The averaged baseline signal and the segmented stimulus signal are input into the emotion recognition model. The feature extractor in the emotion recognition model has one stream of input, which is the baseline and stimulus signal. The structure of the feature extractor is a pair of identical twin convolution modules. The twin convolution modules use a one-dimensional convolution with a kernel size of 9, padding to ensure the output size is unchanged, batch normalization for data standardization in each batch, and ReLU activation function and Dropout to prevent overfitting. The processed baseline and segmented stimulus signal are input into the pair of identical twin convolution modules, and the extracted features are subtracted. The other stream of input is the segmented stimulus signal. The convolution module structure of this part is the same as the twin convolution module. After extraction, the difference is added. The dimension of the result is batch*32*128. The added result is flattened, and the dimension is batch*4096.

[0048] The result of the feature extractor is input into the classifier. The structure of the classifier first uses a fully connected layer to reduce the input feature from 4096 dimensions to 128 dimensions, then uses batch normalization and ReLU activation function, followed by a second fully connected layer to further convert the feature from 128 dimensions to 2 dimensions. Finally, the classification probability is calculated by the LogSoftmax function, and the output is the probability distribution of classification.

[0049] During the training process, the batch parameter is set to 1024, the number of rounds is set to 200, the learning rate is 0.001, the optimizer uses Adam, and the data is divided into ten parts using ten-fold cross-validation. During the training and testing of the emotion recognition model, a cyclic method is used for verification. In each cycle, nine parts are used to train the model, and the remaining one part is used to test the model. This process is repeated ten times, each time selecting a different part as the test set and the remaining part as the training set. In the cycle, the early stopping mechanism threshold is set to 5. If the prediction result of the emotion recognition model after forward propagation does not change for more than 5 times, it enters the next cycle.

[0050] After the above training and testing, the training and testing accuracy of the Arousal label obtained by ten-fold cross-validation is shown in Table 1, and the training and testing accuracy of the Valence label obtained by ten-fold cross-validation is shown in Table 2.

[0051] Table 1

[0052]

[0053]

[0054] Table 2

[0055] Fold Train_acc Test_acc 1 99.68％ 97.93％ 2 99.68％ 97.51％ 3 99.59％ 97.70％ 4 99.71％ 98.13％ 5 99.77％ 98.13％ 6 99.78％ 98.09％ 7 99.82％ 97.95％ 8 99.79％ 98.37％ 9 99.70％ 97.89％ 10 99.80％ 98.21％

[0056] In Table 1 and Table 2, Train_acc is the accuracy obtained during the training process, and Test_acc is the accuracy obtained during the testing process. The results in Table 1 and Table 2 show that the model can learn the features of Arousal labels and Valence labels with high accuracy and maintain stable performance in the testing phase. This demonstrates the reliability of the method in processing electroencephalogram data.

[0057] The average accuracy and variance of the accuracy obtained from the test accuracy in Table 1 and Table 2 are shown in Table 3:

[0058] Table 3

[0059]

[0060]

[0061] In Table 3, Test_mean is the average value of the test accuracy, and Test_variance is the variance of the test accuracy. The results in Table 3 show that the average accuracy of 98.28% and 97.99% indicates a high recognition ability in emotion recognition; at the same time, the variance values of 0.13 and 0.23 indicate that the test results have low volatility, further demonstrating the stability of the method.

[0062] In summary, the method of the embodiment extracts emotional features by comparing the differences between baseline signals and stimulation signals, and uses a dual-flow neural network model to enhance the stability of the features. This processing method can effectively reduce the risk of overfitting, ensure that the model remains stable when dealing with complex emotional signals, and reduce the impact of noise on emotional features by adjusting the signal processing process. The use of a dual-flow neural network model to extract emotional features enhances the stability of the features, effectively avoids the risk of misjudgment or ignoring key emotional features, and increases the recognition accuracy of the model.

[0063] Electroencephalogram emotion recognition system embodiment:

[0064] The electroencephalogram emotion recognition system in the embodiment comprises a data preprocessing module and an emotion classification module. The data preprocessing module is used for preprocessing electroencephalogram data to obtain baseline signals and stimulation signals. The emotion classification module comprises an emotion recognition model. After the baseline signals and the stimulation signals are input into the trained emotion recognition model, electroencephalogram emotion recognition results are obtained. The data preprocessing module is responsible for segmentation processing and baseline adjustment of electroencephalogram data. The electroencephalogram data is processed through signal slicing and baseline signal adjustment. Signal characteristics are extracted through differential processing of baseline signals and stimulation signals. The emotion classification module comprises a double-input structure. Two kinds of inputs are processed in parallel. The training module is used for training the module. The early stopping mechanism is combined to prevent overfitting. The emotion classification module is trained using a loss function and an optimizer during the training process. Network weights are adjusted based on the training to realize emotion recognition classification.

[0065] The emotion recognition model adopts a double-flow neural network model. The emotion recognition model is a model trained using a preprocessed electroencephalogram data set. The data set comprises a plurality of different types of electroencephalogram emotion signals processed by labeling. The double-flow neural network model comprises a feature extractor and a classifier. One end of the feature extractor is used for feature extraction of input baseline signals and stimulation signals and calculation of the difference between the two. The other end is used for feature extraction of input stimulation signals and fusion with the difference to obtain fusion features. The classifier is used for classification of the fusion features. The specific implementation steps and principles of each module are described in detail in the electroencephalogram emotion recognition method embodiment, which will not be described here.

Claims

1. An electroencephalogram emotion recognition method, characterized in that, The method comprises the following steps: The method comprises the following steps: The emotion recognition model is a model trained by using a preprocessed electroencephalogram data set, and the data set comprises a plurality of different types of electroencephalogram emotional signals processed by labeling. The double-flow neural network model comprises a feature extractor and a classifier. 2.The electroencephalogram emotion recognition method of claim 1, wherein, The feature extractor comprises a twin convolution module, which is used to extract features of the baseline signal and the stimulus signal. 3.The electroencephalogram emotion recognition method of claim 1, wherein, The classifier comprises two connected fully connected layer modules. 4.The electroencephalogram emotion recognition method of claim 1, wherein, The preprocessing process comprises the following steps: the electroencephalogram emotional signal is segmented according to a time period to obtain the baseline signal and the stimulus signal; the baseline signal and the stimulus signal are cut with a set window, the cut stimulus signal is used to associate with an emotional label, and the cut baseline signal is used for averaging processing, so that the processed signal is expanded to a dimension that can match the length of the stimulus signal.

5. The electroencephalogram emotion recognition method of claim 1, wherein, The emotion recognition model training process further comprises training by using a cross-entropy loss function and an optimizer, and adjusting the weight of the emotion recognition model based on the training result.

6. The electroencephalogram emotion recognition method of claim 5, wherein, The cross-entropy loss function is: wherein, denotes the number of samples in the dataset, is the true label of the th sample in the th class, is the classifier output probability of the th sample in the th class.

7. The electroencephalogram emotion recognition method of claim 5, wherein, An early stopping mechanism is added in the training process.

8. An electroencephalogram emotion recognition system, characterized by, The method comprises a data preprocessing module and an emotion classification module. The emotion recognition model is a model trained by using a preprocessed electroencephalogram data set, and the data set comprises a plurality of different types of electroencephalogram emotional signals processed by labeling. The double-flow neural network model comprises a feature extractor and a classifier. The feature extractor comprises a twin convolution module, which is used to extract features of the baseline signal and the stimulus signal. The classifier comprises two connected fully connected layer modules. The preprocessing process comprises the following steps: the electroencephalogram emotional signal is segmented according to a time period to obtain the baseline signal and the stimulus signal; the baseline signal and the stimulus signal are cut with a set window, the cut stimulus signal is used to associate with an emotional label, and the cut baseline signal is used for averaging processing, so that the processed signal is expanded to a dimension that can match the length of the stimulus signal.

Images