An electroencephalogram emotion recognition method based on data uncertainty

Abstract

The application discloses an electroencephalogram emotion recognition method based on data uncertainty, and steps of the method comprise the following steps: 1, establishing an additional branch for a network model to calculate the uncertainty value of an electroencephalogram sample; 2, mixing different electroencephalogram features to construct a data uncertainty loss function; 3, using a ten-fold cross method to train a convolutional neural network model of the mixed loss on a public electroencephalogram emotion dataset; and 4, using the established network model to realize an electroencephalogram emotion classification task. The application can realize high-precision emotion recognition, thereby improving the recognition rate.

Description

Technical Field

[0001] This invention relates to the fields of emotion computing and data uncertainty, specifically to a brainwave emotion recognition method based on data uncertainty. Background Technology

[0002] Emotion is a complex psychological and physiological state that can influence a person's cognition, behavior, and interpersonal interactions in daily life. In recent years, with the rise of human-computer interaction, accurately detecting users' emotional states and simulating emotional interactions between humans and machines has become particularly important for intelligent machines. Typically, intelligent robots identify human emotional states through both physiological and non-physiological signals. Compared to physiological signals such as facial expressions, behavior, and language, electroencephalograms (EEGs) can more realistically reflect a person's emotional state, and these signals are not influenced by the user's subjective will. Furthermore, research shows that EEGs have better temporal resolution in reflecting a person's emotional state compared to other non-physiological signals, providing more meaningful research for developing emotion recognition in human-computer interaction. EEG emotion recognition algorithms are generally divided into two categories: algorithms based on traditional methods and algorithms based on deep learning.

[0003] In traditional EEG signal-based emotion recognition algorithms, features for different emotions are typically extracted manually from the EEG signals first, and then these features are used in a classifier for emotion tasks. Currently, commonly used EEG emotion recognition features mainly include time-domain features, frequency-domain features, time-frequency-domain features, and spatial features. Time-domain features reflect information in the time domain of the EEG signal; frequency-domain features reflect frequency-domain information in different frequency bands of the EEG signal; time-frequency-domain features consider more dynamic EEG information in both the frequency and time domains; and spatial features use the spatial location information of different channels during EEG acquisition as features for emotion recognition. Although traditional machine learning algorithms have achieved satisfactory results, the limited generalization ability of traditional classifiers makes it difficult to meet the needs of complex EEG emotion recognition.

[0004] In recent years, the powerful feature extraction capabilities of deep learning networks, which can automatically learn robust and abstract feature representations, have led to superior performance in deep learning-based EEG emotion recognition methods. Various deep learning networks are used as classifiers to extract features from EEG signals. In addition, some researchers directly apply deep learning networks to raw EEG signals to achieve automatic feature extraction and classification, enabling end-to-end emotion recognition research. Therefore, deep learning networks are increasingly being used in emotion recognition tasks. However, deep learning models are susceptible to inference errors and inherent noise in EEG signals, meaning even well-trained models can lead to uncertainties in recognition results, further negatively impacting the accuracy of EEG emotion recognition. Summary of the Invention

[0005] To overcome the shortcomings of existing technologies, this invention proposes an EEG emotion recognition method based on data uncertainty, aiming to reduce the influence of "noise" in EEG samples, achieve high-precision emotion recognition, and thus improve the accuracy of emotion recognition.

[0006] To achieve the above-mentioned objectives, the present invention adopts the following technical solution:

[0007] The present invention provides a method for EEG emotion recognition based on data uncertainty, characterized by the following steps:

[0008] Step 1: Obtain EEG signal data of any subject A with H emotion labels, and perform baseline removal and sample segmentation processing to obtain N EEG signal samples of subject A, denoted as X = {x1, x2, ..., x...} i ,…,x N The emotion label is denoted as L={l1,l2,…,l i ,…,l N}, where x i ∈R m×n Let l represent the i-th EEG signal sample. i ∈R mxn Represents the i-th EEG signal sample x i The emotion label is defined as follows: m represents the number of channels of the EEG signal, n represents the number of sampling points, i = 1, 2, ..., N; N represents the number of EEG signal samples.

[0009] Step 2: Calculate the uncertainty value for each EEG signal sample:

[0010] Step 2.1: Construct a neural network based on EEG emotion recognition, including: a feature extraction model f θ (·), classifier model and uncertainty branch running parallel to the classifier model;

[0011] The EEG signal sample X is input into the feature extraction model f. θ The EEG signal feature representation f = f is obtained by processing (·) in the formula. θ (X);

[0012] The classifier model consists of a linear fully connected layer;

[0013] The EEG signal feature representation f is input into the classifier model for processing to generate a two-dimensional feature vector with a number of nodes;

[0014] The uncertainty branch consists of two smaller branches with the same structure. Each smaller branch includes: a first normalization layer, a Dropout layer, a Flatten layer, a fully connected layer, and a second normalization layer. The first normalization layer uses a normalization function d1 and has p1 channels. The fully connected layer has m input channels and n output channels. The second normalization layer uses a normalization function d2 and has p2 channels.

[0015] The EEG signal feature representation f is input into the two small branches with the same structure, and the depth feature vector of the EEG signal sample X is obtained accordingly. and uncertain vectors Where, μ i Represents the i-th EEG signal sample x i depth features, σ i Represents the i-th EEG signal sample x i The uncertainty characteristics, where D represents the dimension of the EEG signal characteristics;

[0016] Step 2.2: Calculate the uncertainty value of the EEG signal sample X:

[0017] The uncertainty value of all EEG signal samples is obtained by averaging the uncertainty vector Q of the EEG signal sample X. in, For the i-th EEG signal sample x i Uncertain values;

[0018] Step 3: Construct the uncertainty loss function:

[0019] Step 3.1: Use equation (1) to obtain the normalized uncertainty value of the i-th EEG sample.

[0020]

[0021] In equation (1), x represents the j-th EEG signal sample j Uncertain values; j = 1, 2, ..., N;

[0022] Step 3.2: Use equation (2) to obtain the i-th EEG signal sample x. i and the j-th EEG signal sample x j Mixed characteristics

[0023]

[0024] In equation (2), x represents the j-th EEG signal sample j The normalized uncertainty; μ j x represents the j-th EEG signal sample j Depth features;

[0025] Step 3.3: Construct the contrastive loss function L using equation (3). data :

[0026]

[0027] In equation (3), l j x represents the j-th EEG signal sample j Tags; x represents the j-th EEG signal sample j The tag l j Weights in all EEG sample labels; Represents the i-th EEG signal sample x i The tag l i Weights in all EEG sample labels; C h The weights represent the weights obtained by the classifier model after predicting the h-th emotion label.

[0028] Step 4: Training the joint optimization framework:

[0029] Step 4.1: Construct the cross-entropy loss function L using equation (4). cross :

[0030]

[0031] In equation (4), It is the i-th EEG signal sample x i The corresponding label is obtained by a 1-hot encoder as a binary value. It is the i-th EEG signal sample x i The two-dimensional feature vector output after passing through the classifier model;

[0032] Step 3.2: Construct the hybrid loss function L using equation (5). total :

[0033] L total=L cross +λL data (5)

[0034] In equation (5), λ is the regularization parameter;

[0035] Step 3.3: Train the neural network based on EEG emotion recognition using EEG signal samples X, and use the Adam optimizer to optimize the hybrid loss function L. tatal The optimal network model is obtained by minimizing the solution to optimize all parameters in the network and then using it to identify EEG emotions.

[0036] Compared with existing technologies, the beneficial effects of this invention are reflected in:

[0037] 1. This invention constructs a new branch in an end-to-end emotion recognition neural network to quantify the data uncertainty of each EEG sample. Then, by mixing the correlations in different EEG samples, it achieves simultaneous learning of the features and uncertainties of EEG samples, thereby realizing accurate quantification of the uncertainty of EEG samples in the field of EEG emotion recognition and improving the credibility of the neural network model's emotion recognition results.

[0038] 2. This invention trains an emotion recognition neural network model using an additive loss consisting of a data uncertainty loss function and a cross-entropy loss function. After training, the neural network model can accurately identify different expressions of emotional states from mixed EEG features. By optimizing the additive loss to the maximum extent, the network model can learn the uncertainty of each EEG sample, thereby improving the generalization ability and recognition accuracy of the network model. Attached Figure Description

[0039] Figure 1 This is a schematic diagram of the method flow of the present invention;

[0040] Figure 2 This is an indeterminate branch structure diagram of the present invention;

[0041] Figure 3 A graph showing the average recognition accuracy of 32 subjects in the valence dimension of the DEAP database;

[0042] Figure 4 A graph showing the average recognition accuracy of 32 subjects in the arousal dimension of the DEAP database;

[0043] Figure 5 This is a graph showing the average recognition accuracy of 32 subjects on the dominance dimension in the DEAP database. Detailed Implementation

[0044] In this embodiment, an EEG emotion recognition method based on data uncertainty first performs baseline removal and sample segmentation preprocessing on the EEG signal samples. Then, an uncertainty branch is constructed from an end-to-end efficient convolutional neural network. The uncertainty branch is used to calculate the uncertainty value of each EEG signal sample. The uncertainty values ​​of EEG signal samples in different emotional states are used as weights to mix the EEG signal features. The mixed EEG signal features and corresponding sample signal labels are used to calculate the data uncertainty loss. The network model is trained by combining the data uncertainty loss and the cross-entropy loss to achieve uncertainty learning of the EEG samples. Figure 1 As shown, specifically, the method is carried out in the following steps:

[0045] Step 1: Obtain EEG signal data of any subject A with H emotion labels, and perform baseline removal and sample segmentation processing to obtain N EEG signal samples of subject A, denoted as X = {x1, x2, ..., x...} i ,…,x N The emotion label is denoted as L={l1,l2,…,l i ,…,l N}, where x i ∈R m×n Let l represent the i-th EEG signal sample. i ∈R mxn Represents the i-th EEG signal sample x i The emotion label is denoted by m, where m represents the number of EEG signal channels, n represents the number of sampling points, and i = 1, 2, ..., N. This example uses EEG data from the publicly available DEAP database for training and testing. The dataset contains EEG signals collected from 32 subjects after they watched 40 one-minute audio / video clips. The EEG signals collected from each subject in one experiment... The data in the first 3 seconds is the baseline data. The labels are the corresponding ratings given by the subjects after each experiment, which include three emotional dimensions: Arousal, Valence and Dominance. Each emotion is divided into two categories: High and Low.

[0046] Forty segments of EEG signals from each subject were preprocessed, including:

[0047] Assume the original EEG signal sample of an experiment is X∈R m×n m is the number of channels in the EEG signal, n is the number of sampling points, and the baseline signal is x. i ∈R m×L ,i=1,2,...,s. In the DEAP dataset, the baseline duration for each subject is s=3.

[0048] Baseline removal: The average baseline data for 1 second is calculated as shown in equation (1):

[0049]

[0050] In equation (1), Let x represent the 1-second average baseline data corresponding to the i-th EEG signal sample. i This represents the i-th EEG signal sample.

[0051] To reduce interference from resting EEG, baseline-removing preprocessing of the EEG signal was performed using average baseline data, as shown in equation (2):

[0052]

[0053] In equation (2), x′ j x represents the data of the j-th EEG signal sample after removing the baseline signal. j This represents the j-th EEG signal sample.

[0054] Sample Segmentation: To increase the sample size, this invention segments the EEG signal of each subject for 60 seconds using a 1-second sliding window, and the corresponding sample labels are also copied, ultimately obtaining the EEG sample and corresponding label for each subject. The experiment uses a 10-fold cross-validation method, randomly selecting 90% of the EEG samples as the training set and the remaining 10% as the test set. The average recognition rate after ten trials on the test set is used as the final recognition accuracy.

[0055] Step 2: EEG samples contain inherent "noise" such as electromyography (EMG) signals, electrooculography (EOG) signals, and heart rate signals, as well as inherent uncertainties arising from the complexity of EEG itself. To better quantify these uncertainties, this invention proposes a novel method to calculate the uncertainty value for each EEG sample:

[0056] Step 2.1, as follows Figure 1 As shown, a neural network based on EEG emotion recognition is constructed, including: a feature extraction model, a classifier model, and an uncertainty branch constructed in parallel with the classifier model to quantify the uncertainty of EEG signal samples;

[0057] The feature extraction model consists of a standard convolutional module, a block module, a pooling layer, and a dimensionality reduction layer; the EEG signal sample X is input to the feature extraction model f. θ The EEG signal feature representation f = f is obtained by processing (·) in the formula. θ (X);

[0058] The classifier model consists of a single fully connected linear layer;

[0059] The EEG feature representation f is input into the classifier model for processing and generates a two-dimensional feature vector with 2 nodes. This feature vector is regarded as the probability of belonging to each EEG emotion classification result.

[0060] The uncertainty branch consists of two smaller branches with the same structure. Each smaller branch comprises: a first normalization layer, a dropout layer, a flatten layer, a fully connected layer, and a second normalization layer. The first normalization layer uses the normalization function BatchNorm to prevent gradient explosion and vanishing, and has 512 channels. The dropout layer randomly discards 40% of neurons to prevent overfitting. The flatten layer flattens the 3D EEG sample feature map of shape (512, 16, 4) into a 1D EEG sample feature map of shape (512x16x4) to facilitate computation of deep EEG features. The fully connected layer has 32768 input channels and 64 output channels. The second normalization layer uses the normalization function BatchNorm and has 64 channels to make the obtained data more stable.

[0061] The EEG signal feature representation f is input into two small branches with the same structure, and the depth feature vector of the EEG signal sample is obtained accordingly. and uncertain vectors Where, μ i Represents the i-th EEG signal sample x i depth features, σ i Represents the i-th EEG signal sample x i The uncertainty feature is represented by 3, which represents the dimension of the EEG feature, and 100 represents the batch size of the EEG. The deep feature vector is used as the parameter to predict the EEG label, and the uncertainty vector is used as the parameter to quantify the uncertainty in the EEG sample. This makes the vector obtained from the uncertainty branch random and uncertain, thereby increasing the stability of the network in classifying EEG emotion signals and improving the classification accuracy.

[0062] Step 2.2, as follows Figure 2 As shown, the uncertainty value of the EEG signal sample is calculated:

[0063] The uncertainty value of all EEG signal samples is obtained by averaging the uncertainty vector Q of the EEG signal samples. in, For the i-th EEG signal sample x i Uncertain values;

[0064] Step 3, as follows Figure 2 As shown, construct the uncertainty loss function:

[0065] Step 3.1: Since the randomness of the batch input EEG signal samples will be disrupted during the training process, in order to better compare the EEG signal samples in the entire dataset, it is necessary to element-normalize the uncertainty vector of the EEG signal samples before mixing EEG signal samples from different emotional states. The normalized uncertainty value of the i-th EEG signal sample is obtained using equation (3).

[0066]

[0067] Step 3.2: Use equation (4) to obtain the i-th EEG signal sample x with different emotional states. i and the j-th EEG signal sample x j Mixed characteristics

[0068]

[0069] In equation (4), μ represents the normalized uncertainty value of the j-th EEG signal sample; j x represents the j-th EEG signal sample j The depth features; the i-th EEG signal sample and the j-th EEG signal sample have different emotional states;

[0070] Step 3.3: To force the network model to simultaneously identify two emotions from the mixed EEG features of the input sample, this invention designs a data uncertainty loss to encourage the uncertain branch to learn their uncertainty values ​​for different EEG signal samples; the contrastive loss function L is constructed using equation (5). data :

[0071]

[0072] In equation (5), l j x represents the j-th EEG signal sample j Tags; x represents the j-th EEG signal sample j The tag l j Weights in all EEG sample labels; Represents the i-th EEG signal sample x i The tag l i Weights in all EEG sample labels; C hThe weights represent the weights obtained by the classifier model after predicting the h-th emotion label; the data uncertainty loss forces the model to simultaneously identify two different emotion expressions from mixed EEG features. In mixed EEG signal features, one EEG emotion will always be easier for the model to identify than the other. When the model can accurately identify one of the emotion states in the mixed EEG signal features, the other emotion state can also be accurately identified with a small amount of training.

[0073] Step 4: Training the joint optimization framework:

[0074] Step 4.1: The proposed joint optimization strategy was used to solve the emotion classification problem under uncertain EEG data conditions, such as... Figure 1 As shown, the classifier model uses a feature representation vector with b = 64 nodes as input and outputs the probability of belonging to each sample. A hot encoder is used to binarize the sample labels. The cross-entropy loss is calculated using the sample probabilities output by the classifier model and the true sample labels. The cross-entropy loss function L is constructed using equation (6). cr OSS:

[0075]

[0076] In equation (6), It is the i-th EEG signal sample x i The corresponding label is obtained by a 1-hot encoder as a binary value. It is the i-th EEG signal sample x i The two-dimensional feature vector with two nodes output by the classifier model represents the probability of two emotion classifications; the cross-entropy loss function measures the entropy between the actual probability distribution of the sample and the output probability distribution of the model. This invention uses cross-entropy loss to identify the similarity between EEG signal samples.

[0077] Step 4.2, as follows Figure 1 As shown, cross-entropy loss is used to identify the similarity between EEG signal samples, and data uncertainty loss is used to measure the uncertainty of EEG signal samples. By combining these two loss functions, the feature extraction model can obtain more accurate EEG signal samples, thereby achieving a higher recognition accuracy. The hybrid loss function L is constructed using equation (7):

[0078] L total =L cross +λL data (7)

[0079] In Equation (7), λ is the regularization parameter, which refers to the regularization weight of the two loss functions. By changing the value of λ, the actual contribution of data uncertainty loss to the entire network during model training can be adjusted. Since the data uncertainty branch only learns the uncertainty value of EEG during the model training stage to mix different EEG signal features, and is deleted during the testing stage, the EEG emotion recognition model is still considered to be an end-to-end framework and does not increase the additional cost due to the introduction of data uncertainty.

[0080] Step 4.3: Train the neural network based on EEG emotion recognition using EEG signal samples X. This invention employs the Adam optimizer, which effectively addresses issues such as network convergence speed and parameter overfitting through weight decay and L2 regularization. All parameters involved in the feature extraction model, classifier model, and projection model are optimized using the Adam optimizer, and the mixed loss function L... tatal Minimization was performed to optimize all parameters in the network structure, resulting in the optimal recognition network model for identifying EEG emotions. All experiments employed 10-fold cross-validation to obtain the final recognition accuracy, ensuring fairness between different comparative experiments.

[0081] Step 4.5: Use the proposed joint optimization strategy to solve the EEG emotion classification problem, such as... Figure 1 As shown, a hybrid loss function L is established. total It has been proven feasible to achieve high classification accuracy by updating network parameters in the direction of loss reduction.

[0082] In practice, the network model under the joint optimization framework is compared with different regularization weights for cross-entropy loss and data uncertainty loss. The average recognition accuracy of the test set is used as the evaluation index. The higher the recognition accuracy, the better the recognition ability of the model. Experiments show that the best effect can be achieved when the regularization parameter λ=3 of the hybrid loss. This shows that higher accuracy can be obtained by using the hybrid loss composed of cross-entropy loss and data uncertainty loss.

[0083] Table 1. Average recognition results of different models in three dimensions on the DEAP database.

[0084]

[0085]

[0086] As shown in Table 1, this invention demonstrates that applying this combined loss to the network model can improve the model's generalization ability and achieve optimal recognition accuracy.

[0087] The results of the 10-fold cross-validation of 32 subjects are as follows: Figure 3 , Figure 4 and Figure 5 As shown. Result analysis:

[0088] Experimental results show that, compared with traditional methods such as DT, SVM, MLP, and deep learning models such as Cont-CNN, DGCNN, and CRAM, the proposed EEG emotion recognition method based on data uncertainty learning significantly improves the recognition accuracy in three emotion dimensions (valence, arousal, and dominance). This verifies that the data uncertainty branch can effectively learn the uncertainty values ​​between different EEG signal samples. Furthermore, a reasonable mixture of cross-entropy loss and data uncertainty loss enhances the model's representation of EEG data, resulting in higher recognition accuracy. In addition, from... Figure 3 , Figure 4 and Figure 5 It can be found that the EEG emotion recognition method proposed in this invention has the best recognition effect on 32 subjects. The smaller standard deviation indicates that the model has better stability and better generalization ability for different subjects.

[0089] In summary, this invention fully utilizes the relevant information contained in the raw EEG signals. By using the combined network model and joint optimization framework proposed in this invention, the network model can achieve a top-1 recognition accuracy in the public dataset DEAP, and the recognition performance is more stable across all subjects.

Claims

1. A brain electrical emotion recognition method based on data uncertainty, characterized in that, is performed as follows: Step 1: Obtain EEG signal data of any subject A with H emotion labels, and perform baseline removal and sample segmentation processing to obtain N EEG signal samples of subject A, denoted as X = {x1, x2, ..., x...} i ,…,x N The emotion label is denoted as L={l1,l2,…,l i ,…,l N }, where x i ∈R m×n Let l represent the i-th EEG signal sample. i ∈R mxn Represents the i-th EEG signal sample x i The emotion label is defined as follows: m represents the number of channels of the EEG signal, n represents the number of sampling points, i = 1, 2, ..., N; N represents the number of EEG signal samples. Step 2, calculate the uncertainty value of each electroencephalogram signal sample: Step 2.1, constructing a neural network based on electroencephalogram emotion recognition, comprising: a feature extraction model f θ (·) a classifier model and an uncertainty branch in parallel with the classifier model; The electroencephalogram signal sample X is input into the feature extraction model f θ (·) for processing to obtain an electroencephalogram signal feature representation f = f θ (X) The classifier model is composed of a linear fully connected layer; The electroencephalogram signal feature representation f is input into the classifier model for processing to generate a two-dimensional feature vector with a node number of a; The uncertainty branch is composed of two small branches with the same structure, each small branch including: a first normalization layer, a Dropout layer, a Flatten layer, a fully connected layer and a second normalization layer; wherein the first normalization layer uses a normalization function d1, and the channel number is p1; the input channel of the fully connected layer is m, and the output channel is n; the second normalization layer uses a normalization function d2, and the channel number is p2; The brain electrical signal feature f is input into the two small branches with the same structure respectively, and the deep feature vector of the brain electrical signal sample X is obtained accordingly and an uncertainty vector Wherein, μ i represents the deep feature of the i-th brain electrical signal sample x i , σ i represents the uncertainty feature of the i-th brain electrical signal sample x i , and D represents the dimension of the brain electrical signal feature; Step 2.2, calculate the uncertainty value of the electroencephalogram signal sample X: The uncertainty vector Q of the electroencephalogram signal sample X is averaged to obtain the uncertainty value of all electroencephalogram signal samples wherein, is the uncertainty value of the i-th electroencephalogram signal sample x i . Step 3, construct an uncertainty loss function: Step 3.1, obtaining the normalized uncertainty of the i-th electroencephalogram sample with formula (1) In formula (1), represents the jth brain electrical signal sample x j of the uncertainty value; j = 1, 2, …, N; Step 3.2, obtain the ith electroencephalogram signal sample x using formula (2) i and the mixing feature of the jth electroencephalogram signal sample x j ​ In formula (2), represents the jth brain electrical signal sample x j normalized uncertainty value; μ j represents the jth brain electrical signal sample x j depth feature of the jth brain electrical signal sample Step 3.3, constructing the contrastive loss function L with formula (3) data : In formula (3), l j represents the label of the jth electroencephalogram signal sample x j ; represents the label of the jth electroencephalogram signal sample x j ; j represents the weight in all electroencephalogram sample labels; represents the label of the ith electroencephalogram signal sample x i ; i represents the weight in all electroencephalogram sample labels; C h represents the weight obtained after the classifier model predicts the hth emotional label; Step 4, train the joint optimization framework: Step 4.1, constructing a cross-entropy loss function L with formula (4) cross : In formula (4), is the i-th electroencephalogram signal sample x i corresponding to the label after 1-hot encoder to get the binary value, is the i-th electroencephalogram signal sample x i two-dimensional feature vector output after the classifier model Step 3.2, construct the hybrid loss function L with formula (5) total : L total = L cross + λL data (5) In formula (5), λ is a regularization parameter; Step 3.3, training the neural network based on electroencephalogram emotion recognition based on the electroencephalogram signal sample X, and using the Adam optimizer to optimize the mixed loss function L total The minimization solution is solved, so that all parameters in the network are optimized, and the optimal recognition network model is obtained, which is used to realize the recognition of electroencephalogram emotion.

Images