EEG emotion analysis method based on deep neural network

Abstract

The application discloses an EEG emotion analysis method based on a deep neural network, and comprises the following steps: collecting multi-channel data, performing multi-dimensional feature extraction on the data, and performing smoothing and standardization processing on the features; inputting the processed features into a CNN-LSTM model, analyzing the features, and obtaining emotion features; further analyzing the emotion features and signal feature dimensions, and outputting accurate emotion scores; the application realizes multi-dimensional feature extraction and emotion quantization discrimination on full-channel electroencephalogram data, replaces traditional category discrimination, and achieves high efficiency, high precision and meticulous effect.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of physiological digital information processing, and particularly relates to an EEG emotion analysis method based on a deep neural network. BACKGROUND

[0002] Emotion research is an interdisciplinary field that brings together research from computer science, psychology, cognitive science, and neuroscience to explore the complexity of human emotions. In this field, traditional methods such as facial recognition and motion analysis are often subject to subjective control and deception, limiting their accuracy. Electroencephalogram (EEG) analysis, as an emerging technology, is considered an effective way to improve emotion recognition accuracy because it is difficult to fake and manipulate. With the advancement of wearable technology and dry electrode technology, EEG data acquisition is no longer limited to laboratory environments, greatly facilitating practical applications.

[0003] Despite the significant progress in obtaining EEG data, the asymmetry, instability, and low signal-to-noise ratio of EEG signals remain major challenges in data processing. Currently, most research still focuses on classifying EEG emotional signals into positive, neutral, and negative categories through classification tasks; while this approach can reflect human emotional states to some extent, it ignores the continuity and complexity of emotions, resulting in insufficient accuracy and detail in emotion recognition. In addition, some research attempts to quantify emotions from three dimensions: valence, arousal, and dominance. However, when processing EEG data, the instability of the signal and the low signal-to-noise ratio often pose technical challenges, limiting the accuracy and practicality of emotion recognition. SUMMARY

[0004] To address the problems in the prior art, the present application aims to provide an EEG emotion analysis method based on a deep neural network, which realizes multi-dimensional feature extraction and emotion quantification on full-channel EEG data, replacing traditional category discrimination and achieving high efficiency, high precision, and detailed results.

[0005] To achieve the above-mentioned purposes, the technical solution adopted by the present application is as follows: an EEG emotion analysis method based on a deep neural network, comprising the following steps:

[0006] Step 1: Collect multi-channel data, extract multi-dimensional features from the data, and perform smoothing and standardization processing on the features;

[0007] Step 2: Input the processed features into a CNN-LSTM model, analyze the features, and obtain emotional features;

[0008] Step 3: Further analyze the emotional features and signal feature dimensions, and output accurate emotion scores.

[0009] As a further improvement of the present application, step 1 is specifically as follows:

[0010] The EEG dataset is read into a Raw object and loaded with the corresponding [1, 0, -1] label at the same time, for each Raw object, it is decomposed into five frequency bands of δ, θ, α, β and γ, and feature extraction is performed, including calculating the power spectral density of each frequency band to extract its frequency domain features; time domain features are extracted using short-time Fourier transform; differential entropy is calculated to extract nonlinear dynamic features; the extracted features are combined into a feature array and feature smoothing processing is performed using a Kalman filter and feature standardization is performed using StandardScaler.

[0011] As a further improvement of the present application, in step 2, the CNN-LSTM model specifically includes:

[0012] The first convolutional layer has 32 convolutional kernels with a size of 3*3 and uses a ReLU activation function; the first pooling layer is a maximum pooling layer with a pooling window of 2*2; the second convolutional layer has 64 convolutional kernels with a size of 3*3 and uses a ReLU activation function; the second pooling layer is a maximum pooling layer with a pooling window of 2*2; the third convolutional layer has 128 convolutional kernels with a size of 3*3 and uses a ReLU activation function; the third pooling layer is a maximum pooling layer with a pooling window of 2*2; the fourth convolutional layer has 256 convolutional kernels with a size of 3*3 and uses a ReLU activation function; the fourth pooling layer is a maximum pooling layer with a pooling window of 2*2; the fifth convolutional layer has 512 convolutional kernels with a size of 3*3 and uses a ReLU activation function; the fifth pooling layer is a maximum pooling layer with a pooling window of 2*2; the Dropout layer sets the dropout rate to 0.25; the first LSTM layer is configured with 100 units and returns the entire sequence; the second LSTM layer is configured with 100 units and returns the entire sequence; the third LSTM layer is configured with 100 units and returns the output of the last time step; a Dropout layer is added after each LSTM layer with a dropout rate of 0.5; the fully connected layer uses 128 units and uses a ReLU activation function; the output layer has one unit and uses a tanh activation function to output the emotion value; in the entire CNN-LSTM model, the mean square error is used as the loss function and the Adam optimizer is set.

[0013] As a further improvement of the present application, step 3 is specifically as follows:

[0014] Firstly, the recursive feature elimination method is used to remove features step by step and evaluate the performance of the model; secondly, the Pearson correlation coefficient between the sentiment features and the signal features is calculated to analyze the linear relationship between each feature and the sentiment score; thirdly, principal component analysis is performed on the extracted high-dimensional features to reduce the dimension and retain the most explanatory features.

[0015] The application uses a hybrid deep learning framework of CNN+LSTM to process emotional information in brain waves. Unlike traditional classification methods, the application realizes continuous quantification of emotions, i.e. by assigning a score from 1 to -1 to emotions (1 represents positive, 0 represents neutral, and -1 represents negative). This method not only optimizes the granularity of emotion recognition, but also improves the accuracy of recognition. Compared with other deep learning methods, the application has significantly improved accuracy, better solving the problems of low signal-to-noise ratio and unstable signals in processing brain wave data in the prior art.

[0016] The application has the following beneficial effects:

[0017] The application uses a hybrid deep learning network of CNN+LSTM to process emotional information in brain waves through continuous quantification, thereby overcoming the shortcomings of the prior art; it aims to improve the fineness and accuracy of emotion recognition by carefully analyzing brain wave signals to achieve emotion scores from -1 to 1, to more truly and comprehensively reflect the emotional state of people. In addition, the application uses the powerful features of deep learning to automatically extract and select features, effectively improving the processing capability of low signal-to-noise ratio and unstable signals, making emotion recognition more reliable and effective in various practical application environments. BRIEF DESCRIPTION OF DRAWINGS

[0018] Figure 1 is a schematic diagram of the data acquisition electrode position in the embodiment of the application;

[0019] Figure 2 is a flowchart of the electroencephalogram emotion analysis method in the embodiment of the application;

[0020] Figure 3 is a system block diagram in the embodiment of the application;

[0021] Figure 4 is a structural schematic diagram of the CNN-LSTM model in the embodiment of the application. DETAILED DESCRIPTION

[0022] The embodiments of the application will be described in detail below with reference to the accompanying drawings.

[0023] EMBODIMENT

[0024] Hardware device: multi-channel EEG acquisition device (function: collect the user's multi-channel device), display (show the EEG emotion recognition result to the user).

[0025] Software device: data processing module (function: filter, resample, denoise, normalize and standardize the original EEG), deep learning model (function: analyze the emotional features of EEG and return the emotion value), result output module (return the result to the display), device with GPU or cloud server (ensure the calculation speed, analysis accuracy and processing efficiency).

[0026] As shown in Figure 2 and Figure 3 , an EEG emotion analysis method based on deep neural network includes:

[0027] This embodiment uses the EEG dataset SEED provided by the BCMI laboratory of Shanghai Jiaotong University. The data acquisition electrode position is as shown in Figure 1 . First, the EEG data of 45 experiments in the dataset is read into the Raw object with a sampling frequency of 200Hz and a time window of 0.5s, and the corresponding [1, 0, -1] label is loaded at the same time.

[0028] For each Raw object, it is decomposed into five frequency bands: δ, θ, α, β, and γ, and the following features are extracted:

[0029] For each frequency band, its PSD (power spectral density) is calculated to extract its frequency domain features; STFT (Short Time Fourier Transform) is used to extract its time domain features; differential entropy is calculated to extract its nonlinear dynamic features.

[0030] After obtaining these features, this embodiment forms a feature array and uses a Kalman filter for feature smoothing (n_dim_state=n_dim_obs=2, n_iter=5) and StandardScaler for feature standardization.

[0031] For the emotional features in a segment of EEG, the time domain features (including peak value, skewness, kurtosis, energy), frequency domain features (including frequency band energy, spectral entropy, spectral centroid), time-frequency domain features (including short-time Fourier transform), nonlinear features (multi-scale entropy, fractal dimension), EEG frequency band features (including α / β wave ratio, θ / β wave ratio, α wave relative energy)

[0032] This embodiment innovatively constructs a CNN+LSTM deep learning network for the complex physiological signal of EEG. As shown inFigure 4 As shown, the network is set as follows: the first convolutional layer, 32 convolutional kernels, size 3*3, using ReLU activation function; the first pooling layer, maximum pooling, pooling window 2*2; the second convolutional layer, 64 convolutional kernels, size 3*3, using ReLU activation function; the second pooling layer, maximum pooling, pooling window 2*2; the third convolutional layer, 128 convolutional kernels, size 3*3, using ReLU activation function; the third pooling layer, maximum pooling, pooling window 2*2; the fourth convolutional layer, 256 convolutional kernels, size 3*3, using ReLU activation function; the fourth pooling layer, maximum pooling, pooling window 2*2; the fifth convolutional layer, 512 convolutional kernels, size 3*3, using ReLU activation function; the fifth pooling layer, maximum pooling, pooling window 2*2; the Dropout layer, set the dropout rate to 0.25; the first layer of LSTM, configured with 100 units, returns the entire sequence; the second layer of LSTM, configured with 100 units, returns the entire sequence; the third layer of LSTM, configured with 100 units, returns the output of the last time step; add a Dropout layer after each LSTM layer, set the dropout rate to 0.5; the fully connected layer, using 128 units, using ReLU activation function; the output layer, 1 unit, using tanh activation function, outputting the emotion value. In the entire neural network, the MSE (Mean Square Error) is used as the loss function, and the Adam optimizer is set, the initial learning rate is 0.001, beta_1=0.9, beta_2=0.999, epsilon=1e-08, clipvalue=0.5.

[0033] Subsequently, the model is trained for the SEED dataset, and the smoothed features after processing the SEED dataset are used as the model input, the batch_size is set to 32, the training times are set to 200, and the early stopping mechanism is set when the model converges, and the model is saved.

[0034] For the analysis of the dimensions of emotional features and signal features, first, the Recursive Feature Elimination (RFE) method is used to remove features step by step and evaluate the performance of the model. Second, the Pearson Correlation Coefficient between emotional features and signal features is calculated to analyze the linear relationship between each feature and the emotional score. Third, principal component analysis is performed on the extracted high-dimensional features to reduce the dimension and retain the most explanatory features. PCA can transform the original high-dimensional feature data into a new set of low-dimensional features through linear transformation. These new features are called principal components, which retain as much important information in the original data as possible. Then, for the time series features extracted by the LSTM model, Dynamic Time Warping (DTW) analysis is performed to compare the changes in emotional features at different time periods and find the key time points of emotional state changes. Finally, the Generalized Additive Models (GAM) are used to analyze the interaction effects between features to understand the influence of multi-feature combinations on the prediction of emotional scores.

[0035] The main contribution of this model is to change the EEG emotion signal prediction from discrete values to continuous values. The actual implementation method is as follows: first, select the appropriate loss function and activation function. When dealing with regression problems, the mean squared error (MSE) is used as the loss function, so that the predicted value of the model is as close to the true value as possible. For the output layer, the tanh activation function is used, which has an output range of -1 to 1, which meets the requirement of limiting the output of the model to continuous values between -1 and 1. In the specific implementation, first, the SEED dataset containing EEG signals and their corresponding emotion labels is collected and preprocessed, and the labels are -1, 0 and 1. Then, a simple neural network structure is defined, including the input layer, hidden layer and output layer. The input layer receives the preprocessed EEG signal data, the hidden layer processes the data through a series of linear transformations and nonlinear activation functions, and the output layer uses the tanh activation function to ensure that the output value is within the range of -1 to 1. Next, initialize the network, loss function and optimizer, and train the model. During training, the dataset is divided into training set and validation set to evaluate the performance of the model. For each training batch, input data and corresponding labels are obtained from the training set, then input data is passed to the model, output value is calculated, and compared with the true label, loss value is calculated. Through the backpropagation algorithm, the loss value is transmitted back to each network parameter, and the optimizer is used to update the parameters to minimize the loss function. Through this method, although the labels in the training set are discrete -1, 0 and 1, the final output of the model can be continuous values within the range of -1 to 1, so as to more accurately reflect the emotional state in the EEG signal. This method not only improves the accuracy and robustness of the model in dealing with emotion prediction tasks, but also provides more rich emotion score representation, so that the model has higher value and interpretability in practical application. Through this technology, discrete labels are successfully mapped to continuous emotion scores, providing a new solution for the field of EEG emotion computing.

[0036] Finally, the EEG data from the Digital Fusion Laboratory of Chinese Culture Inheritance and Global Communication of Sichuan University was used to further verify the model. A 3-fold cross-validation without replacement and a grid search strategy were used to find the best hyperparameter settings. Further training and improvement were carried out, and finally the best fitting innovative CNN+LSTM structure EEG emotion quantification analysis model was obtained.

[0037] The embodiment introduces an innovative quantitative analysis method in the field of electroencephalogram emotion analysis, aiming to improve the efficiency of emotion analysis, reduce memory occupation, and significantly improve the analysis accuracy. By comparing the accuracy of existing models, as shown in the following table, it is found that the accuracy of traditional CNN model in electroencephalogram emotion analysis is 93.12%, ResNet model reaches 94.95%, and LSTM and BiLSTM are 90.12% and 84.21% respectively, and the accuracy of SRU model is 83.13%. Compared with these prior art, the accuracy of the model of the embodiment in electroencephalogram emotion analysis is significantly improved to 97.67%.

[0038] Models Precision Ensemble CNN 93.12％ ResNet 94.95％ LSTM 90.12％ BiLSTM 84.21％ SRU 83.13％ OURS 97.67％

[0039] This significant improvement not only highlights the efficiency and low resource consumption advantage of the invention in processing complex electroencephalogram data, but also highlights its potential value in practical application. By accurately quantifying emotional state, the technology provides strong support for further emotional research and related applications, such as emotion-driven human-computer interaction system. This achievement not only reflects the foresight of technology, but also provides a new perspective and efficient solution for the development direction of future emotion analysis technology.

[0040] The above-described embodiments only express the specific implementation of the present application, which is described in detail and specifically, but it cannot be understood as a limitation on the scope of the present patent. It should be noted that for ordinary skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are within the scope of protection of the present application.

Claims

1.A deep neural network-based EEG emotion analysis method, characterized in that, Comprise the following steps: Step 1, collect multi-channel data, multi-dimensional feature extraction is carried out on the data, and the features are smoothed and standardized; the step 1 is specifically as follows: The EEG data set is read into a Raw object and the corresponding [1, 0, -1] label is loaded at the same time, for each Raw object, it is decomposed into five frequency bands of delta, theta, alpha, beta and gamma, and feature extraction is carried out, including calculating the power spectral density of each frequency band, extracting its frequency domain features; time domain features are extracted by using short-time Fourier transform; differential entropy is calculated, and nonlinear dynamic features are extracted; the extracted features are combined into a feature array and are subjected to feature smoothing treatment using Kalman filter and feature standardization using StandardScaler; Step 2, input the processed features into the CNN-LSTM model, analyze the features and obtain the emotional features; in step 2, the CNN-LSTM model specifically comprises: The first convolutional layer, 32 convolutional kernels, size 3*3, using ReLU activation function; the first pooling layer, maximum pooling, pooling window 2*2; the second convolutional layer, 64 convolutional kernels, size 3*3, using ReLU activation function; the second pooling layer, maximum pooling, pooling window 2*2; the third convolutional layer, 128 convolutional kernels, size 3*3, using ReLU activation function; the third pooling layer, maximum pooling, pooling window 2*2; the fourth convolutional layer, 256 convolutional kernels, size 3*3, using ReLU activation function; the fourth pooling layer, maximum pooling, pooling window 2*2; the fifth convolutional layer, 512 convolutional kernels, size 3*3, using ReLU activation function; the fifth pooling layer, maximum pooling, pooling window 2*2; the Dropout layer, set the dropout rate to 0.25; the first layer LSTM, configure 100 units, return the entire sequence; the second layer LSTM, configure 100 units, return the entire sequence; the third layer LSTM, configure 100 units, return the output of the last time step; add a Dropout layer after each LSTM layer, set the dropout rate to 0.5; the fully connected layer uses 128 units, uses ReLU activation function; the output layer, 1 unit, uses tanh activation function, outputs the emotion value; in the whole CNN-LSTM model, mean square error is used as the loss function, and Adam optimizer is set; Step 3, further analyze the emotional features and signal feature dimensions, and output continuous accurate emotion scores in the range of [-1, 1]. 2.The deep neural network-based EEG emotion analysis method according to claim 1, wherein, The step 3 is specifically as follows: Firstly, the recursive feature elimination method is used to remove the features step by step and evaluate the model performance; secondly, the Pearson correlation coefficient between the emotional features and the signal features is calculated to analyze the linear relationship between each feature and the emotion score; thirdly, principal component analysis is performed on the extracted high-dimensional features to reduce the dimension and retain the most explanatory features.

Images