EEG emotion analysis method based on deep neural network

Abstract

The invention discloses an EEG emotion analysis method based on a deep neural network, and the method comprises the steps: collecting multi-channel data, carrying out the multi-dimensional feature extraction of the data, and carrying out the smoothing and standardization processing of the features; inputting the processed features into a CNN-LSTM model, analyzing the features and obtaining emotion features; further analyzing the emotional features and the signal feature dimensions, and outputting accurate emotional scores; according to the method, multi-dimensional feature extraction and emotion quantitative discrimination of all-channel electroencephalogram data are realized, traditional category discrimination is replaced, and the effects of high efficiency, high precision and meticulous effects are achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of physiological digital information processing, and in particular to an EEG emotion analysis method based on a deep neural network. Background Art

[0002] Emotion research is an interdisciplinary field that brings together research results from multiple disciplines such as computer science, psychology, cognitive science, and neuroscience to deeply explore the complexity of human emotions. In this field, traditional methods such as facial recognition and motion analysis are often affected by subjective control and disguise, so the recognition accuracy is limited. As an emerging technology, brain wave analysis is considered to be an effective way to improve the accuracy of emotion recognition because it is difficult to be disguised and artificially changed. With the advancement of wearable technology and dry electrode technology, the acquisition of brain wave data is no longer limited to laboratory environments, which greatly facilitates practical applications.

[0003] Although the technology for obtaining brain wave data has made significant progress, the asymmetry, instability and low signal-to-noise ratio of brain wave signals are still the main challenges in data processing. At present, most studies are still focused on how to deal with these problems by dividing the emotional signals of brain waves into positive, neutral, negative and other categories through classification tasks; although this method can reflect people's emotional state to a certain extent, it ignores the continuity and complexity of emotions, resulting in insufficient accuracy and detail in emotion recognition. In addition, there are also studies that attempt to quantitatively evaluate emotions from three dimensions: pleasure, arousal and dominance. When processing brain wave data, we often face problems such as unstable signals and low signal-to-noise ratio. These technical challenges limit the accuracy and practicality of emotion recognition. Summary of the invention

[0004] In order to solve the problems existing in the prior art, the purpose of the present invention is to provide an EEG emotion analysis method based on deep neural network. The present invention realizes multi-dimensional feature extraction and emotion quantification judgment of full-channel EEG data, replaces the traditional category judgment, and achieves high efficiency, high precision and meticulous effect.

[0005] To achieve the above object, the technical solution adopted by the present invention is: an EEG emotion analysis method based on a deep neural network, comprising the following steps:

[0006] Step 1: Collect multi-channel data, extract multi-dimensional features from the data, and smooth and standardize the features;

[0007] Step 2: Input the processed features into the CNN-LSTM model, analyze the features and derive the sentiment features;

[0008] Step 3: Further analyze the emotional features and signal feature dimensions and output accurate emotional scores.

[0009] As a further improvement of the present invention, the step 1 is specifically as follows:

[0010] The EEG dataset is read as a Raw object and the corresponding [1,0,-1] labels are loaded at the same time. For each Raw object, it is decomposed into five frequency bands: δ, θ, α, β, and γ, and feature extraction is performed, including calculating the power spectral density of each frequency band and extracting its frequency domain features; using short-time Fourier transform to extract time domain features; calculating differential entropy to extract nonlinear dynamic features; organizing the extracted features into a feature array and using the Kalman filter for feature smoothing and StandardScaler for feature standardization.

[0011] As a further improvement of the present invention, in step 2, the CNN-LSTM model specifically includes:

[0012] The first convolution layer, 32 convolution kernels, size is 3*3, using ReLU activation function; the first pooling layer, maximum pooling, pooling window is 2*2; the second convolution layer, 64 convolution kernels, size is 3*3, using ReLU activation function; the second pooling layer, maximum pooling, pooling window is 2*2; the third convolution layer, 128 convolution kernels, size is 3*3, using ReLU activation function; the third pooling layer, maximum pooling, pooling window is 2*2; the fourth convolution layer, 256 convolution kernels, size is 3*3, using ReLU activation function; the fourth pooling layer, maximum pooling, pooling window is 2*2; the fifth convolution layer, 512 convolution kernels, size is 3*3, using ReLU activation function; the fifth pooling Layer, maximum pooling, pooling window is 2*2; Dropout layer, set the dropout rate to 0.25; the first layer of LSTM, configure 100 units, return the entire sequence; the second layer of LSTM, configure 100 units, return the entire sequence; the third layer of LSTM, configure 100 units, return the output of the last time step; add a Dropout layer after each layer of LSTM, set the dropout rate to 0.5; fully connected layer, use 128 units, use ReLU activation function; output layer, 1 unit, use tanh activation function, output emotion value; in the entire CNN-LSTM model, use mean square error as the loss function, and set Adam optimizer.

[0013] As a further improvement of the present invention, the step 3 is specifically as follows:

[0014] First, the recursive feature elimination method is used to gradually remove features and evaluate the model performance. Secondly, the Pearson correlation coefficient between sentiment features and signal features is calculated to analyze the linear relationship between each feature and the sentiment score. Thirdly, principal component analysis is performed on the extracted high-dimensional features to reduce the dimensionality and retain the most explanatory features.

[0015] The present invention uses a hybrid deep learning framework of CNN+LSTM to process emotional information in brain waves. Different from traditional classification methods, the present invention realizes the continuous quantification of emotions, that is, it evaluates emotions by assigning scores from 1 to -1 (1 represents positive, 0 represents neutral, and -1 represents negative). This method not only optimizes the fine-grainedness of emotion recognition, but also improves the accuracy of recognition. Compared with other deep learning methods, the present invention has significantly improved the accuracy rate and better solves the problems of low signal-to-noise ratio and unstable signal faced by the prior art when processing brain wave data.

[0016] The beneficial effects of the present invention are:

[0017] The present invention is based on a hybrid deep learning network of CNN+LSTM, and processes the emotional information in brain waves by continuous quantization, thereby overcoming the shortcomings of the prior art; it aims to improve the fineness and accuracy of emotion recognition, and achieves an emotion score from -1 to 1 by carefully analyzing brain wave signals, so as to more realistically and comprehensively reflect the emotional state of a person. In addition, the present invention uses the powerful function of deep learning to automatically extract and select features, effectively improving the processing ability of low signal-to-noise ratio and unstable signals, making emotion recognition more reliable and effective in various practical application environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 A schematic diagram of the position of the data acquisition electrodes in an embodiment of the present invention;

[0019] Figure 2 Flow chart of the EEG emotion analysis method in an embodiment of the present invention;

[0020] Figure 3 is a system block diagram in an embodiment of the present invention;

[0021] Figure 4 Schematic diagram of the structure of the CNN-LSTM model in an embodiment of the present invention. DETAILED DESCRIPTION

[0022] The embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0023] Example

[0024] Hardware equipment: multi-channel EEG acquisition equipment (function: multi-channel equipment to collect user's information), display (to display EEG emotion recognition results to the user).

[0025] Software equipment: data processing module (function: filtering, resampling, denoising, normalizing and standardizing the original EEG), deep learning model (function; analyzing the emotional characteristics of EEG and returning the emotional value), result output module (returning the results to the display), device with GPU or cloud server (to ensure computing speed, analysis accuracy and processing efficiency).

[0026] like Figure 2 and Figure 3 As shown, an EEG emotion analysis method based on a deep neural network includes:

[0027] This example uses the EEG dataset SEED provided by the BCMI laboratory of Shanghai Jiao Tong University. The data acquisition electrode locations are as follows: Figure 1 First, the EEG data of 45 experiments in the dataset are read into the Raw object with a sampling frequency of 200 Hz and a time window of 0.5 s, and the corresponding [1, 0, -1] labels are loaded at the same time.

[0028] For each Raw object, it is decomposed into five frequency bands: δ, θ, α, β, and γ, and the following features are extracted:

[0029] The PSD (power spectral density) of each frequency band was calculated to extract its frequency domain features; the STFT (short time Fourier transform) was used to extract its time domain features; and the differential entropy was calculated to extract its nonlinear dynamic characteristics.

[0030] After obtaining these features, this embodiment organizes them into a feature array and uses a Kalman filter to perform feature smoothing processing (n_dim_state=n_dim_obs=2, n_i ter=5) and uses StandardScaler to standardize the features.

[0031] For the emotional features in a piece of EEG, the above features are mainly used to extract time domain features (including peak value, skewness, kurtosis, energy), frequency domain features (including frequency band energy, spectrum entropy, spectrum centroid), time-frequency domain features (including short-time Fourier transform), nonlinear features (multi-scale entropy, fractal dimension), EEG frequency band features (including α / β wave ratio, θ / β wave ratio, α wave relative energy)

[0032] This embodiment constructs an innovative CNN+LSTM deep learning network for complex physiological signals such as EEG. Figure 4 As shown in the figure, the network settings are as follows: the first convolution layer, 32 convolution kernels, size is 3*3, using ReLU activation function; the first pooling layer, maximum pooling, pooling window is 2*2; the second convolution layer, 64 convolution kernels, size is 3*3, using ReLU activation function; the second pooling layer, maximum pooling, pooling window is 2*2; the third convolution layer, 128 convolution kernels, size is 3*3, using ReLU activation function; the third pooling layer, maximum pooling, pooling window is 2*2; the fourth convolution layer, 256 convolution kernels, size is 3*3, using ReLU activation function; the fourth pooling layer, maximum pooling, pooling window is 2*2; the fifth convolution layer, 512 convolution kernels, size is 3*3, using ReLU activation function; The kernel size is 3*3, and the ReLU activation function is used; the fifth pooling layer is the maximum pooling, and the pooling window is 2*2; the Dropout layer sets the dropout rate to 0.25; the first layer of LSTM is configured with 100 units and returns the entire sequence; the second layer of LSTM is configured with 100 units and returns the entire sequence; the third layer of LSTM is configured with 100 units and returns the output of the last time step; a Dropout layer is added after each layer of LSTM, and the dropout rate is set to 0.5; the fully connected layer uses 128 units and uses the ReLU activation function; the output layer has 1 unit and uses the tanh activation function to output the emotion value. In the entire neural network, MSE (mean square error) is used as the loss function, and the Adam optimizer is set, with the initial learning rate of 0.001, beta_1=0.9, beta_2=0.999, eps i lon=1e-08, and client value=0.5.

[0033] Then the model is trained on the SEED dataset, and the smooth features after processing the SEED dataset are used as the model input. The batch_size is set to 32, the number of training times is set to 200, and the early stopping mechanism is set. When the model converges, save the model.

[0034] For the analysis of the emotional features and signal feature dimensions, the Recursive Feature Elimination (RFE) method is first used to gradually remove features and evaluate the model performance; secondly, the Pearson Correlation Coefficient between the emotional features and the signal features is calculated to analyze the linear relationship between each feature and the emotional score; then the principal component analysis is performed on the extracted high-dimensional features to reduce the dimension and retain the most explanatory features. PCA can transform the original high-dimensional feature data into a set of new low-dimensional features through linear transformation. These new features are called principal components, which can retain the important information in the original data as much as possible; then, the dynamic time warping (DTW) analysis is performed on the time series features extracted by the LSTM model to compare the changes in emotional features in different time periods and find the key time points of emotional state changes; finally, the generalized additive models (GAM) are used to analyze the interaction effects between features and understand the impact of multi-feature combinations on the prediction of emotional scores.

[0035] The main contribution of this model is to transform the prediction of EEG emotion signals from discrete values ​​to continuous values. The actual implementation method is as follows. First, the appropriate loss function and activation function are selected. When dealing with regression problems, the mean squared error (MSE) is used as the loss function to make the predicted value of the model as close to the true value as possible. For the output layer, the tanh activation function is used. The output range of this function is -1 to 1, which just meets the requirements, that is, the output of the model is limited to a continuous value between -1 and 1. In the specific implementation, the SEED dataset is first collected and preprocessed. The dataset contains EEG signals and their corresponding emotion labels, which are -1, 0, and 1. Then, a simple neural network structure is defined, including an input layer, a hidden layer, and an output layer. The input layer receives the preprocessed EEG signal data, the hidden layer processes the data through a series of linear transformations and nonlinear activation functions, and the output layer uses the tanh activation function to ensure that the output value is in the range of -1 to 1. Next, the network, loss function, and optimizer are initialized, and the model is trained. During the training process, the dataset is divided into a training set and a validation set to evaluate the performance of the model. For each training batch, the input data and corresponding labels are obtained from the training set, and then the input data is passed to the model, the output value is calculated, and compared with the true label to calculate the loss value. Through the back-propagation algorithm, the loss value is back-propagated to each network parameter, and the optimizer is used to update the parameters to minimize the loss function. In this way, it is ensured that although the labels in the training set are discrete -1, 0, and 1, the output of the final model can be a continuous value in the range of -1 to 1, thereby more accurately reflecting the emotional state in the EEG signal. This method not only improves the accuracy and robustness of the model when dealing with emotion prediction tasks, but also provides a richer representation of emotion scores, making the model more valuable and interpretable in practical applications. Through this technology, discrete labels are successfully mapped to continuous emotion scores, providing a new solution for the field of EEG emotion computing.

[0036] Finally, the EEG data from the Digital Fusion Laboratory of Chinese Cultural Heritage and Global Communication of Sichuan University was used to further verify the model, and a 3-fold cross-validation without repeated sampling and a grid search strategy were used to find the best hyperparameter settings. After further training and improvement, an innovative CNN+LSTM structure EEG emotion quantitative analysis model with the best fitting effect was finally obtained.

[0037] This embodiment introduces an innovative quantitative analysis method in the field of EEG emotion analysis, aiming to improve the efficiency of emotion analysis, reduce memory usage, and significantly improve the analysis accuracy. By comparing the accuracy of existing models, as shown in the following table, it is found that the accuracy of the traditional CNN model in EEG emotion analysis is 93.12%, the ResNet model reaches 94.95%, while the LSTM and BiLSTM are 90.12% and 84.21% respectively, and the accuracy of the SRU model is 83.13%. Compared with these existing technologies, the accuracy of the model in EEG emotion analysis is significantly improved to 97.67%.

[0038] Models Precision Ensemble CNN 93.12％ ResNet 94.95％ LSTM 90.12％ BiLSTM 84.21％ SRU 83.13％ OURS 97.67％

[0039] This significant improvement not only demonstrates the high efficiency and low resource consumption advantages of the present invention in processing complex EEG data, but also highlights its potential value in practical applications. By accurately quantifying emotional states, this technology provides strong support for further emotion research and related applications, such as emotion-driven human-computer interaction systems. This achievement not only reflects the forward-looking nature of the technology, but also provides a new perspective and efficient solution for the future development direction of emotion analysis technology.

[0040] The above-mentioned embodiments only express the specific implementation of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the present invention. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention.

Claims

1. An EEG emotion analysis method based on deep neural network, characterized in that: The following steps are involved: Step 1: Collect multi-channel data, extract multi-dimensional features from the data, and smooth and standardize the features; Step 2: Input the processed features into the CNN-LSTM model, analyze the features and derive the sentiment features; Step 3: Further analyze the emotional features and signal feature dimensions and output accurate emotional scores.

2. The EEG emotion analysis method based on deep neural network according to claim 1, characterized in that: The step 1 is specifically as follows: Read the EEG dataset as a Raw object and load the corresponding [1,0,-1] labels at the same time. For each Raw object, decompose it into five frequency bands: δ, θ, α, β, and γ, and perform feature extraction, including calculating the power spectral density of each frequency band and extracting its frequency domain features. Use short-time Fourier transform to extract time domain features; calculate differential entropy to extract nonlinear dynamic features; organize the extracted features into feature arrays and use Kalman filter for feature smoothing and StandardScaler for feature standardization.

3. The EEG emotion analysis method based on deep neural network according to claim 2 is characterized in that: In step 2, the CNN-LSTM model specifically includes: The first convolution layer, 32 convolution kernels, size is 3*3, using ReLU activation function; the first pooling layer, maximum pooling, pooling window is 2*2; the second convolution layer, 64 convolution kernels, size is 3*3, using ReLU activation function; the second pooling layer, maximum pooling, pooling window is 2*2; the third convolution layer, 128 convolution kernels, size is 3*3, using ReLU activation function; the third pooling layer, maximum pooling, pooling window is 2*2; the fourth convolution layer, 256 convolution kernels, size is 3*3, using ReLU activation function; the fourth pooling layer, maximum pooling, pooling window is 2*2; the fifth convolution layer, 512 convolution kernels, size is 3*3, using ReLU activation function; the fifth pooling Layer, maximum pooling, pooling window is 2*2; Dropout layer, set the dropout rate to 0.25; the first layer of LSTM, configure 100 units, return the entire sequence; the second layer of LSTM, configure 100 units, return the entire sequence; the third layer of LSTM, configure 100 units, return the output of the last time step; add a Dropout layer after each layer of LSTM, set the dropout rate to 0.5; fully connected layer, use 128 units, use ReLU activation function; output layer, 1 unit, use tanh activation function, output emotion value; in the entire CNN-LSTM model, use mean square error as the loss function, and set Adam optimizer.

4. The EEG emotion analysis method based on deep neural network according to claim 3 is characterized in that: The step 3 is as follows: First, the recursive feature elimination method is used to gradually remove features and evaluate the model performance. Secondly, the Pearson correlation coefficient between sentiment features and signal features is calculated to analyze the linear relationship between each feature and the sentiment score. Thirdly, principal component analysis is performed on the extracted high-dimensional features to reduce the dimensionality and retain the most explanatory features.

Images