Emotion recognition method based on EEG space-time frequency feature fusion

Abstract

The invention discloses an emotion recognition method based on EEG space-time frequency feature fusion, which comprises the following steps: S1, carrying out a series of preprocessing steps on an original EEG signal, including baseline correction, standardization and time window addition; s2, performing form conversion on the electroencephalogram signals processed in the S1 to obtain spatial information and time-frequency information of the electroencephalogram signals; s3, multi-aspect feature extraction is carried out; s4, fusing the extracted emotion features, and performing multi-feature emotion recognition; s5, further extracting time features of the electroencephalogram signals by using BILSTM added with a channel attention mechanism, and performing deep fusion of the features; and S6, obtaining an emotion classification result. According to the emotion recognition method based on EEG space-time-frequency feature fusion, feature information of the time dimension, the frequency dimension and the space dimension of the EEG signals is fully considered, the highest recognition accuracy is achieved on the awakening dimension and the titer dimension, the emotion recognition accuracy and robustness are improved, and the effectiveness of the method is verified.

Description

Technical Field

[0001] The present invention relates to the field of emotion recognition technology, and in particular to an emotion recognition method based on EEG spatiotemporal-frequency feature fusion. Background Art

[0002] As a comprehensive state produced by people under external stimulation, emotion recognition plays an important role in the process of expressing and perceiving information in daily life. Given the accuracy and non-invasiveness of EEG signals, more and more research focuses on using EEG for emotion recognition. In recent years, the rapid development of Internet technology has promoted the advent of the artificial intelligence era, and deep learning technology has also flourished. Although various deep learning frameworks have been applied to the field of emotion recognition, it is still challenging to fully extract the multidimensional features of EEG signals and design a model that can reduce information loss and improve the accuracy and robustness of emotion recognition. Summary of the invention

[0003] The purpose of the present invention is to provide an emotion recognition method based on the fusion of EEG time-space-frequency features, which fully considers the characteristic information of the three dimensions of time-frequency-space of EEG signals, achieves the highest recognition accuracy in the dimensions of arousal and valence, improves the accuracy and robustness of emotion recognition, and verifies the effectiveness of this method.

[0004] To achieve the above object, the present invention provides an emotion recognition method based on EEG spatiotemporal-frequency feature fusion, comprising:

[0005] S1, a series of preprocessing steps were performed on the raw EEG signals, including baseline correction, normalization, and adding time windows;

[0006] S2, converting the EEG signal processed by S1 to obtain spatial information and time-frequency information of the EEG signal;

[0007] S3, perform multi-faceted feature extraction;

[0008] S4, integrating the extracted emotional features and performing multi-feature emotion recognition;

[0009] S5. Use BILSTM with added channel attention mechanism to further extract the temporal features of EEG signals and perform deep fusion of features;

[0010] S6. Obtain sentiment classification results.

[0011] Preferably, in the signal form conversion in step S2:

[0012] S21. Based on the PLI theory, a dynamic brain function network is constructed. The spatial characteristics of the signal are captured by extracting the topological characteristics of the brain network. The sliding window method is used to process the EEG signal in segments. The phase synchronization between nodes is calculated using PLI to generate a relationship matrix reflecting the node connection strength. The original expression is:

[0013]

[0014] Where, E<·> represents the expected operation on the data in the time window; N represents the time point; Represents two signals at time t n The phase difference at ; sign is a sign function; the value of PLI is between 0 and 1. If PLI is closer to 1, it means the strength of the connection is stronger, otherwise, it means the strength of the connection is weaker;

[0015] The relationship matrix constructed based on PLI theory needs further threshold processing. The adaptive threshold method is used to perform threshold processing on the relationship matrix of each time window. According to the characteristics of the data, the adaptive threshold is expressed as:

[0016] threshold=mean_pli-std_pli / 5 (2)

[0017] Among them, mean_pli represents the mean of the relationship matrix, and std_pli represents the variance of the relationship matrix;

[0018] S22. Use the short-time Fourier transform of the Hamming window function to convert the EEG signal into a time-frequency diagram, which is expressed as:

[0019] S(n,ω)=|X(n,ω)| 2 (3)

[0020]

[0021] Among them, S(n,ω) represents the time-frequency diagram; X(n,ω) is the short-time Fourier transform, which is a two-dimensional function that relates time and frequency; x(m) is the input signal; w(nm) is the window function, which uses a Hanning window with a window length of 1s and a superposition of 0.75s.

[0022] Preferably, in step S3, in feature extraction:

[0023] S31. Based on the construction of the dynamic brain functional network, eight graph theory features of the brain functional network were extracted, including clustering coefficient, shortest path length, global efficiency, local efficiency, betweenness centrality, compactness, degree centrality, and eigenvector centrality.

[0024] S32, based on the ResNet-18 model, fine-tune it with the time-frequency graph as input to extract deep time-frequency features;

[0025] S33. Extract power spectrum density and differential entropy features of the EEG data in each time window.

[0026] Preferably, in step S4, a feature splicing method is used to combine the time-frequency-space domain features of the collected EEG signals, and the last dimension of the feature data of each field is spliced ​​to obtain information of fused multi-feature data, and the feature information of the EEG signals is characterized by using the features of different fields, which is calculated as follows:

[0027] F = concatenate(f deep ,f brain ,f de ,f psd ) (5)

[0028] Among them, F is the fusion feature; f deep It is the deep time-frequency features extracted by fine-tuning the pre-trained ResNet-18; f brain It is the spatial domain feature extracted by constructing a dynamic brain functional network; f de is the extracted differential entropy; f psd is the extracted power spectral density;

[0029] Preferably, in step S5, a bidirectional long short-term memory network structure with a channel attention mechanism is used to further extract the temporal features of the EEG signal and perform deep fusion of the features. The expression of the channel attention mechanism is as follows:

[0030] M c (F)=softmax(MLP(AvgPool(F))+MLP(MaxPool(F))) (6)

[0031] Among them, softmax is the activation function used to map the output probability; AvgPool represents average pooling; MaxPool represents maximum pooling; M c (F)∈R C×1 , represents the attention weight; C represents the number of channels;

[0032] Preferably, the flow of information is controlled by a bidirectional long short-term memory network structure, wherein:

[0033] (1) The forget gate is used to control the update of the cell state at the current time; the short-term memory output h at the previous moment t-1 and the sequence data x to be input at the current moment t As input, the probability of long-term memory being forgotten is determined by the activation function f t, the calculation process is as shown in Equation 7:

[0034] f (t) =σ(W f ·[h (t-1) ,x (t) ]+b f ) (7)

[0035] Among them, σ is the sigmoid activation function, which is used to map the calculation results to the interval [0,1]; W f is the weight matrix; b f It is a deviation;

[0036] (2) The input gate is used to determine which information in the current input should be added to the cell state; it consists of two parts. The first part is used to calculate the probability of retaining information. The calculation process equation is as follows;

[0037] i t =σ(W i ·[h t-1 ,x t ]+b i ) (8)

[0038] The second part is used to calculate the output value at the current moment. The calculation process equation is as follows:

[0039]

[0040] Among them, i t represents the probability value of retaining information; represents the output value at the current moment; σ is the sigmoid activation function, which is used to map the calculation result to the interval [0,1]; W is the weight matrix; b is the bias; then Indicates the information to be updated at the current moment;

[0041] (3) The output gate is used to control the amount of information in the short-term memory output at the current time; the new information obtained by the input gate and the forget gate is used to update the cell state C at the current time t ; Cell state C t Together with the output gate, it determines the output h of this layer t , the calculation process is as follows:

[0042]

[0043] o t =σ(W o ·[h t-1 ,x t ]+b o ) (11)

[0044] h t =ot *tanh(C t ). (12)

[0045] Therefore, the present invention adopts the above-mentioned emotion recognition method based on the fusion of EEG time, space and frequency features, fully considering the characteristic information of the three dimensions of time, frequency and space of EEG signals, achieving the highest recognition accuracy in the dimensions of arousal and valence, improving the accuracy and robustness of emotion recognition, and verifying the effectiveness of this method.

[0046] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 This is a framework diagram of the emotion recognition method based on the fusion of EEG time-space-frequency features; (a) is a 32-channel EEG signal; (b) the EEG signal is converted into a time-frequency diagram and a dynamic brain function network; (c) DE features and PSD features are extracted from the EEG signal; the pre-trained ResNet-18 is fine-tuned to extract the deep time-frequency features of the EEG signal; the spatial features of the EEG signal are manually extracted based on the dynamic brain function network; (d) the extracted emotion features are simply fused; (e) the BILSTM with a channel attention mechanism is used to further extract the temporal features of the EEG signal and perform deep fusion of the features; (f) the emotion classification results are obtained;

[0048] Figure 2 This is the architecture diagram of the fine-tuned pre-trained ResNet-18 model based on the emotion recognition method based on the fusion of EEG spatiotemporal and frequency features;

[0049] Figure 3 This is the architecture diagram of BILSTM, an emotion recognition method based on the fusion of EEG spatiotemporal and frequency features;

[0050] Figure 4 This is the structural diagram of the LSTM unit of the emotion recognition method based on the fusion of EEG spatiotemporal and frequency features;

[0051] Figure 5 This is the confusion matrix diagram of the arousal dimension of the emotion recognition method based on the fusion of EEG spatiotemporal and frequency features;

[0052] Figure 6 This is the confusion matrix diagram of the valence dimension of the emotion recognition method based on the fusion of EEG spatiotemporal and frequency features;

[0053] Figure 7 It is the single-subject emotion recognition accuracy in the arousal and valence dimensions of the emotion recognition method based on the fusion of EEG spatiotemporal and frequency features;

[0054] Figure 8It is the recognition accuracy of the five models of emotion recognition method based on the fusion of EEG spatiotemporal and frequency features in the arousal dimension;

[0055] Fig. 9 It is the recognition accuracy of five models of emotion recognition method based on the fusion of EEG temporal, spatial and frequency features in the valence dimension. DETAILED DESCRIPTION

[0056] The technical solution of the present invention is further described below through the accompanying drawings and embodiments.

[0057] like Figure 1 As shown in the figure, first, a series of preprocessing steps were performed on the raw EEG signals, including baseline correction, normalization, and adding time windows. Next, two key data conversion stages were performed to obtain the time-frequency information and spatial information of the EEG signals. In the first stage, the EEG signals were converted into time-frequency graphs. In order to better process the time-frequency graphs and extract relevant time-frequency features, the pre-trained ResNet-18 model architecture was adjusted and successfully migrated to the study. The second stage involves building a dynamic brain functional network based on the PLI theory, capturing the spatial characteristics of the signal by extracting the topological features of the brain network. In addition, power spectral density and differential entropy features were extracted to supplement the comprehensive feature information. Considering that the EEG signals are continuously collected according to each electrode channel and have a time order, in order to better capture the time information of the EEG signals of each channel, we introduced the BILSTM model that performs well on sequence data. At the same time, we also introduced the channel attention mechanism as part of the classification model for the prediction of the final emotional results. The implementation of this comprehensive method aims to improve the performance of emotion recognition on small sample datasets.

[0058] Example 1

[0059] S1, preprocessing the original EEG signal;

[0060] (1) Baseline processing: To address the problem that the baseline data before the 3 seconds of the signal data in the DEAP dataset may affect the experimental accuracy, the present invention uses a baseline averaging method to process the baseline data.

[0061] First, the first 3 seconds of the EEG signal of each subject were taken out and divided into three 1 second signal segments. The baseline average value was calculated by averaging the signal values ​​of the three segments using formula (13).

[0062]

[0063] Then, the 60s EEG signal after the baseline time of each subject was divided into 60 1s signal segments, and the baseline average value was subtracted from the signal value of each segment to obtain the signal value after baseline processing. The calculation relationship is shown in equation (14).

[0064] data_re i =raw_data i -b_mean (14)

[0065] Finally, the 60 signal segments after baseline processing are spliced ​​to obtain the complete signal data. The calculation relationship is shown in equation (15).

[0066] data = concat(data_re i ) (15)

[0067] Among them, b1, b2, b3 represent the baseline signal of 1s respectively, b_mean∈R E×S , raw_data∈R E×S , data_re i ∈R E×S , data∈R E×S×N , E represents the number of electrode channels, S represents the length of the 1s signal segment, N represents 60 signal segments, i=1,2,3...,60, and concat represents the concatenation operation. After baseline processing, the signal data format of each subject is (40,7680), 40 represents the number of electrode channels, and 7680 represents the data length.

[0068] Among them, the summary of the DEAP dataset is shown in Table 1.

[0069] Table 1 Detailed information of the DEAP dataset

[0070]

[0071] (2) Sliding time window setting and sample division: Human emotions usually change within 0.5-4 seconds, and choosing the appropriate time is crucial for emotion recognition. A large number of experiments have shown that setting the sliding time window to a window length of 3s and a step size of 1s can achieve the best recognition performance. Therefore, the present invention sets the window size to 3s (including 384 sampling points) and the step size to 2s.

[0072] According to the setting of the time window, the signal of each electrode channel is divided into 58 time windows. Therefore, based on the characteristic that each subject in the DEAP dataset conducts 40 experiments, each subject can collect approximately 2380 samples, and the data of each sample is represented as 32×384. Since only 18 subjects in the dataset have complete face video data, the 18 subjects with complete video data are selected for this emotion recognition experiment.

[0073] According to the data processing method, the arousal and valence labels are divided into categories, and the label threshold is set to 5. When the experimental score is greater than or equal to 5, the label is classified as high arousal / valence, otherwise the label is classified as low arousal / valence. After distinguishing the categories, classification is performed for each time window.

[0074] (3) Data standardization: In order to eliminate the dimensional differences between EEG signals and improve the accuracy of emotion recognition while retaining the distribution information of the original data, the present invention uses the minimum and maximum standardization method to standardize the EEG signals in each time window and uniformly map the data to the range of [0, 1]. The process can be described as:

[0075]

[0076] Among them, data represents the data of each electrode in each time window, min_data represents the minimum data point of each electrode in each time window, and max_data represents the maximum data point of each electrode in each time window.

[0077] S2, signal form conversion

[0078] S21. Construction of dynamic brain functional network

[0079] The brain is a dynamic system that is used to perform complex information interactions under different emotional states. By analyzing brain networks, we can study the collaborative working methods between different brain regions, providing a new way to gain a deeper understanding of the working principles of the brain. By recording electroencephalogram (EEG) signals, a dynamic brain functional network is constructed, and the network characteristics are analyzed to capture the spatiotemporal relationship between multiple brain regions, thereby obtaining more comprehensive and multidimensional brain spatial information, which is helpful for emotion recognition. When constructing a brain functional network, the selection of nodes and the construction of edges are very important. Usually, scalp electrodes are used as network nodes, and network edges are defined based on the correlation of scalp electrode time series. Phase lag index (PLI) is a phase-based functional connectivity analysis method, which is mainly used to study the functional connectivity between multi-channel EEG signals. In the present invention, a sliding window method is used to process EEG signals in segments, and PLI is used to calculate the phase synchronization between nodes to generate a relationship matrix reflecting the strength of node connections. Its original expression is:

[0080]

[0081] Where, E<·> represents the expected operation on the data in the time window; N represents the time point; Represents two signals at time t n The phase difference at ; sign is a sign function. The value of PLI is between 0 and 1. If PLI is closer to 1, it means the strength of the connection is stronger, otherwise, it means the strength of the connection is weaker.

[0082] The relationship matrix constructed based on PLI theory needs further thresholding. In order to avoid information loss to the greatest extent, the present invention chooses to use an adaptive thresholding method to perform thresholding on the relationship matrix of each time window. According to the characteristics of the data, the adaptive threshold is expressed as:

[0083] threshold=mean_pli-std_pli / 5 (2)

[0084] Among them, mean_pli represents the mean of the relationship matrix, and std_pli represents the variance of the relationship matrix. After selecting a suitable threshold, we set the values ​​in the relationship matrix that are less than the threshold to 0, thereby obtaining the final brain functional network.

[0085] S22, time-frequency diagram conversion

[0086] The study found that different types of spectrograms can capture different characteristics of the signal, and these features can be automatically extracted and generated through deep learning methods. Among them, the time-frequency diagram performs well in describing the time-frequency changes of the signal. Therefore, the short-time Fourier transform of the Hamming window function is used to convert the EEG signal into a time-frequency diagram. Its expression is:

[0087] S(n,ω)=|X(n,ω)| 2 (3)

[0088]

[0089] Among them, S(n,ω) represents the time-frequency diagram; X(n,ω) is the short-time Fourier transform, which is a two-dimensional function that relates time and frequency; x(m) is the input signal; w(nm) is the window function. Here we use a window length of 1s and a Hanning window with a superposition of 0.75s.

[0090] S3. Multi-faceted feature extraction

[0091] The main goal of feature extraction is to extract the most valuable information from the data to improve the efficiency of data processing and enhance the performance of the model. In this invention, three aspects of feature extraction are carried out: first, spatial feature extraction based on brain functional network; second, deep time-frequency feature extraction using time-frequency graph as input; finally, in order to supplement the comprehensive feature information, power spectrum and differential entropy feature extraction are carried out.

[0092] S31, based on the construction of the dynamic brain function network in step S21, eight graph theory features of the brain function network, namely, clustering coefficient, shortest path length, global efficiency, local efficiency, betweenness centrality, closeness, degree centrality, and eigenvector centrality, are extracted. By extracting these graph theory features, detailed information about the structure and characteristics of the brain function network can be obtained, thereby providing more comprehensive brain spatial information for emotion recognition.

[0093] S32. In order to achieve the best emotion recognition performance in the whole emotion recognition process, the present invention sets the parameters of the proposed model. The pre-trained ResNet-18 model used to extract deep features is fine-tuned. In order to minimize the loss of information, the 7×7 downsampling convolution is replaced with a 3×3 downsampling convolution, and the step size and padding size of the convolution layer are reduced, and the maximum pooling layer is removed. As shown in Table 2, the detailed information of the ResNet-18 model architecture.

[0094] Table 2 ResNet-18 model architecture

[0095]

[0096] ResNet-18 was chosen as the model because of its simple network structure, fewer parameters, and low computational complexity, making it particularly suitable for relatively small data sets. In addition, since ResNet-18 has learned a wealth of deep features during pre-training, fine-tuning it using the time-frequency graph as input to extract deep time-frequency features can effectively accelerate the model training process and improve accuracy. The advantage of this strategy is that it not only makes full use of the image features learned by ResNet-18 in pre-training, but can also further adapt to our data set to capture the correlation between the time domain and the frequency domain, thereby improving the recognition of time-frequency features. In this way, the feature extraction potential of ResNet-18 can be maximized and the model performance can be further optimized. Figure 2 As shown, the architecture of the fine-tuned pre-trained ResNet-18 model.

[0097] S33. Existing studies have shown that power spectral density (PSD) and differential entropy (DE) are widely used in emotion recognition tasks. Differential entropy (DE) is used to measure the uncertainty or information content of continuous random variables; power spectral density (PSD) is used to analyze the frequency components and energy distribution of signals, which provides information about the energy distribution of signals in different frequency ranges. In order to supplement the comprehensive feature information, this paper extracts power spectral density and differential entropy features from EEG data in each time window to capture frequency domain features related to emotions.

[0098] S4, integrating the extracted emotional features and performing multi-feature emotion recognition;

[0099] The sufficiency of features has an important impact on the performance of emotion recognition. In order to fully consider the information in various fields and improve the performance of emotion recognition, the present invention adopts a feature splicing method to combine the time-frequency-space domain features of the collected EEG signals. Specifically, the last dimension of the feature data of each field is spliced ​​to obtain information that integrates multiple feature data. In this way, the features of different fields can be comprehensively utilized to more comprehensively characterize the feature information of EEG signals, further improving the accuracy and performance of emotion recognition. The calculation is as follows:

[0100] F = concatenate(f deep ,f brain ,f de ,f psd ) (5)

[0101] Among them, F is the fusion feature; f deep It is the deep time-frequency features extracted by fine-tuning the pre-trained ResNet-18; f brain It is the spatial domain feature extracted by constructing a dynamic brain functional network; f de is the extracted differential entropy; f psd is the extracted power spectral density.

[0102] S5. Use BILSTM with added channel attention mechanism to further extract the temporal features of EEG signals and perform deep fusion of features;

[0103] Since emotional expression is related to specific brain regions, the channel information collected in the brain regions related to emotional expression should be of higher importance. However, simply splicing this information cannot highlight the more important channel information. The channel attention mechanism can effectively compress the spatial information of multi-channel EEG signals and generate statistical information about the channels by focusing on the importance of different channels.

[0104] In order to achieve the best classification accuracy, the architecture and parameters of the classification model BILSTM were adjusted several times. As shown in Table 3, the detailed information of the model architecture.

[0105] Table 3 BILSTM model structure

[0106]

[0107] The LSTM layer is used to model the input data and extract features. In order to improve the expressiveness and robustness of feature representation, the channel attention layer is used to increase the weight of important features. The weighted data is linearly transformed through the fully connected layer to learn the complex nonlinear relationship in the features. Then the batch normalization layer is used to make each feature dimension have a similar distribution to accelerate training convergence and prevent gradient disappearance or explosion problems. Next, the output of the batch normalization layer uses the ReLU activation function for nonlinear mapping to increase the expressiveness of the model, and randomly discards some neurons with the probability of dropout to prevent the model from overfitting. Finally, the fully connected layer and the ReLU activation function are used to generate the final prediction results.

[0108] Based on the whole model architecture, as shown in Table 4, the detailed information of the hyperparameters is given:

[0109] (1) The number of nodes contained in BILSTM is 128*3; (2) In the Dropout layer, some neurons are dropped with a probability of dropout=0.4; (3) The model is trained using the Adam optimizer with an initial learning rate of 0.001, and then the learning rate is reduced by 50% every 20 rounds of training until the learning rate reaches 1e-6; (4) An L2 regularization term with a weight of 0.0001 is added to prevent fitting; (5) The batch size is set to 32; (6) The number of training rounds is set to 200 rounds.

[0110] Table 4 BILSTM model hyperparameters

[0111]

[0112] Therefore, in the present invention, a channel attention mechanism is introduced to adaptively adjust the weight of each channel to better focus on channels with important information, thereby improving the recognition performance of the model. The expression of the channel attention mechanism is as follows:

[0113] M c (F)=softmax(MLP(AvgPool(F))+MLP(MaxPool(F))) (6)

[0114] Among them, softmax is the activation function used to map the output probability; AvgPool represents average pooling; MaxPool represents maximum pooling; M c (F)∈R C×1 , represents the attention weight; C represents the number of channels.

[0115] The Bidirectional Long Short-Term Memory (BILSTM) network represents an improved recurrent neural network structure. Figure 3 As shown in the figure, the architecture of BILSTM combines two LSTM layers, forward and backward, each of which contains multiple LSTM units. In the forward LSTM, the input sequence is input step by step in chronological order, and the hidden state of each time step is passed to the next time step; in the backward LSTM, the input sequence is input in the opposite chronological order, and the hidden state of each time step is passed forward. Finally, the outputs of the forward and backward LSTM are fused together to form the final output result. Through this bidirectional structure, the network can consider past and future information at the same time, thereby improving the ability to understand each moment in the sequence. The core idea of ​​this architecture is to use a series of structures called "gates" to control the flow of information. As shown in the figure, the outputs of the forward and backward LSTM are fused together to form the final output result. Figure 4 As shown in the figure, the structure of the LSTM unit includes three key gates: forget gate, input gate and output gate. The functions of these three gates are explained in detail below:

[0116] (1) The forget gate is used to control the update of the cell state at the current time. It uses the short-term memory output h of the previous moment t-1 and the sequence data x to be input at the current moment t As input, the probability of long-term memory being forgotten is determined by the activation function f t The calculation process is as shown in Equation 7:

[0117] f (t) =σ(W f ·[h (t-1) ,x (t) ]+b f ) (7)

[0118] Among them, σ is the sigmoid activation function, which is used to map the calculation results to the interval [0,1]; W f is the weight matrix; b f It's a deviation.

[0119] (2) The input gate is used to determine which information in the current input should be added to the cell state. It consists of two parts. The first part is used to calculate the probability of retaining information. The calculation process equation is as follows;

[0120] i t =σ(W i ·[h t-1,x t ]+b i ) (8)

[0121] The second part is used to calculate the output value at the current moment. The calculation process equation is as follows:

[0122]

[0123] Among them, i t represents the probability value of retaining information; represents the output value at the current moment; σ is the sigmoid activation function, which is used to map the calculation result to the interval [0,1]; W is the weight matrix; b is the bias. Then Indicates the information to be updated at the current moment.

[0124] (3) The output gate is used to control the amount of short-term memory information output at the current time. The new information obtained by the input gate and the forget gate is used to update the cell state C at the current moment. t . Cell State C t Together with the output gate, it determines the output h of this layer t The calculation process is as follows:

[0125]

[0126] o t =σ(W o ·[h t-1 ,x t ]+b o ) (11)

[0127] h t =o t *tanh(C t ) (12)

[0128] S6, obtain sentiment classification results;

[0129] EEG signal is a time-varying sequence. In order to further capture the time domain characteristics of the signal to achieve better recognition performance, the present invention uses the fused features as input and utilizes a bidirectional long short-term memory neural network containing 128*3 nodes to obtain the contextual time information of the EEG signal and obtain the final emotion recognition result.

[0130] In order to evaluate the performance of the emotion recognition method proposed in the present invention, the dataset is divided into a training set, a validation set, and a test set according to a specific ratio. The samples in the dataset are randomly divided into a test set at a ratio of 20% to evaluate the generalization ability of the model. For the remaining data, a five-fold cross validation is used to divide the training set and the validation set, which are used for model training and model tuning, respectively. The summary of the DEAP dataset is shown in Table 1

[0131] In order to verify the effectiveness and feasibility of the proposed BRPD-SE-BILSTM EEG emotion recognition method, the method of the present invention is compared with the methods of related studies under the premise of using the same data set. The experimental results of these methods are given in their original texts, as shown in Table 5, which gives the results of the comparative experiment. The emotion recognition method proposed in the present invention fully considers the characteristic information of the three dimensions of time, frequency and space of EEG signals, and achieves the highest recognition accuracy of 97.01% and 93.92% in the dimensions of arousal and valence, which are 1.97% and 0.03% higher than the best recognition method in terms of accuracy, respectively, verifying the effectiveness of the proposed method.

[0132] Table 5 Results of different emotion recognition methods

[0133]

[0134] In order to evaluate the performance of the model used in the present invention, precision, recall and F1 score are used as the evaluation criteria of the model. The evaluation criteria are as follows:

[0135] Precision: Measures the ability of a classification model to predict the true class in the results.

[0136]

[0137] Recall: Measures the model’s ability to correctly capture positive samples.

[0138]

[0139] F1 score: measures the balance of the model on positive and negative samples.

[0140] Among them, TP is true positive, FN is false negative, and FP is false positive. As shown in Table 6, the three performance indicators of the model in terms of precision, recall, and F1 score in the dimensions of arousal and valence. As can be seen from the table, the model of the present invention shows good precision, recall, and F1 score performance indicators in the dimensions of arousal and valence, which indicates that the model has a good ability to perform emotion classification based on EEG data.

[0141] Table 6 Performance index results for arousal and valence dimensions

[0142]

[0143] Although the three evaluation indicators of precision, recall and F1 score can better understand and evaluate the performance of the model in different tasks, their limitation is that they only provide the performance evaluation of the model at a certain point, without providing more detailed category information. Therefore, the confusion matrix is ​​drawn to show the comparison between the prediction results of the classifier and the actual category, so as to provide the detailed performance of the classifier in different categories. Figure 5 Confusion matrix representing the arousal dimension; Figure 6 The confusion matrix represents the valence dimension. The horizontal and vertical axes of the confusion matrix represent the predicted labels and the true labels, respectively, and the values ​​on the main diagonal represent the number of samples predicted correctly. The number of true positives, false positives, true negatives, and false negatives in the confusion matrix clearly shows that the model of the present invention exhibits excellent performance in the arousal and valence dimensions.

[0144] In order to better understand the emotion classification ability of each subject, evaluate the stability and consistency of the model on different subjects, and thus improve the generalization ability of the model, an emotion recognition experiment was conducted on each subject separately, and the recognition accuracy of all subjects in the arousal and valence dimensions was plotted into a bar graph, as shown in Figure 2. Figure 7 As shown in the figure, it can be seen that subjects 7, 8 and 15 achieved the highest recognition accuracy of 100% in the valence dimension and arousal dimension respectively. In addition, the recognition accuracy of subjects 6, 9, 17 and 21 in the valence dimension was 88.65%, 89.15%, 88.54% and 88.29% respectively, and the recognition accuracy of subject 17 in the arousal dimension was 89.71%. The recognition accuracy of the remaining subjects in the arousal and valence dimensions was higher than 90%, which fully demonstrated the generalization ability and stability of our model.

[0145] Example 2

[0146] In order to solve the problem of insufficient feature extraction, this paper proposes a method of automatically extracting features using a deep learning model and manually extracting features, and has achieved remarkable results. In order to verify the important influence of feature sufficiency on the effect of emotion recognition, a control experiment was conducted. Tables 7 and 8 show the experimental results on the feature problem, comparing the classification accuracy of the commonly used features for emotion recognition based on EEG (such as DE and PSD) and the fused deep time-frequency features (DEEP) and spatial features (Brain) proposed in this invention. It can be seen from the table that the accuracy of any combined feature is higher than the accuracy of emotion recognition using only the spatial features extracted from the brain functional network and the deep time-frequency features extracted from the deep neural network model. The average accuracy of the fused deep time-frequency feature DEEP and the spatial feature Brain in the arousal and valence dimensions is 86.94% and 88.41%, respectively. At the same time, the deep time-frequency feature DEEP, the spatial feature Brain, the manually extracted frequency domain features PSD and DE in the combined features show the best classification results in the arousal and valence dimensions, with average accuracy rates of 97.01% and 93.92%, respectively. These classification results fully demonstrate that the adequacy of feature extraction has a serious impact on the accuracy of sentiment classification.

[0147] Table 7 Results of combining brain functional network features and deep features with manually extracted frequency domain features in the arousal dimension

[0148]

[0149] Table 8 Results of combining brain functional network features and deep features with manually extracted frequency domain features in the valence dimension

[0150]

[0151] In order to further discuss the impact of different model architectures in the feature extraction stage and the classification stage on the overall research, the present invention constructs a set of comparative models, as follows:

[0152] Model 1: The pre-trained ResNet-18 model in the deep feature extraction process was adjusted. The convolution kernel, stride, and padding of the first convolution layer were adjusted from 3×3, 1, and 1 to 7×7, 2, and 3, respectively, and a maximum pooling layer was added. The rest of the model architecture remained unchanged. The purpose of this was to compare the impact of information loss on the experimental results.

[0153] Model 2: Remove the channel attention mechanism in the BILSTM model. The purpose of this is to compare the impact of the attention mechanism on the experimental results in the classification stage.

[0154] Considering the impact of different numbers of LSTM network layers on the experimental results, two models were constructed to compare the differences between using 1-layer and 2-layer LSTM networks and the 3-layer LSTM network used in the present invention.

[0155] Model 3: Contains a 2-layer LSTM network and a channel attention module.

[0156] Model 4: Contains a 1-layer LSTM network and a channel attention module.

[0157] The above is a model combination constructed by the present invention for comparison. The performance of the model of the present invention is compared with the four constructed comparative models in terms of arousal and valence dimensions. Figure 8 and Fig. 9 As shown, the recognition accuracy of the five models in the dimensions of arousal and valence are shown respectively. It can be seen that our model shows the highest recognition accuracy in both the dimensions of arousal and valence. Compared with model 1, the accuracy of our model in the dimensions of arousal and valence is improved by 5.15% and 4.7%, respectively. This is mainly because model 1 uses a larger convolution kernel and maximum pooling operation, which may lead to information loss and thus reduce the accuracy of emotion recognition. In the emotion recognition task, EEG information of different EEG channels is collected, and by constructing a brain network, it is found that different EEG channels have different degrees of influence on emotions. Therefore, channel attention is used to assign different weights to EEG channels. Compared with model 2, the accuracy of the model of the present invention in the dimensions of arousal and valence is improved by 3.43% and 2.87%, respectively. This shows that the attention mechanism has an important influence on the performance of emotion recognition. In the comparison with models 3 and 4, the main focus is on the number of LSTM network layers, and it is found that with the increase of the number of network layers, the performance of emotion recognition shows a gradual increase. In the arousal dimension, our model is 0.03% and 6.49% higher than Model 3 and Model 4, respectively; in the valence dimension, the model of the present invention is 1.47% and 2.36% higher than Model 3 and Model 4, respectively. The experimental results show that different LSTM network layers consider the contextual time information to different degrees. The model of the present invention fully considers the contextual time information of the EEG signal and achieves the best emotion recognition performance in the arousal and valence dimensions.

[0158] Therefore, the present invention adopts the above-mentioned emotion recognition method based on the fusion of EEG time, space and frequency features to solve the problem of insufficient feature extraction of EEG signals in emotion recognition tasks, designs a model that reduces information loss and improves accuracy and robustness under small sample data, fully considers the feature information of the three dimensions of time, frequency and space of EEG signals, achieves the highest recognition accuracy in the dimensions of arousal and valence, and verifies the effectiveness of this method.

[0159] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.

Claims

1. An emotion recognition method based on EEG spatiotemporal-frequency feature fusion, characterized by: include: S1, a series of preprocessing steps were performed on the raw EEG signals, including baseline correction, normalization, and adding time windows; S2, converting the EEG signal processed by S1 to obtain spatial information and time-frequency information of the EEG signal; S3, perform multi-faceted feature extraction; S4, integrating the extracted emotional features and performing multi-feature emotion recognition; S5. Use BILSTM with added channel attention mechanism to further extract the temporal features of EEG signals and perform deep fusion of features; S6. Obtain sentiment classification results.

2. The emotion recognition method based on EEG spatiotemporal-frequency feature fusion according to claim 1, characterized in that: In step S2, the signal form conversion is performed: S21. Based on the PLI theory, a dynamic brain function network is constructed. The spatial characteristics of the signal are captured by extracting the topological characteristics of the brain network. The sliding window method is used to process the EEG signal in segments. The phase synchronization between nodes is calculated using PLI to generate a relationship matrix reflecting the node connection strength. The original expression is: Where, E<·> represents the expected operation on the data in the time window; N represents the time point; Represents two signals at time t n The phase difference at ; sign is a sign function; the value of PLI is between 0 and 1. If PLI is closer to 1, it means the strength of the connection is stronger, otherwise, it means the strength of the connection is weaker; The relationship matrix constructed based on PLI theory needs further threshold processing. The adaptive threshold method is used to perform threshold processing on the relationship matrix of each time window. According to the characteristics of the data, the adaptive threshold is expressed as: threshold=mean_pli-std_pli / 5 (2) Among them, mean_pli represents the mean of the relationship matrix, and std_pli represents the variance of the relationship matrix; S22. Use the short-time Fourier transform of the Hamming window function to convert the EEG signal into a time-frequency diagram, which is expressed as: S(n,ω)=|X(n,ω)| 2 (3) Among them, S(n,ω) represents the time-frequency diagram; X(n,ω) is the short-time Fourier transform, which is a two-dimensional function that relates time and frequency; x(m) is the input signal; w(nm) is the window function, which uses a Hanning window with a window length of 1s and a superposition of 0.75s.

3. The emotion recognition method based on EEG spatiotemporal-frequency feature fusion according to claim 1, characterized in that: In step S3, feature extraction is performed: S31. Based on the construction of the dynamic brain functional network, eight graph theory features of the brain functional network were extracted, including clustering coefficient, shortest path length, global efficiency, local efficiency, betweenness centrality, compactness, degree centrality, and eigenvector centrality. S32, based on the ResNet-18 model, fine-tune it with the time-frequency graph as input to extract deep time-frequency features; S33. Extract power spectrum density and differential entropy features of the EEG data in each time window.

4. The emotion recognition method based on EEG spatiotemporal-frequency feature fusion according to claim 1, characterized in that: In step S4, the time-frequency-spatial domain features of the collected EEG signals are combined by the feature splicing method, and the last dimension of the feature data of each field is spliced ​​to obtain information of fused multi-feature data. The feature information of the EEG signals is characterized by using the features of different fields, and the calculation is as follows: F=concatenate(f deep ,f brain ,f de ,f psd ) (5) Among them, F is the fusion feature; f deep It is the deep time-frequency features extracted by fine-tuning the pre-trained ResNet-18; f brain It is the spatial domain feature extracted by constructing a dynamic brain functional network; f de is the extracted differential entropy; f psd is the extracted power spectral density.

5. The emotion recognition method based on EEG spatiotemporal-frequency feature fusion according to claim 1, characterized in that: In step S5, a bidirectional long short-term memory network structure with a channel attention mechanism is used to further extract the temporal features of the EEG signal and perform deep fusion of the features. The expression of the channel attention mechanism is as follows: M c (F)=softmax(MLP(AvgPool(F))+MLP(MaxPool(F))) (6) Among them, softmax is the activation function used to map the output probability; AvgPool represents average pooling; MaxPool represents maximum pooling; M c (F)∈R C×1 , represents the attention weight; C represents the number of channels.

6. The emotion recognition method based on EEG spatiotemporal-frequency feature fusion according to claim 4, characterized in that: The flow of information is controlled through a bidirectional long short-term memory network structure, where: (1) The forget gate is used to control the update of the cell state at the current time; the short-term memory output h at the previous moment t-1 and the sequence data x to be input at the current moment t As input, the probability of long-term memory being forgotten is determined by the activation function f t , the calculation process equation is as follows: f (t) =σ(W f ·[h (t-1) ,x (t) ]+b f ) (7) Among them, σ is the sigmoid activation function, which is used to map the calculation results to the interval [0,1]; W f is the weight matrix; b f It is a deviation; (2) The input gate is used to determine which information in the current input should be added to the cell state; it consists of two parts. The first part is used to calculate the probability of retaining information. The calculation process equation is as follows; i t =σ(W i ·[h t-1 ,x t ]+b i ) (8) The second part is used to calculate the output value at the current moment. The calculation process equation is as follows: Among them, i t represents the probability value of retaining information; represents the output value at the current moment; σ is the sigmoid activation function, which is used to map the calculation result to the interval [0,1]; W is the weight matrix; b is the bias; then Indicates the information to be updated at the current moment; (3) The output gate is used to control the amount of information in the short-term memory output at the current time; the new information obtained by the input gate and the forget gate is used to update the cell state C at the current time t ; Cell state C t Together with the output gate, it determines the output h of this layer t , the calculation process is as follows: the t =σ(W o ·[h t-1 ,x t ]+b o ) (11) h t =o t *tanh(C t )。 (12)。

Images