A method of emotion recognition based on EEG signals

Abstract

This invention discloses an emotion recognition method based on electroencephalogram (EEG) signals. After preprocessing the EEG signals, connectivity indices are used as adjacency matrices to extract graph features using a graph convolutional neural network (GCNN). Simultaneously, a temporal convolutional neural network (TCNN) is used to extract temporal features from the EEG signals. These two methods are then effectively combined for emotion classification. This method combines graph and temporal convolutional neural networks. On one hand, the EEG signals are constructed as a graph model, and functional connectivity indices are used as the adjacency matrix of the graph model for graph convolutional feature extraction. On the other hand, a temporal convolutional network is used to extract temporal features. Combining the spatial and temporal features extracted by these two parallel networks improves the model's ability to recognize different emotions.

Description

Technical Field

[0001] The present invention belongs to the field of electroencephalogram (EEG) signal recognition, and in particular relates to an emotion recognition method based on EEG signals. Background Art

[0002] Emotion recognition methods can be divided into methods based on non-physiological signals (facial expressions, speech, and body movements) and methods based on physiological signals (galvanic skin response (GSR), respiratory rate (RR), electrocardiogram (ECG), and electroencephalogram (EEG)). Non-physiological signals are easily affected and controlled by people's subjective consciousness. In contrast, physiological signals can capture people's true emotional state more objectively and reliably. Unlike other physiological signals, electroencephalogram (EEG) is captured from the cerebral cortex, and various psychophysiological studies have proven that it is related to human emotions. In addition, EEG devices have the advantages of high temporal resolution, low cost, and portable equipment compared to other brain physiological signal acquisition devices. Therefore, emotion recognition based on EEG has received more and more attention in recent years.

[0003] Deep learning methods are increasingly being used in EEG signals to identify people's emotional states. Most studies use deep learning to extract suitable features from raw EEG signals for emotion classification. However, EEG signals are often susceptible to external factors, acquisition equipment, and the subject's physiological activities, so EEG data contain a lot of noise. Therefore, it is beneficial to first extract meaningful information from EEG signals and then learn through deep learning models. Although instantaneous changes in amplitude or latency and instantaneous changes in spectral power have been used to characterize EEG signals, some brain connectivity-related features have recently emerged because they can consider the relationship between different brain regions. Brain connectivity has been an active research area in neuroscience. In addition, recent studies have confirmed the effectiveness of brain connectivity features in identifying emotional states.

[0004] In recent years, graph signal processing methods have been proposed to effectively process signals with irregular structures. It extends traditional digital signal processing techniques to signals that are not sampled on regular domains (such as time and grid space), but are located on graphs composed of vertices and edges. In addition, deep learning of graph signals has been studied, and graph convolutional neural network structures have been proposed. Brain signals are a good example of graph signals because graphs are suitable for representing physical or functional connections between different brain regions. However, there are few studies on applying graph signal processing techniques and deep learning methods based on graph signals to modeling brain signals, especially electroencephalograms (EEGs).

[0005] There are many different EEG-based feature extraction methods, but most of them have their own advantages and disadvantages, and the mathematical theory of the model is not fully explained. Although the model has high recognition accuracy, it has poor interpretability. Summary of the invention

[0006] In order to address the shortcomings of the prior art, the present invention aims to provide an emotion recognition method based on EEG signals. The method combines a graph convolutional neural network and a temporal convolutional neural network. On the one hand, the EEG signals are established as a graph model, and the functional connectivity features are used as the adjacency matrix of the graph model to extract graph convolution features; on the other hand, the temporal convolutional network is used to extract temporal features, and the features extracted by the two parallel networks are combined for emotion recognition.

[0007] In order to achieve the above object, the technical solution adopted by the present invention is:

[0008] An emotion recognition method based on EEG signals comprises the following steps: 1) EEG signal preprocessing:

[0009] 11) In the DEAP dataset, the first 32 channels are selected as EEG signal channels among the 40 physiological signal channels collected according to the international 10-20 system;

[0010] 12) performing data processing on the EEG signal;

[0011] 2) The three connectivity indicators are used as the adjacency matrix of graph convolution, the preprocessed data samples are used as input samples, and the graph convolutional neural network is used to extract graph features;

[0012] 3) The preprocessed EEG data is used as sample input, and the temporal features contained in the EEG signal are extracted using a temporal convolutional neural network;

[0013] 4) Combine graph features and time features for sentiment classification.

[0014] Among them, step 12) specifically includes: dividing the EEG signal into 1 second segments, obtaining 2400 data samples for training the neural network; and using the first 3 seconds of silent data to represent the basic emotional state; in order to remove the influence of the basic emotional state, calculating the average value of each test sample minus the first 3 1s of data, and finally obtaining the final preprocessed sample X∈R by Z-score standardization. 32×128 .

[0015] In step 12), the baseline signals contained in the DEAP dataset are used as reference signals, which are recorded when the subjects were not stimulated by the music video.

[0016] Step 2) specifically includes:

[0017] Assume that the graph data has N nodes, each node has its own features, and the features of these nodes form an N×D-dimensional matrix X. The relationship between each node will form an N×N-dimensional matrix A as the adjacency matrix; then X and A are the inputs of the model;

[0018] A graph convolutional neural network is a neural network layer, and the propagation between its layers is:

[0019]

[0020] in I N is the identity matrix, yes The degree matrix, H (l) ∈R N×D is the feature of the lth layer, W (l) is the parameter of the lth layer of the graph convolutional neural network; for the input layer, H (0) =X, σ1 is a nonlinear activation function;

[0021] The three connectivity indices, namely Pearson correlation coefficient, phase-locking value and phase delay index, are used as the adjacency matrix of graph convolution, and the three connectivity indices are calculated respectively;

[0022] The Pearson correlation coefficient calculation includes:

[0023] The Pearson correlation coefficient measures the linear relationship between two signals as a continuous number from -1 to 1. Pearson correlation coefficient values ​​of -1 and 1 indicate a negative linear relationship and a positive linear relationship, respectively. A Pearson correlation coefficient of 0 indicates that the two signals are uncorrelated.

[0024] use represents the EEG data of the i-th electrode; where i = 1, 2, ..., 32, T is the signal length; the absolute value of the Pearson correlation coefficient is used as the adjacency matrix, the EEG signal X of the i-th channel and the k-th channel i and X k The Pearson correlation coefficient between them is expressed as:

[0025]

[0026] Wherein, μ and σ are the mean and standard deviation of the EEG signals of the corresponding channels, respectively; t represents the time;

[0027] The phase lock value calculation includes:

[0028] The phase lock value describes the phase synchronization between two signals. The phase lock value ranges from 0 to 1. A phase lock value of 0 means that the two signals are completely independent, and a phase lock value of 1 means that the two signals are completely synchronized. The calculation method is to average the absolute phase difference, as shown below:

[0029]

[0030] in, is the phase of the signal at time t, j is the imaginary unit;

[0031] The phase delay index calculation includes:

[0032] The phase delay index is used to measure the phase synchronization degree of the two channel signals. The phase delay index ranges from 0 to 1. The larger the value, the stronger the phase synchronization degree between the two signals. The phase delay index is not sensitive to the volume conduction effect, but it seems to be more sensitive to noise. It is specifically expressed as:

[0033]

[0034] Among them, sign is the sign function;

[0035] The measurement matrices of the above three connectivity indicators are used as the adjacency matrix of graph convolution, and the preprocessed samples X∈R 32×128 As the input sample, the features are extracted through graph convolution; it can be expressed by the formula:

[0036] f G =E G (X,A)

[0037] where f G Represents graph features, E G (·) represents the graph feature extraction function, A is the adjacency matrix;

[0038] Input the fully connected layer to further extract features, expressed as:

[0039] f G =FC(f G )∈R 256×1

[0040] Step 3) specifically includes:

[0041] The preprocessed EEG data X∈R 32×128 As sample input, the temporal features of EEG data are extracted through a temporal convolutional neural network containing three residual blocks, and then the temporal features are obtained from the input data. The process of the entire temporal feature extractor is described as:

[0042] f T =E T (X)∈R 64×32

[0043] where f T Represents the time feature, E T (g) represents the time feature extraction function;

[0044] Input the fully connected layer to further extract the temporal features, expressed as:

[0045] f T =FC(f T )∈R 256×1 .

[0046] Step 4) The specific steps are as follows:

[0047] The graph features and time features are connected as the final feature vector, and the final feature vector is sent to a fully connected layer containing 20 neurons. The process is described as:

[0048] o=FC(f G ||f T )∈R 1×20

[0049] where || is the value of f G and f T Join operations that connect together;

[0050] Then perform a linear transformation:

[0051] y=Wo T +b=[y1,y2] T ∈R 2×1

[0052] Where W and b are the weight matrix and bias respectively, and 2 represents the number of categories for emotion classification;

[0053] Finally, the elements of y are input into the softmax function for emotion recognition, which can be described as:

[0054]

[0055] Among them, P(c|X) is the probability that X belongs to the cth class, Indicates the prediction result.

[0056] The present invention has the following beneficial effects:

[0057] The present invention proposes an emotion recognition method based on EEG signals, which effectively extracts features from EEG data and improves the performance of emotion classification. Finally, the results of using only graph convolutional neural network classification or only temporal convolutional neural network classification are compared to measure the effectiveness of the proposed model. The present invention combines graph convolutional neural network and temporal convolutional neural network. On the one hand, EEG signals are established as a graph model, and the connectivity index is used as the adjacency matrix of the graph model to extract graph convolution features; on the other hand, the temporal convolutional network is used to extract temporal features. The features extracted by the two parallel networks are combined for emotion classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 The box plots of the accuracy of the experiments on potency using three adjacency matrices of the present invention are shown;

[0059] Figure 2 The box plots of the accuracy of the experiments on arousal using three adjacency matrices of the present invention are shown;

[0060] Figure 3 It is a bar chart comparing the average accuracy under different adjacency matrices of the present invention;

[0061] Figure 4 This is a comparison chart of the average accuracy of the three models of the present invention on potency;

[0062] Figure 5 This is a comparison chart of the average accuracy of the three models of the present invention on the arousal level;

[0063] Figure 6 This is a model framework diagram of the present invention;

[0064] Figure 7 This is a framework diagram of a temporal convolutional network composed of two residual blocks of the present invention;

[0065] Figure 8 FIG. 4 is a specific structural diagram of the residual block of the present invention. DETAILED DESCRIPTION

[0066] The present invention proposes an emotion recognition method based on electroencephalogram signals, comprising the following steps:

[0067] Step 1) EEG signal preprocessing.

[0068] The data set used in the present invention is the DEAP data set, which is collected by Koelstra et al. through experiments. It is multi-channel data used to study human emotional states and is publicly available for free. The database is based on physiological signals generated by music video materials under induced stimulation. It records the physiological signals of 32 subjects watching 40 one-minute music videos and the psychological scales of the subjects' valence, arousal, dominance, and liking of the videos. The official provides pre-processed 128Hz complex sampling data. The physiological signal matrix of each subject is 40*40*8064 (40 experimental music, 40 physiological signal channels, 8064 sampling points). Among the 40 physiological signal channels collected, the first 32 channels collect EEG signals. The EEG channels select the positions of 32 channels according to the international 10-20 system.

[0069] Data processing of EEG signals:

[0070] The DEAP dataset contains baseline signals, which are recorded when the subjects were not stimulated by the music video. These baseline signals can be used as reference signals to remove the basic emotional state without any stimulation, which can improve the accuracy of emotion recognition.

[0071] The EEG signal corresponding to each music clip is divided into 1 second segments, and finally 2400 data samples are obtained for training the neural network. The first 3 seconds of silent data are used to represent the basic emotional state. In order to remove the influence of the basic emotional state, the average value of each test sample minus the first 3 1s of data is calculated. Finally, the final preprocessed sample X∈R is obtained by Z-score standardization. 32×128 .

[0072] Step 2) The three connectivity indicators are used as the adjacency matrix of the graph convolution, the preprocessed data samples are used as input samples, and the graph convolutional neural network is used to extract features.

[0073] The structure of a graph is generally very irregular and can be considered as an infinite-dimensional data, so it has no translation invariance. The surrounding structure of each node may be unique, and this kind of structured data instantly makes traditional CNN and RNN ineffective, so the graph convolutional neural network (GCN) appeared. The graph convolutional neural network (GCN) actually has the same function as CNN, which is a feature extractor, except that its object is graph data. GCN has cleverly designed a method to extract features from graph data, so that these features can be used to perform node classification, graph classification, edge prediction, and obtain graph embedding representation.

[0074] Assume that the graph data has N nodes, each node has its own features, and the features of these nodes form an N×D-dimensional matrix X. The relationship between each node will form an N×N-dimensional matrix A as the adjacency matrix; then X and A are the inputs of the model;

[0075] GCN is a neural network layer, and the propagation between its layers is:

[0076]

[0077] in I N is the identity matrix, yes The degree matrix, H (l) ∈R N×D is the feature of the lth layer, W (l) is the parameter of the lth layer of the graph convolutional neural network; for the input layer, H (0) =X, σ1 is a nonlinear activation function.

[0078] The specific architecture of graph convolution:

[0079] Three connectivity indicators are used as the adjacency matrix of graph convolution, namely Pearson correlation coefficient (PCC), phase locking value (PLV) and phase delay index (PLI).

[0080] Pearson correlation coefficient (PCC):

[0081] PCC measures the linear relationship between two signals as a continuous number ranging from -1 to 1. PCC values ​​of -1 and 1 represent a perfect negative and positive linear relationship, respectively, and a PCC value of 0 indicates that the two signals are uncorrelated. represents the EEG data of the i-th electrode (i=1,2,…,32), where T is the signal length; the absolute value of PCC is used as the adjacency matrix of GCN, and the EEG signal X of the i-th channel and the k-th channel i and X k The PCC value between is expressed as:

[0082]

[0083] Among them, μ and σ are the mean and standard deviation of the corresponding channel of the EEG signal respectively; t represents the time.

[0084] Phase lock value (PLV):

[0085] PLV describes the phase synchronization between two signals. The value range of PLV is 0 to 1. A PLV value of 0 means that the two signals are completely independent, and a PLV value of 1 means that the two signals are completely synchronized. The calculation method is to average the absolute phase difference, as shown below:

[0086]

[0087] in, is the phase of the signal at time t, j is the imaginary unit;

[0088] Phase Lag Index (PLI):

[0089] The phase delay index (PLI) is used to measure the phase synchronization of two channel signals. The value range of PLI is 0 to 1. The larger the value, the stronger the phase synchronization between the two signals. The main advantage of PLI is that it is insensitive to volume conduction effect, but it seems to be more sensitive to noise, which can be expressed as:

[0090]

[0091] Among them, sign is the sign function, is the phase of the signal at time t;

[0092] The measurement matrices of these three connectivity indicators are used as the adjacency matrix of graph convolution respectively, and the preprocessed samples X∈R 32×128 The input sample is subjected to graph convolution to extract features. The formula is:

[0093] fG =E G (X,A)

[0094] where f G Represents graph features, E G (·) represents the graph feature extraction function, A is the adjacency matrix;

[0095] Input the fully connected layer to further extract features, expressed as:

[0096] f G =FC(f G )∈R 256×1

[0097] Step 3) The preprocessed EEG data is used as sample input, and the temporal features contained in the EEG signal are extracted using a temporal convolutional neural network.

[0098] Temporal Convolutional Neural Network (TCN), TCN = 1D FCN + causal convolution. TCN uses the structure of 1-D FCN. The time length of the input and output of each hidden layer is the same, maintaining the same time step. For the first time step, there is no historical information. TCN believes that its historical data is all 0 (in fact, it is the padding of the convolution operation). At the same time, TCN has a stronger ability to retain long-term historical information than LSTM. In order to solve the second problem, TCN uses causal convolution, that is, for the data output at time t, its input can only be at time t and before t. In order to effectively deal with the problem of long historical information, the expansion causal convolution is used. It still has causality, but it introduces the expansion factor d. At the same time, even if the expansion causal convolution is used, sometimes the model may still be very deep. The deeper network structure may cause problems such as gradient disappearance. In order to deal with this situation, a structure similar to the residual block in ResNet is used. The TCN structure designed in this way has more generalization ability. It can be seen that the residual structure replaces the simple connection between TCN layers. Since the number of channels between x and F(x) may be different, a 1×1 Conv is designed to perform a simple transformation on x so that the transformed x and F(x) can be added.

[0099] The preprocessed EEG data X∈R 32×128 As sample input, the temporal features of EEG data are extracted through TCN containing three residual blocks. After this process, the temporal features can be obtained from the input data. The process of the entire temporal feature extractor can be described as:

[0100] f T =E T (X)∈R 64×32

[0101] where fT Represents the time feature, E T (g) represents the time feature extraction function;

[0102] Input the fully connected layer to further extract the temporal features, expressed as:

[0103] f T =FC(f T )∈R 256×1

[0104] Step 4) Combine functional connectivity features and temporal features for emotion classification.

[0105] The functional connectivity features and the temporal features are concatenated as the final feature vector, and the final feature vector is fed into a fully connected layer containing 20 neurons. The process is described as:

[0106] o=FC(f G ||f T )∈R 1×20

[0107] where || is the value of f G and f T Join operations that connect together;

[0108] Then perform a linear transformation:

[0109] y=Wo T +b=[y1,y2] T ∈R 2×1

[0110] Where W and b are the weight matrix and bias respectively, and 2 represents the number of categories for emotion classification.

[0111] Finally, the elements of y are input into the softmax function for emotion recognition, which can be described as:

[0112]

[0113] Among them, P(c|X) is the probability that X belongs to the cth class, Indicates the prediction result.

[0114] Simulation experiment

[0115] As shown in Table 1, the method proposed in the present invention is compared with the traditional method, and the experimental results also show the advantages of the present invention.

[0116]

[0117] Table 1: Comparison of the results of this method with other existing methods on the DEAP dataset (average precision / bias)

[0118] Table 1 shows the comparison results of the above methods on the DEAP dataset. It can be seen that the present invention can achieve relatively advanced results. Deep Canonical Correlation Analysis (DCCA) extracts DE features from four frequency bands (theta, alpha, beta and gamma) of the original EEG signal, and calculates its time domain statistical features: maximum, minimum, mean, standard deviation, variance, sum of squares. For the peripheral physiological signals of 8 channels, 48 ​​(6×8) dimensional features are extracted. Then, each modality is transformed by multi-layer nonlinear transformation, and the specified canonical correlation analysis constraints are used to coordinate it into the hyperspace to obtain higher mutual information. Then, the transformed features are fused by weighted sum fusion method, and the fused features are used to train the SVM classifier. The present invention is about 12% higher than the deep canonical correlation analysis (DCCA) method. Multi-column convolutional neural network (MC-CNN) proposes a multi-column EEG signal emotion recognition model based on CNN, and its accuracy is also improved by about 7% compared with MC-CNN.

[0119] The multimodal residual LSTM (MMResLSTM) constructs 4 LSTM layers for EEG signals and peripheral physiological signals respectively. By sharing the weights of each modality on each LSTM layer, the temporal correlation between different modalities is learned. Compared with MMResLSTM and spatial symmetric feature extraction network (SFE-Net), the present invention improves by about 5%. The attention-based convolutional recurrent neural network (ACRNN) first adopts the channel attention mechanism to adaptively allocate the weights of different channels, and uses the convolutional neural network to extract the spatial information of the encoded EEG signal, and then integrates the extended self-attention into the RNN to encode the importance based on the intrinsic similarity of the EEG signal. Compared with ACRNN, its accuracy is also improved by about 4%.

[0120] Parallel Sequential Channel Projection Network (PSCP-Net) This network consists of a time stream subnetwork (TS), a space stream subnetwork (SS) and a fusion classification block. Among them, the TS and SS subnetworks constitute a parallel spatiotemporal network, which extracts temporal and spatial features of EEG signals through the sequence projection layer and the channel projection layer respectively. The fusion classification block combines the extracted spatiotemporal features into a joint spatiotemporal feature vector, and then sends the vector to the fully connected layer for emotion classification. The present invention is improved by about 1% compared with the PSCP-Net method. The local asymmetric convolutional neural network (RACNN) can learn both the regional information between adjacent channels and the asymmetric differences between the two hemispheres. Both RACNN and the present invention are end-to-end and deep learning models, but the model of the present invention obtains better recognition results because it can obtain more comprehensive representative features.

[0121] In addition, compared with multimodal methods such as DCCA and MMResLSTM, the recognition rate of the method proposed in the present invention is improved by more than 4%, which shows that the present invention has achieved better performance by using only single-modal data. Unlike traditional methods, the model proposed in the present invention only uses EEG data, which can extract functional connectivity and temporal features, providing key information for emotion recognition. Of course, the present invention ignores the additional information provided by other modalities.

[0122] In addition, the present invention uses three adjacency matrices, namely PCC, PLV, and PLI, and conducts three experiments to compare the differences between the three adjacency matrices.

[0123] The accuracy box plots of the three experiments are shown in Figure 1 and 2 As shown. Figure 1 It can be seen that at the valence level, the accuracy distribution of the three experimental results is not much different, and the experimental results using PCC as the adjacency matrix are better than the other two. Figure 2 It can be seen that at the level of arousal, the accuracy distribution of the three experimental results is not much different, and the experimental results using PLV as the adjacency matrix are better than the other two.

[0124] The experimental results of all subjects were averaged to obtain the average accuracy on the DEAP dataset. Figure 3 The comparison bar graphs of the experimental results under the two emotional states of valence and arousal are given in Figure 2. Figure 3 It can be seen that in terms of valence, the method using PCC as the adjacency matrix has a better effect, while the method using PLV as the adjacency matrix has a worse effect. In terms of arousal, the results are just the opposite. The method using PLV as the adjacency matrix has a better effect, while the method using PCC as the adjacency matrix has a worse effect.

[0125] In addition, in order to better illustrate the effectiveness of this method, the present invention also conducted experiments using only graph convolution or only temporal convolution, and the experiments using only graph convolution still used three adjacency matrices. Figure 4 and 5 A bar chart showing the average accuracy comparison of emotion classification under the three models under valence and arousal.

[0126] By the attached Figure 4 It can be seen intuitively that the effect of using only TCN for emotion classification is poor, the effect of GCN is second, and the model combining TCN and GCN has better performance, which illustrates the effectiveness of the method proposed in this invention.

[0127] Depend on Figure 5It can be seen that the average accuracy of emotion classification using only TCN or only GCN is similar, but the performance of the model combining the two is improved, which shows that the model of the present invention has a better classification effect.

[0128] Figure 6 The model framework proposed by the present invention is demonstrated. Figure 7 A temporal convolutional network consisting of two residual blocks is given, where the convolution kernels k=2 of the two residual blocks, the dilation rate d=1 of the first residual block, and the dilation rate d=2 of the second residual block. Figure 8 is the specific structure of the residual block.

[0129] It can be seen that the present invention combines the graph convolutional neural network and the temporal convolutional neural network. On the one hand, the EEG signal is established as a graph model, and the functional connectivity index is used as the adjacency matrix of the graph model to extract graph convolution features; on the other hand, the temporal convolutional network is used to extract temporal features. The features extracted by the two parallel networks are combined for emotion classification, so that the emotion classification has better classification performance.

Claims

1. An emotion recognition method based on EEG signals, characterized in that: The following steps are involved: 1) EEG signal preprocessing: 11) In the DEAP dataset, the first 32 channels are selected as EEG signal channels among the 40 physiological signal channels collected according to the international 10-20 system; 12) performing data processing on the EEG signal; 2) The three connectivity indicators are used as the adjacency matrix of graph convolution, the preprocessed data samples are used as input samples, and the graph convolutional neural network is used to extract graph features; 3) The preprocessed EEG data is used as sample input, and the temporal features contained in the EEG signal are extracted using a temporal convolutional neural network; 4) Combine graph features and time features for emotion classification; Wherein, step 2) specifically includes: Assume that the graph data has N nodes, each node has its own features, and the features of these nodes form an N×D-dimensional matrix X. The relationship between each node will form an N×N-dimensional matrix A as the adjacency matrix; then X and A are the inputs of the model; A graph convolutional neural network is a neural network layer, and the propagation between its layers is: in I N is the identity matrix, yes The degree matrix, H (l) ∈R N×D is the feature of the lth layer, W (l) is the parameter of the lth layer of the graph convolutional neural network; for the input layer, H (0) =X, σ1 is a nonlinear activation function; The three connectivity indices, namely Pearson correlation coefficient, phase-locking value and phase delay index, are used as the adjacency matrix of graph convolution, and the three connectivity indices are calculated respectively; The Pearson correlation coefficient calculation includes: The Pearson correlation coefficient measures the linear relationship between two signals as a continuous number from -1 to 1. Pearson correlation coefficient values ​​of -1 and 1 indicate a negative linear relationship and a positive linear relationship, respectively. A Pearson correlation coefficient of 0 indicates that the two signals are uncorrelated. use represents the EEG data of the i-th electrode; where i = 1, 2, ..., 32, T is the signal length; the absolute value of the Pearson correlation coefficient is used as the adjacency matrix, the EEG signal X of the i-th channel and the k-th channel i and X k The Pearson correlation coefficient between them is expressed as: Wherein, μ and σ are the mean and standard deviation of the EEG signals of the corresponding channels, respectively; t represents the time; The phase lock value calculation includes: The phase lock value describes the phase synchronization between two signals. The phase lock value ranges from 0 to 1. A phase lock value of 0 means that the two signals are completely independent, and a phase lock value of 1 means that the two signals are completely synchronized. The calculation method is to average the absolute phase difference, as shown below: in, is the phase of the signal at time t, j is the imaginary unit; The phase delay index calculation includes: The phase delay index is used to measure the phase synchronization degree of the two channel signals. The phase delay index ranges from 0 to 1. The larger the value, the stronger the phase synchronization degree between the two signals. The phase delay index is not sensitive to the volume conduction effect, but it seems to be more sensitive to noise. It is specifically expressed as: Among them, sign is the sign function; The measurement matrices of the above three connectivity indicators are used as the adjacency matrix of graph convolution, and the preprocessed samples X∈R 32×128 As the input sample, the features are extracted through graph convolution; it can be expressed by the formula: f G =E G (X,A) where f G Represents graph features, E G (·) represents the graph feature extraction function, A is the adjacency matrix; Input the fully connected layer to further extract features, expressed as: f G =FC(f G )∈R 256×1 ; Step 3) specifically includes: The preprocessed EEG data X∈R 32×128 As sample input, the temporal features of EEG data are extracted through a temporal convolutional neural network containing three residual blocks, and then the temporal features are obtained from the input data. The process of the entire temporal feature extractor is described as: f T =E T (X)∈R 64×32 where f T Represents the time feature, E T (g) represents the time feature extraction function; Input the fully connected layer to further extract the temporal features, expressed as: f T =FC(f T )∈R 256×1 ; Step 4) The specific steps are as follows: The graph features and time features are connected as the final feature vector, and the final feature vector is sent to a fully connected layer containing 20 neurons. The process is described as: o=FC(f G ||f T )∈R 1×20 where || is the value of f G and f T Join operations that connect together; Then perform a linear transformation: y=Wo T +b=[y1,y2] T ∈R 2×1 Where W and b are the weight matrix and bias respectively, and 2 represents the number of categories for emotion classification; Finally, the elements of y are input into the softmax function for emotion recognition, which is described as: Among them, P(c|X) is the probability that X belongs to the cth class, Indicates the prediction result.

2. The method for emotion recognition based on EEG signals according to claim 1, characterized in that: Step 12) specifically includes: dividing the EEG signal into 1 second segments, obtaining 2400 data samples for training the neural network; and using the first 3 seconds of silent data to represent the basic emotional state; in order to remove the influence of the basic emotional state, calculating the average value of each test sample minus the first 3 1s of data, and finally obtaining the final preprocessed sample X∈R by Z-score standardization. 32×128 .

3. The method for emotion recognition based on EEG signals according to claim 2, characterized in that: In step 12), the baseline signals contained in the DEAP dataset are used as reference signals, which are recorded when the subjects were not stimulated by the music video.

Images