An EEG emotion recognition method based on a fusion graph convolutional network

Abstract

The present invention discloses an electroencephalogram (EEG) emotion recognition method based on a fused graph convolutional network. The EEG emotion recognition method is as follows: First, EEG is collected from the object to be measured, and the target features of each channel in the obtained EEG signal are extracted as the data to be recognized. Second, according to the data to be recognized obtained in the first step, the physical information matrix, correlation matrix, and causal relationship matrix between two channels are calculated. Third, an adjacency matrix is constructed. Fourth, a regularization matrix is constructed. Fifth, graph convolution operations are performed on the target features and the regularization matrix L to complete feature extraction. Sixth, the features obtained in the fifth step are input into a trained depthwise separable convolutional network, and the obtained feature map is input into a fully connected layer for classification to obtain the emotion category of the object to be measured during EEG collection. The present invention performs fusion processing on the adjacency matrix of graph convolution, so that the adjacency matrix contains diversified information instead of only single information, further improving the accuracy of model emotion classification.

Description

Technical Field

[0001] The present invention belongs to the field of electroencephalogram (EEG) emotion recognition, and relates to an EEG emotion classification method based on FGCNN (fusional graph convolutional neural network). Background Art

[0002] In the research on emotion recognition using EEG signals in the past decade, there have been various classifier methods to classify different types of emotional states. Such as K-nearest neighbor, Bayesian network, support vector machine (SVM), canonical correlation analysis, linear discriminant analysis (LDA), and backpropagation have been used by researchers to classify different emotions. However, due to the use of different classifiers, corresponding classifiers need to be used for different training datasets and test datasets, so that the model does not have universal applicability. With the popularity of deep learning, deep learning has gradually been applied to machine learning. The ultimate goal of deep learning is to make machines more intelligent and be able to learn useful and required information like humans.

[0003] Although traditional deep learning algorithms have achieved great success in feature extraction, many algorithms are only limited to data in Euclidean space and are based on the assumption that each channel is independent. However, in practical applications, a large amount of data is generated in non-Euclidean space, and there must be some interdependent relationship between EEG channels, making the performance of these traditional algorithms unsatisfactory. This has led to the emergence of graph neural network (GNN) based on graph data (based on graph theory), which uses an adjacency matrix to calculate the mutual influence relationship between nodes, and graph convolutional neural network (GCNN) can further extend the convolutional operation from traditional data to graph data.

[0004] Compared with CNN, GCNN has better signal processing ability, and the mutual relationship between EEG channels can be considered during model training. The GCNNs proposed in recent years can be mainly divided into three categories according to the construction method of the adjacency matrix. One is to construct the adjacency matrix using the physical relationship between channels, the second is to construct the adjacency matrix using the functional connection between channels, and the third is to construct the adjacency matrix using the causal relationship between channels.

[0005] In 2018, "EEG-based video identification using graph signal modeling and graph convolutional neural network" used the correlation coefficient, the physical distance between electrodes, and a random graph to obtain the in-band graph corresponding to each frequency band. This operation can obtain a matrix containing in-band and inter-band connection relationships, represent EEG data as a graph signal, and successfully apply GCNN to EEG-based video identification. "EEG-based emotion recognition using regularized graph neural networks" considered the biological topology between different brain regions to capture the local and global relationships between different EEG channels, thus constructing an adjacency matrix for graph convolution. "EEG emotion recognition using dynamical graph convolutional neural networks" proposed to dynamically update the matrix during the training of the graph convolution model to improve the emotion recognition accuracy.

[0006] In 2019, "Phase-locking value based graph convolutional neural networks for emotion recognition" used the Phase Lock Value (PLV) to model the multi-channel EEG signal features as a graph signal to extract the inter-band information hidden in the EEG signal. "GCNs-net: a graph convolutional neural network approach for decoding time-resolved EEG motor imagery signals" introduced the absolute Pearson matrix of the overall signal and distinguished four types of mental imagery graphs by establishing the Laplacian graph of EEG electrodes. The results proved that this method could fuse personalized and group predictions.

[0007] The causal graph convolution proposed in "A method for EEG emotion recognition based on depthwise separable causal graph convolutional network" used the Granger causality test to find the causal relationship between channels and constructed an asymmetric adjacency matrix, thereby improving the classification accuracy of graph convolution for emotion recognition.

[0008] Based on the above research, it can be found that there is a problem of single - dimensional information in the adjacency matrix constructed by existing graph convolution models. In order to further optimize the graph convolution model and enable graph convolution to extract diversified information between channels, a new model is needed. This model takes into account both the physical connections and correlation connections of EEG signals, as well as the causal connections between channels. Summary of the Invention

[0009] The purpose of the present invention is to provide an EEG emotion recognition method based on FGCNN for the deficiencies of the existing technology. First, calculate the physical distance relationship, correlation relationship, and causal relationship between EEG channels respectively, then fuse the three obtained matrices to get a fusion adjacency matrix containing diversified information required by the model. Next, use graph convolution to extract features from the fusion adjacency matrix and the original EEG signal, and finally enter the fully - connected layer to achieve the classification of EEG emotions.

[0010] In the first aspect, the present invention provides an EEG emotion recognition method based on a depth - separable causal graph convolution network, which includes the following steps:

[0011] Step 1: Collect EEG of the measured object and extract the target features of each channel in the obtained EEG signal as the data to be recognized. The target feature is any one of DE, DASM, and DCAU features.

[0012] Step 2: According to the data to be recognized obtained in Step 1, calculate the physical information matrix A d between two channels, the correlation matrix A p , and the causal relationship matrix Ac.

[0013] Step 3: Construct the adjacency matrix A F as follows:

[0014]

[0015] Wherein, represents element - by - element addition of matrix elements; respectively represent the elements at the corresponding positions of the physical information matrix, the correlation matrix, and the causal relationship matrix.

[0016] Step 4: Construct the regularization matrix L as follows:

[0017] L = I N - D -1 / 2 A F D -1 / 2

[0018] Wherein, I N is the identity matrix; D ∈ R N*N is the degree matrix of the graph; is calculated.

[0019] Step Five: Perform a graph convolution operation on the target feature and the regularization matrix L to complete feature extraction.

[0020] Step Six: Input the feature obtained in Step Five into the trained depthwise separable convolution network, input the obtained feature map into the fully connected layer for classification, and obtain the emotional category of the measured object during EEG acquisition.

[0021] Preferably, the target feature described in Step One is the DE feature.

[0022] Preferably, in Step Two, the physical information matrix A d is expressed as:

[0023]

[0024] where θ is the bandwidth that determines the radial range of action. ci and cj represent the features of any channel. τ is the boundary, and dist(i, j) represents the distance between the feature ci and the feature cj.

[0025] Preferably, in Step Two, the correlation matrix A p is expressed as:

[0026]

[0027] where cov(i, j) is the covariance between the feature ci and the feature cj; σ i and σ j are the standard deviations of the feature ci and the feature cj respectively.

[0028] Preferably, in Step Two, the causal relationship matrix A c is expressed as:

[0029]

[0030] where is the prediction error variance of the feature ci in the univariate AR model; e i is the prediction error of the feature ci in the univariate AR model; is the prediction error variance of the binary AR model of the feature d with respect to the feature cj; e ji are the binary prediction errors of the feature ci with respect to the feature cj respectively.

[0031] Preferably, the expression of the feature y extracted in Step Five is as follows:

[0032]

[0033] Among them, K is the order of the Chebyshev polynomial; θ k is the coefficient of the Chebyshev polynomial; is a recursive expression; k = 0, 1,... K-1; The expression of λ max is the largest element of the diagonal matrix Λ; the diagonal matrix Λ is obtained by singular value decomposition of the regularization matrix L.

[0034] Preferably, the recognized emotion categories include positive, neutral, and negative.

[0035] In a second aspect, the present invention provides an electroencephalogram emotion recognition system, which includes an electroencephalogram acquisition module, a channel relationship extraction module, a graph convolution module, and an identification module. The electroencephalogram acquisition module is used to collect and preprocess the electroencephalogram of the object to be measured; the relationship extraction module is used to extract the distance relationship, correlation relationship, and causal relationship between electroencephalogram channels, and perform fusion to obtain a regularization matrix. The graph convolution module is used to perform graph convolution operations on the target features in the electroencephalogram signal and the regularization matrix. The identification module is used to identify the features output by the graph convolution module through a depthwise separable convolution network and a fully connected layer, and obtain the emotion category of the object to be measured during electroencephalogram acquisition.

[0036] In a third aspect, the present invention provides a computer device, which includes a memory and at least one processor; the memory stores computer execution instructions; the at least one processor executes the computer execution instructions stored in the memory, so that the at least one processor executes the foregoing recognition method.

[0037] In a fourth aspect, the present invention provides a readable storage medium, which stores computer instructions, and the computer instructions are used to implement the foregoing recognition method when executed by a processor.

[0038] The beneficial effects of the present invention are:

[0039] The present invention performs fusion processing on the adjacency matrix of graph convolution, so that the adjacency matrix contains diversified information, rather than just having single information. By fully exploring the topological, functional, and causal relationships between electroencephalogram channels, it better utilizes various features of electroencephalogram data, enhances the interpretability of the model, and further improves the accuracy of model emotion classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 is the network framework diagram of the FGCNN model proposed by the present invention.

[0041] Figure 2 is the comparison diagram of the recognition results achieved by the relationship matrix used in the present invention and the existing relationship matrix. DETAILED DESCRIPTION OF THE INVENTION

[0042] To make the objectives, technical solutions, and key points of the present invention clearer, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0043] An electroencephalogram (EEG) emotion recognition method based on a depthwise separable causal graph convolutional network includes the following steps:

[0044] Step 1: Select a dataset

[0045] Use the Sentiment EEG SEED dataset provided by Shanghai Jiao Tong University; this dataset is generated by 15 subjects (7 males and 8 females) watching 15 segments from 6 movies respectively, which evoke different emotions (positive, neutral, negative). Therefore, each experiment of each subject generates 15 groups of data; each subject conducts three experiments, with each experiment separated by one week; each subject generates 15×3 = 45 groups of data. For each subject, in each experiment, the first 9 segments are taken as the training set, and the last 6 segments are taken as the test set. For each subject, two experiments are selected, and the final average result is used to measure the performance of the model. The present invention mainly processes the preprocessed DE, DASM, and DCAU features in the SEED dataset.

[0046] Step 2: Calculate the single-channel information

[0047] In this step, three matrices need to be established first, which respectively contain the physical information, correlation information, and causal information between channels.

[0048] Generally, the methods for determining physical distance include distance functions and K-nearest neighbor methods. The most commonly used classical distance function is the Gaussian kernel function, which is a relatively commonly used kernel function at present. It can realize the mapping of data to a high-dimensional space; the physical information matrix A between any two channels d can be expressed as:

[0049]

[0050] where θ is the bandwidth that determines the radial action range. That is to say, θ can determine the local domain of the Gaussian kernel function. ci and cj represent the EEG features of channels i and j. τ is the boundary, and dist(ci, cj) represents the distance between feature ci and feature cj.

[0051] Since EEG signals are continuous data and each channel is relatively independent, the present invention mainly uses the Pearson correlation coefficient, which is most suitable for such data, to obtain the correlation information between channels. The correlation coefficient refers to the change of one variable when other variables change. If there are two variables X and Y, the meaning of the finally obtained correlation coefficient can be understood as follows: (1) When there is no association between X and Y, the correlation coefficient is 0. (2) When variable X increases (decreases), variable Y increases (decreases), and the relationship between these two variables is between 0 and 1. (3) As variable X increases (increases), Y decreases (decreases), and this value is between -1 and 0.

[0052] To understand the Pearson correlation coefficient, it is necessary to first understand the covariance, which is an index reflecting the relationship between two variables X and Y, and its relational expression is as follows:

[0053]

[0054] Among them, are the means of variables X and Y respectively, and the total amount of variables is n.

[0055] Although the covariance formula can also reflect the correlation degree between two variables, in a two-dimensional space, if the correlation degree between two variables is low and the data distribution is relatively dispersed, the obtained covariance value is large, and the result is not very reasonable at this time. Therefore, the Pearson coefficient is introduced. The Pearson coefficient is obtained by dividing the covariance by their standard deviations on the basis of the covariance; the correlation matrix A between any two channels p is defined as:

[0056]

[0057] Among them, cov(ci, cj) represents the covariance of the features ci and cj of channel i and channel j, and σ i , σ j are the standard deviations corresponding to channels i and j respectively.

[0058] For the causal relationship between channels, select the features of two EEG channels and define them as c1 and c2. For the two time series c1 and c2, use the autoregressive (AR) model of c1 and the joint regression model of c1 and c2 to predict the influencing factors between the two channel features.

[0059]

[0060]

[0061]

[0062] Among them, Xt represents the time series of channel feature c1 within [0, t]; Y t represents the time series of channel feature c2 within [0, t]; α p , β p contains the AR coefficients of the AR model at time p; γ 11，p , γ 12，p , γ 21，p and γ 22，p are the binary regression coefficients of the bivariate regression model at time p; e1, e2 are the prediction errors of the univariate AR model; e 12 , e 21 are the binary prediction errors of channel feature c2 with respect to channel feature c1 and channel feature c1 with respect to channel feature c2, respectively.

[0063] We use the least squares method to estimate the regression model parameters and calculate the prediction errors of the regression model. Therefore, the Granger causal influence between the two channels is defined as:

[0064]

[0065]

[0066] where, is the variance of the prediction errors of feature c1 and feature c2 in the univariate AR model; is the variance of the prediction errors of the binary AR model of feature c2 with respect to feature c1 and feature c1 with respect to feature c2, which can be obtained through GC c1←c2 , GC c2←c1 represent the Granger causal influence of feature c2 on feature c1 and the Granger causal influence of feature c1 on feature c2, respectively.

[0067] Therefore, the matrix A containing causal information c is defined as:

[0068]

[0069] where, i, j represent any channel;

[0070] Step 3: Construction of the adjacency matrix

[0071] Aiming at the problem of the singularity of the adjacency matrix information in the existing graph convolution models, this step constructs a fused adjacency matrix. This adjacency matrix contains three relationships between EEG channels: distance relationship, correlation relationship, and causality relationship.

[0072] In Step 2, the distance relationship, correlation relationship, and causal relationship of EEG channels are calculated respectively, and three relationship matrices are obtained. In order to more intuitively integrate the information of the three matrices, the integration method of point-by-point addition is performed in this step.

[0073] Therefore, the fused adjacency matrix A F is defined as:

[0074]

[0075] where represents point-by-point addition of matrix elements; respectively represent the corresponding position elements of the physical information matrix, correlation matrix, and causal relationship matrix.

[0076] The diversified adjacency matrix obtained by this method contains three types of information: physical information, correlation information, and causal relationships between channels. Such a matrix is called a fused adjacency matrix. Such a matrix can provide more comprehensive and rich information about various relationships between channels for the model, which helps the model utilize this information to extract more effective emotion-related features.

[0077] Step 4. Regularize the matrix

[0078] According to the definition of the regularized Laplacian matrix L:

[0079] L = I N - D -1 / 2 A F D -1 / 2 (4.1)

[0080] where I N is the identity matrix; D ∈ R N*N is the degree matrix of the graph; D can be calculated by , and A F is the fused adjacency matrix obtained in Step 3.

[0081] Step 5. Feature extraction

[0082] The graph convolution operation refers to using the information of other nodes to predict and judge the information of this node. Its essence is to transmit features. This method transmits the feature information of the category to be predicted to the feature nodes of known categories, and uses the categories of these nodes to achieve the category prediction of unknown category nodes. The core idea is to use the information of edges to aggregate node information, thereby generating new node representations. Therefore, the more comprehensive the information of each node, the more accurate the prediction result.

[0083] On the graph convolutional neural network, the convolution of the signal x and the graph filter θ is defined as: y = g θ (L)x, g θIt is the filtering function of the graph signal. The singular value decomposition of the graph Laplacian L is: L = UΛU T , where U is an orthogonal function, composed of the eigenvectors of L, U = [u0, u1,..., u N-1 ∈ R N×N , Λ = diag([λ0, λ1,..., λ N-1 ) is a diagonal matrix, x ∈ R N*F is the input signal, N is the number of channels, F is the number of features, so:

[0084] y = g θ (L)x = Ug θ (Λ)U T x (5.1)

[0085] where U is the orthogonal function obtained by the singular value decomposition of the graph Laplacian L; g θ (Λ) is the operator of the filter θ.

[0086] However, to simplify the calculation, the K-order Chebyshev polynomial is introduced to calculate g θ (Λ).

[0087]

[0088] where, θ k is the coefficient of the Chebyshev polynomial; λ max is the largest element of the diagonal matrix Λ; and is calculated through a recursive expression, where The graph convolution operation can become:

[0089]

[0090] where, Substituting the regularized Laplacian matrix of formula (3.1) again, the final graph convolution formula (5.3) is obtained.

[0091] As Figure 1 shown, the fused graph convolution mainly consists of three parts: First, the original physiological signal is copied three times, and the physical distance information, correlation information, and causality information between channels are extracted respectively. The three matrices of channel number × channel number obtained are fused to obtain the final fused adjacency matrix.

[0092] The second part is the graph convolution part, where the graph convolution operation is performed through the Chebyshev polynomial to extract the spatial features of the electroencephalogram topological map. The third part is to classify the features obtained in the previous layer and output the final predicted category.

[0093] Step 6: Input the features extracted in Step 5 into a trained depthwise separable convolutional network, and input the obtained feature map into a fully connected layer and a softmax function for classification. The training process is to use the features processed by Steps 2 to 5 of the training set to train the separable convolutional network and adjust the parameters. The specific process belongs to the prior art and will not be elaborated here.

[0094] Step 7: Collect EEG data from the subjects whose emotional types need to be tested, and preprocess it to obtain the target features of each channel; the target feature is any one of DE, DASM, and DCAU; process the obtained target features of each channel according to the methods in Steps 2 to 6; the fully connected layer outputs the probabilities of the subjects being in three emotional types (positive, neutral, negative); take the emotional category corresponding to the maximum probability as the recognition result of the current emotional category of the subject.

[0095] Compare the EEG emotion recognition method (denoted as FGCNN) provided in this embodiment with four existing methods (SVM, GCNN, DGCNN, CGCNN). The results are shown in Table 1-3.

[0096] Table 1 Accuracy and standard deviation of the model on the DE feature of the SEED dataset

[0097]

[0098]

[0099] Table 2 Accuracy and standard deviation of the model on the DASM feature of the SEED dataset

[0100]

[0101] Table 3 Accuracy and standard deviation of the model on the DCAU feature of the SEED dataset

[0102]

[0103] It can be found from Table 1, Table 2, and Table 3 that compared with the SVM, GCNN, DGCNN, and CGCNN methods, the FGCNN method proposed in the present invention has a certain improvement in the accuracy rate in most frequency bands of the three features (DE, DASM, and DCAU). Among the comparison methods, SVM is a classic machine learning method. GCNN is the baseline of the FGCNN method proposed in the present invention, and the recognition accuracy rate reaches 87.40%. DGCNN updates the GCN by dynamically updating the constructed graph, and improves the recognition accuracy rate to 90.40%. CGCNN constructs a graph network using causal relationships and improves the recognition accuracy rate to 93.36%. The FGCNN method proposed in the present invention fuses graphs containing different brain connection features to obtain a unified representation of EEG data, achieving a higher accuracy rate of 94.10%.

[0104] Experiments conducted on the SEED dataset show that compared with other graph models, the proposed fused graph convolutional neural network (FGCN) improves the accuracy of emotion recognition, indicating that the fused graph contains rich spatial information of EEG data and proving the effectiveness of FGCN.

[0105] Experiments conducted in different frequency bands prove that the β band and γ band are more effective for EEG-based emotion recognition, and at the same time prove that the diversification of adjacency matrix information helps to improve the emotion recognition ability of the graph convolutional model.

[0106] Experiments conducted with different relational matrices prove that the richer the brain connection information contained in the graph, the more effective the FGCN is for EEG-based emotion recognition, as Figure 2 shown.

[0107] At the same time, we tried different fusion strategies (pointwise addition, cross-diffusion process, pointwise multiplication, Kronecker product, and Kronecker addition). Among them, the local fusion strategy (pointwise addition) and pointwise multiplication we selected do not fuse the information of adjacent channels, while the other three fusion methods have information interaction of adjacent channels. It can be found from Table 4 that the addition of elements has a better recognition effect than the multiplication of elements implemented in any way. Different connection information transmission between adjacent channels may lead to confusion in the recognition results and reduce the recognition accuracy rate. Therefore, the fusion strategy of pointwise addition has a better fusion effect in EEG-based emotion recognition than other complex fusion strategies.

[0108] Table 4 Accuracy rates of different fusion strategies for the DE feature on the SEED dataset

[0109]

Claims

1. A method for electroencephalogram emotion recognition based on a fusion graph convolutional network, characterized in that: Including the following steps: Step 1: Perform electroencephalogram (EEG) acquisition on the object to be measured, and extract the target features of each channel in the obtained EEG signals as the data to be recognized; The target feature is any one of DE, DASM, and DCAU features; Step 2: Calculate the physical information matrix A between two channels based on the recognized data obtained in Step 1 d , correlation matrix A p , and causality matrix A c ; Physical information matrix A d Expressed as: Where θ is the bandwidth determining the radial action range; ci and cj represent the features of any channel; τ is the boundary, and dist(i,j) represents the distance between feature ci and feature cj; Step 3: Construct the adjacency matrix A F As follows: Among them, represents the element-by-element addition of matrix elements; respectively represent the corresponding position elements of the physical information matrix, the correlation matrix, and the causal relationship matrix; Step 4: Construct a regularization matrix L as follows: L = I N -D -1 / 2 A F D -1 / 2 where I N is the identity matrix; D ∈ R N*N is the degree matrix of the graph; is calculated; Step 5: Perform a graph convolution operation on the target feature and the regularization matrix L to complete feature extraction; Step 6: Input the features obtained in Step 5 into a trained depthwise separable convolution network, input the obtained feature map into a fully connected layer for classification, and obtain the emotional category of the object to be measured during EEG acquisition.

2. The electroencephalogram emotion recognition method based on a fusion graph convolutional network according to claim 1, wherein: The target feature described in Step 1 is the DE feature.

3. The EEG emotion recognition method based on the fusion graph convolutional network according to claim 1, wherein: In step 2, the correlation matrix A p is expressed as: where cov(i,j) is the covariance between feature ci and feature cj; σ i and σ j are the standard deviations of feature ci and feature cj, respectively.

4. A method for electroencephalogram emotion recognition based on a fusion graph convolutional network according to claim 1, characterized in that: In step 2, the causal relationship matrix A c is expressed as: Among them, is the prediction error variance of feature ci in the univariate AR model; e i is the prediction error of feature ci in the univariate AR model; is the prediction error variance of the bivariate AR model of feature ci with respect to feature cj; e ji are respectively the bivariate prediction errors of feature ci with respect to feature cj.

5. The electroencephalogram emotion recognition method based on a fusion graph convolutional network according to claim 1, characterized in that: The expression of the feature y extracted in Step 5 is as follows: where K is the order of the Chebyshev polynomial; θ k is the coefficient of the Chebyshev polynomial; is a recursive expression; k = 0, 1,... K-1; The expression of λ max is the largest element of the diagonal matrix Λ; the diagonal matrix Λ is obtained by singular value decomposition of the regularization matrix L.

6. The method for electroencephalogram emotion recognition based on a fusion graph convolutional network according to claim 1, wherein: The recognized emotional categories include positive, neutral, and negative.

7. An electroencephalogram (EEG) emotion recognition system, comprising an EEG acquisition module; characterized in that: It further includes a channel relationship extraction module, a graph convolution module, and an identification module; the EEG emotion recognition system is used to execute the method described in any one of claims 1-6; The EEG acquisition module is used to perform EEG acquisition and preprocessing on the object to be measured; the relationship extraction module is used to extract the distance relationship, correlation relationship, and causal relationship between EEG channels, and fuse them to obtain a regularization matrix; the graph convolution module is used to perform a graph convolution operation on the target feature and the regularization matrix in the EEG signal; the identification module is used to identify the features output by the graph convolution module through a depthwise separable convolution network and a fully connected layer to obtain the emotional category of the object to be measured during EEG acquisition.

8. A computer device, comprising a memory and at least one processor; characterized in that: The memory stores computer execution instructions; the at least one processor executes the computer execution instructions stored in the memory, so that the at least one processor executes the recognition method described in any one of claims 1-6.

9. A readable storage medium stores computer instructions; characterized in that: When the computer instructions are executed by the processor, they are used to implement the recognition method described in any one of claims 1-6.

Images