A method for EEG emotion recognition based on feature and state bipartite graph

Abstract

The present invention provides an EEG emotion recognition method based on a feature and state bipartite graph. The specific steps of the method include guiding the subject to watch a film clip with obvious emotional tendencies, collecting EEG data at the same time, preprocessing and extracting features, and generating a data matrix. In order to obtain a bipartite graph that can well characterize the association between features and samples, the bipartite graph is first initialized using a data matrix, one layer is a feature, the other layer is a sample, there is no connection within the layer, and the layers are fully connected. Then, based on the graph structured learning theory, the optimal bipartite graph is learned by constraining it to be non-negative, row normalization and Laplace rank constraints, and then combined with a semi-supervised graph label propagation model to obtain the label of the unlabeled sample, the internal mechanism of which is that the same type of emotional samples have similar feature distributions. Finally, the learned label matrix is ​​split by column, and each column represents the degree of association between a type of emotion and a feature.

Description

Technical Field

[0001] The present invention belongs to the technical field of electroencephalogram signal processing, and specifically relates to an electroencephalogram emotion recognition method based on a feature and state bipartite graph. Background Art

[0002] Emotions are essential in interpersonal communication and affect our decision-making, cognition and behavioral expression, which makes emotion recognition a research hotspot in many fields such as cognitive neuroscience, health informatics, and psychology. Emotions are usually a psychological state that occurs spontaneously rather than consciously, and are often accompanied by central nervous and peripheral physiological reactions. Therefore, the electroencephalogram (EEG) generated by the electrical activity of the human cerebral cortex provides us with a reliable data source for emotion recognition. Unlike other data sources such as text, facial expressions, voice and other non-physiological signals, EEG signals are physiological signals and are not easy to disguise, making their recognition results more reliable. In recent years, with the rapid development of wearable devices, signal acquisition equipment and EEG data processing technology, emotion recognition based on EEG has attracted widespread attention from industry and academia.

[0003] Graph-based semi-supervised learning methods usually construct graph adjacency matrices by comparing the similarities between samples, where similarity can be measured by calculating Euclidean distance, Gaussian kernel, and other metrics. In the graph adjacency matrix, the larger the weight between two samples, the more similar the two samples are, and their labels should be the same. Therefore, the label propagation process based on the graph can be regarded as the label propagation of a limited number of labeled samples to a large number of unlabeled samples. Unlike traditional graph construction methods, we use preprocessed and feature-extracted EEG data as the input of a bipartite graph, with features on one side and samples on the other side (some samples are labeled, and the rest are unlabeled). There is no connection within the layer, and the layers are fully connected. Based on the graph structured theory, we obtain the optimal bipartite graph that can well reflect the relationship between features and samples.

[0004] Based on the learned optimal bipartite graph, on the one hand, we can get the labels of unlabeled samples, the internal mechanism of which is that the same type of emotion samples have similar feature distributions; on the other hand, we can analyze the association between each type of emotion sample and the feature. And each EEG feature is coupled with the corresponding EEG frequency band and channel. Assuming there are P frequency bands (usually P is 5, namely Delta, Theta, Alpha, Beta, Gamma) and Q channels (Q is usually 64, 128, etc.), each EEG sample vector is formed by connecting the Q features of each frequency band in the P frequency bands. By measuring the importance of each type of emotion sample and feature, the key frequency bands and channels for each type of emotion recognition are obtained. Summary of the invention

[0005] The purpose of the present invention is to provide an EEG emotion recognition method based on a feature and state bipartite graph; the method obtains a bipartite graph that can well describe the association between features and samples by jointly iteratively optimizing the structured graph P and the label matrix F. According to this bipartite graph, on the one hand, we can obtain the label of the unlabeled sample, and the predicted value label is the emotional state of the subject corresponding to the sample at the time of collection; on the other hand, the feature importance of each type of emotion can be obtained, and the feature is coupled with the channel and frequency band of the EEG, so as to analyze the EEG emotion activation pattern.

[0006] The specific steps of the EEG feature emotion category recognition method based on bipartite graph are as follows:

[0007] Step 1: Collect EEG data when the subjects are in different emotion categories.

[0008] Step 2: Preprocess and extract features of the EEG data collected in step 1 to obtain a data matrix d is the number of features, n is the number of samples. Use the data matrix B to initialize the bipartite graph. The number of nodes in one layer is the number of rows d of the data matrix B, and the number of nodes in the other layer is the number of columns n of the data matrix B. There is no connection within the layer, and the layers are fully connected. The element B of the data matrix B is ij Represents the weight between the i-th feature and the j-th sample.

[0009] Step 3: Based on the semi-supervised label propagation model of the feature-sample bipartite graph, the structured graph P is learned by enforcing non-negativity, row normalization, and Laplace matrix rank constraints, and the rank constraints are optimized and relaxed to obtain the optimized objective function. The obtained objective function is as follows:

[0010]

[0011] in, is the label matrix used for label propagation, is the label matrix of labeled samples, l is the number of labeled samples, c is the number of categories, G=[F u ; F f ]; is the label matrix of unlabeled samples, u is the number of unlabeled samples, is the label matrix describing the feature, d is the feature dimension; 1 is the all-one matrix; Tr(·) represents the trace of the matrix; λ is the parameter.

[0012] Step 4: Perform joint iterative optimization on the relation matrix P and label matrix F in the objective function.

[0013] Step 5: After the EEG data of the tested object is preprocessed, it is input into the optimization model after joint iterative optimization to obtain the label matrix F u; According to the label matrix F u Obtain the emotion category of the subject during EEG acquisition.

[0014] Preferably, the structured graph P learned in step 3 is:

[0015]

[0016] Among them, d is the original feature dimension, n is the number of samples; the graph correlation matrix is the Laplace matrix, obtained by the incidence matrix S; L S =D S -S;D S is a (n+d)-dimensional diagonal matrix whose i-th diagonal element is The first two constraints are non-negativity and row normalization, and the third constraint is that the graph association matrix S corresponds to the connected components of each emotional state, and then the EEG samples can be divided into each category; Denotes the calculation of the Frobenius norm; the superscript T denotes the transpose.

[0017] Preferably, in step 3, the process of obtaining the objective function is as follows:

[0018] Transform the structured graph P into:

[0019]

[0020] Among them, σ i (L S ) is the matrix L S The i-th smallest eigenvalue of , according to Ky Fan's theory:

[0021]

[0022] By relaxing the orthogonality constraints on F to be row normalized and non-negative, the objective function is obtained.

[0023] Preferably, in step 4, the specific process of performing joint iterative optimization on the association matrix P and the label matrix F is as follows:

[0024] Step 4.1, fix the label matrix F, update the association matrix P, and the objective function is:

[0025]

[0026] Solve formula (5) to obtain the updated correlation matrix P.

[0027] Step 4.2, fix the correlation matrix P, update the label matrix F, and the objective function is:

[0028]

[0029] F and L S Expand and sort, and the objective function is:

[0030]

[0031] Solve formula (7) to obtain the updated label matrix G.

[0032] Step 4.3: Repeat steps 4.1 and 4.2 to complete the joint iterative optimization of the bipartite graph association matrix P and the label matrix G.

[0033] Preferably, after step 5 is executed, key frequency bands and channels corresponding to different emotion categories are extracted, and the process is as follows:

[0034] Extract the label matrix G Take out f by column, study the correlation between each type of emotion sample and feature dimension, and calculate the importance θ of the i-th EEG feature to this type of emotion according to formula (8): i .

[0035]

[0036] θ i The larger the value of is, the more important this feature is for identifying this type of emotion. The importance measure ω(p) of the frequency band and The importance measure of each channel According to the numerical value of the importance metric, the key frequency bands and channels corresponding to different emotion categories are identified.

[0037] ω(p)=θ (p-1)*Q+1 +θ (p-1)*Q+2 +…+θ p*Q (9)

[0038]

[0039] Preferably, when it is necessary to identify whether the subject is in a certain emotion, the key frequency band and channel EEG data corresponding to the emotion category of the subject are collected; the EEG data are input into the objective function after preprocessing to confirm whether the subject is in this emotion.

[0040] The present invention has the following beneficial effects:

[0041] 1. The present invention constructs a bipartite graph that can well describe the relationship between features and samples, so as to accurately identify the emotional state of the subjects.

[0042] 2. Different from the existing methods that only focus on improving the performance of emotion recognition, it is necessary to further analyze where these important features come from (from which channels or frequency bands). f , the feature importance of each type of emotion sample can be obtained. On this basis, the key EEG frequency bands and channels corresponding to this type of emotion in the recognition process can be obtained, thereby reducing the amount of calculation to identify whether the object being tested is in a certain emotion category and analyzing the EEG emotion activation pattern. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 It is a schematic diagram of the identification process of the present invention. DETAILED DESCRIPTION

[0044] The present invention will be further explained below with reference to the accompanying drawings;

[0045] like Figure 1 As shown, a method for EEG emotion recognition based on a feature and state bipartite graph specifically includes the following steps:

[0046] Step 1: EEG data collection.

[0047] Human emotions do not appear to be very strong in daily situations. Therefore, in order to collect strong emotional information, it is necessary to induce the subjects to a certain extent. Therefore, several film clips with obvious emotional tendencies (such as movies with sad emotions, movies with happy emotions, etc.) are selected and played to the subjects at different times. While watching the movies, the EEG data of the subjects are collected through the EEG cap leads connected to the corresponding brain areas as the original emotional EEG data set.

[0048] Step 2: Data preprocessing.

[0049] The EEG data collected in step 1 are sampled at a sampling rate of 200 Hz, and then the noise and artifacts are filtered out through a bandpass filter of 1 Hz to 75 Hz. The differential entropy (DE) is calculated in Q (Q is usually 5) frequency bands (Delta (1-4 Hz), Theta (4-8 Hz), Alpha (8-14 Hz), Beta (14-31 Hz) and Gamma (31-50 Hz)) as the data matrix B:

[0050]

[0051] Among them, σ is the standard deviation of the probability density function; μ is the expectation of the probability density function.

[0052] It can be seen that the differential entropy feature is essentially the logarithmic form of the power spectrum density feature, that is, The preprocessing of EEG signals aims to improve the signal-to-noise ratio, thereby improving the preprocessing effect of the data and reducing interference. The label vector Y is the emotion label corresponding to the data matrix B.

[0053] Initialize the bipartite graph and take the data matrix B as input, where d is the number of features, n is the number of samples. One layer is for features, and the number of its nodes is the number of rows d of the data matrix B; the other layer is for samples, and the number of its nodes is the number of columns n of B. There is no connection within the layer, and the layers are fully connected. Among them, B ij Represents the weight between the i-th feature and the j-th sample.

[0054] Step 3.1, create a structured graph P:

[0055]

[0056] The structured graph P is learned by enforcing non-negativity, row normalization, and rank constraints to ensure that a good bipartite graph is constructed. d is the original feature dimension, n is the number of samples; Figure correlation matrix is the Laplace matrix, which is a symmetric matrix, obtained by the correlation matrix S, L S =D S -S, D S is a (n+d)-dimensional diagonal matrix, the i-th diagonal element The first two constraints are non-negativity and row normalization, and the third constraint is that we want the graph association matrix S to correspond to c connected components corresponding to each type of emotional state, and then the EEG samples can be divided into each type; Denotes the calculation of the Frobenius norm; the superscript T denotes the transpose.

[0057] Step 3.2: Since direct optimization of rank processing is non-convex and NP-hard, transform it into:

[0058]

[0059] Among them, σ i (L S ) is the Laplace matrix L S The i-th smallest eigenvalue of , according to Ky Fan's theory:

[0060]

[0061] By relaxing the orthogonality constraint on the label matrix F to row normalization, we obtain our objective function:

[0062]

[0063] in, is the label matrix used for label propagation, is the label matrix of labeled samples, l is the number of labeled samples, c is the number of categories; G = [F u ; F f ]; is the label matrix of the unlabeled samples (i.e. the input test data), and u is the number of unlabeled samples; is the label matrix describing the feature, d is the feature dimension, 1 is the all-one matrix, Tr(·) represents the trace operation of the matrix, and λ is a preset parameter.

[0064] Step 4: According to the optimization model established in step 3.2, the association matrix P and the label matrix F are jointly iteratively optimized.

[0065] Step 4.1, fix the label matrix F, update the association matrix P, and the objective function is:

[0066]

[0067] Among them, the first Item 2 According to the structure of matrix S, it can be further simplified Therefore, formula (16) becomes

[0068]

[0069] The objective function is independent for different i, and is defined as Defining vectors (p i and b i Similarly), so for each i, formula (17) can be written in the following vector form:

[0070]

[0071] At this time, Then formula (18) can be transformed into:

[0072]

[0073] The Lagrangian function of formula (19) for:

[0074]

[0075] Where α and φ are Lagrange multipliers. Taking the derivative about x and setting it to 0 gives:

[0076] xm-α1-φ=0(21)

[0077] Formula (21) combined with the constraint condition x T 1=1, we get:

[0078]

[0079] Substituting equation (22) back into equation (21), we can obtain:

[0080]

[0081] For formula (23), define the following variables:

[0082]

[0083] At this time, the formula can be written as:

[0084] x=u+(φ-φ * )(25)

[0085] According to the KKT condition in convex optimization problem, the following conclusions can be drawn:

[0086]

[0087] Combining equation (25) and equation (26), we can know that: Then φ k >0,x k =0; if Then φ k =0,

[0088] Therefore, formula (25) can be transformed into:

[0089]

[0090] In the formula, (a) + =max(a,0).

[0091] Formula (27) combined with the constraint condition x T 1=1, we can get the following function:

[0092]

[0093] At this time, Newton's method can be used to solve equation (28) to obtain φ * The optimal solution is:

[0094]

[0095] Step 4.2, fix the correlation matrix P, update the label matrix F, and the objective function is:

[0096]

[0097] F and LS Expand and sort, and the objective function is:

[0098]

[0099] Item 1 Where u is the number of unlabeled samples, d is the feature dimension, S ij is the connection weight corresponding to the matrix S. By optimizing G row by row, the objective function of formula (31) can be transformed into Related:

[0100]

[0101] in, l is the number of labeled samples, n is the total number of samples, and d is the feature dimension; Indicates that the matrix L is obtained first gl F l The i-th row of , and then transpose this row vector into a column vector.

[0102] Re-order And introduce the auxiliary variable z, let z = x, then equation (32) becomes:

[0103]

[0104] Based on the augmented Lagrangian method (ALM), equation (33) can be transformed into:

[0105]

[0106] Formula (34) is solved by iterative updating method:

[0107] Fix z and update x. Through the matching method, equation (34) can be further simplified to:

[0108]

[0109] In the formula, β is the Lagrange multiplier and μ is the growth coefficient. It can be solved by formula (19).

[0110] Fix x and update z. At this time, we can directly derive equation (33) with respect to z and set the derivative to 0, and the update formula of z is:

[0111]

[0112] Step 4.3: Repeat steps 4.1 and 4.2 to complete the joint iterative optimization of the bipartite graph association matrix P and the labeling matrix G of unlabeled samples and features.

[0113] Step 5: The label matrix G obtained in step 4 = [F u ; F f ], according to F u , take out each row, if one of the rows is [0.1 0.6 0.15 0.15] (assuming there are 4 emotions), then the predicted value label corresponding to this unlabeled sample is the second category, and the predicted value label is the emotional state of the subject corresponding to the sample at the time of collection.

[0114] Step 6: Label matrix G Take out f by column and study the correlation between each type of sentiment sample and the feature dimension.

[0115] According to formula (8), the importance θ of the i-th EEG feature to this type of emotion is calculated i .

[0116]

[0117] θ i The larger the value of , the more important this feature is for identifying this type of emotion. Each EEG feature is associated with an EEG frequency band and channel. Assume that we have P (P is usually 5, namely Delta, Theta, Alpha, Beta, Gamma) frequency bands and Q channels (Q is usually 64, 128, etc.). Each EEG sample vector is formed by connecting the Q features of each frequency band in the P frequency bands. The first EEG feature is calculated by equations (38) and (39) respectively. frequency band and The importance of each channel is measured to identify key EEG frequency bands and channels.

[0118] ω(p)=θ (p-1)*Q+1 +θ (p-1)*Q+2 +…+θ p*Q (38)

[0119]

[0120] In step 7 and step 6, the EEG frequency bands and channels corresponding to different emotion categories are obtained; when it is necessary to identify whether the subject needs to be in a certain emotion (such as whether it is in a sad state), the key EEG frequency band and channel EEG data corresponding to the emotion category are collected from the subject in advance; steps 2 to 4 are re-executed based on the obtained EEG data to obtain a new objective function; only the EEG data of the key EEG frequency bands and channels of the subject are collected, and input into the new objective function through preprocessing to confirm whether the subject is in this emotion.

Claims

1. A method for EEG emotion recognition based on feature and state bipartite graph, characterized by: The following steps are involved: Step 1: Collect EEG data when the subjects are in different emotion categories; Step 2: Preprocess and extract features of the EEG data collected in step 1 to obtain a data matrix d is the number of features, n is the number of samples; the bipartite graph is initialized using the data matrix B. The number of nodes in one layer is the number of rows d of the data matrix B, and the number of nodes in the other layer is the number of columns n of the data matrix B. There is no connection within the layer, and the layers are fully connected; the element B of the data matrix B ij Represents the weight between the i-th feature and the j-th sample; Step 3: Based on the semi-supervised label propagation model of the feature-sample bipartite graph, the structured graph P is learned by enforcing non-negativity, row normalization, and Laplace matrix rank constraints, and the rank constraints are optimized and relaxed to obtain the optimized objective function; the obtained objective function is as follows: in, is the label matrix used for label propagation, is the label matrix of labeled samples, l is the number of labeled samples, c is the number of categories, G=[F u ; F f ]; is the label matrix of unlabeled samples, u is the number of unlabeled samples, is the label matrix describing the feature, d is the feature dimension; 1 is the all-one matrix; Tr(·) represents the trace of the matrix; λ is the parameter; The learned structured graph P is: Among them, d is the original feature dimension, n is the number of samples; the graph correlation matrix is the Laplace matrix, obtained by the incidence matrix S; L S =D S -S;D S is a (n+d)-dimensional diagonal matrix, whose i-th diagonal element The first two constraints are non-negativity and row normalization, and the third constraint is that the graph association matrix S corresponds to the connected components of each emotional state, and then the EEG samples can be divided into each category; Indicates the calculation of the Frobenius norm; the superscript T indicates transpose; Step 4: Jointly iteratively optimize the correlation matrix P and the label matrix F in the objective function; Step 5: After the EEG data of the tested subject is preprocessed, it is input into the optimization model after joint iterative optimization to obtain the label matrix F u ; According to the label matrix F u Obtain the emotion category of the subject during EEG acquisition.

2. According to claim 1, a method for EEG emotion recognition based on feature and state bipartite graph is characterized in that: In step 3, the process of obtaining the objective function is as follows: Transform the structured graph P into: Among them, σ i (L S ) is the matrix L S The i-th smallest eigenvalue of , according to Ky Fan's theory: By relaxing the orthogonality constraints on F to be row normalized and non-negative, the objective function is obtained.

3. The method for EEG emotion recognition based on feature and state bipartite graph according to claim 1 is characterized in that: In step 4, the specific process of joint iterative optimization of the correlation matrix P and the label matrix F is as follows: Step 4.1, fix the label matrix F, update the association matrix P, and the objective function is: Solve formula (5) to obtain the updated correlation matrix P; Step 4.2, fix the correlation matrix P, update the label matrix F, and the objective function is: F and L S Expand and sort, and the objective function is: Solve formula (7) to obtain the updated label matrix G; Step 4.3: Repeat steps 4.1 and 4.2 to complete the joint iterative optimization of the bipartite graph association matrix P and the label matrix G.

4. The method for EEG emotion recognition based on feature and state bipartite graph according to claim 1, characterized in that: After step 5 is executed, the key frequency bands and channels corresponding to different emotion categories are extracted. The process is as follows: Extract the label matrix G Take out f by column, study the correlation between each type of emotion sample and feature dimension, and calculate the importance θ of the i-th EEG feature to this type of emotion according to formula (8): i ; θ i The larger the value of is, the more important this feature is for the recognition of this type of emotion. By using formula (9) and formula (10), we can calculate the first The importance measure ω(p) of the frequency band and The importance measure of each channel According to the numerical value of importance measurement, the key frequency bands and channels corresponding to different emotion categories are identified; ω(p)=θ (p-1)*Q+1 +θ (p-1)*Q+2 +…+θ p*Q (9) 5. The method for EEG emotion recognition based on feature and state bipartite graph according to claim 1, characterized in that: When it is necessary to identify whether the subject is in a certain emotion, the key frequency band and channel EEG data corresponding to the emotion category of the subject are collected; the EEG data are input into the objective function after preprocessing to confirm whether the subject is in this emotion.

Images