A Clustering-Based Method for Extracting and Recognizing Multitask Emotion EEG Features

Abstract

The present invention provides a clustering-based multi-task emotional EEG feature extraction and recognition method. The steps of the present invention are as follows: 1. Multiple subjects respectively collect EEG data in an induced emotional state scenario. 2. Preprocess the EEG data obtained in step 1. 3. Cluster the processed EEG data to obtain a subclass label matrix. 4. Establish a clustering multi-task feature extraction algorithm to solve for the feature weight distribution. 5. Extract features from the EEG data according to the feature weight distribution to train an EEG emotion recognition model. The present invention improves the prediction accuracy of the EEG emotion recognition model through clustering algorithms, multi-task learning, and feature extraction.

Description

Technical Field

[0001] The present invention belongs to the technical field of electroencephalogram signal processing, and in particular relates to a multi-task emotion electroencephalogram feature extraction and recognition method based on clustering. Background Art

[0002] With the development of science and technology, the study of EEG emotion recognition and related brain mechanisms has gradually become a hot research field in neuroscience. Using computer technology for EEG emotion recognition is a key technology for realizing advanced human-computer interaction. Real-time and efficient EEG emotion recognition is of great significance for realizing human-computer interaction, human-computer interface and intelligent computers. In terms of specific applications, EEG emotion recognition can be used to judge the user's emotional state, help the computer make appropriate responses, and improve the effect of human-computer interaction. EEG emotion recognition technology has received widespread attention in the field of artificial intelligence.

[0003] Due to the instability of EEG signals, the collected EEG signals have the characteristics of low signal-to-noise ratio, resulting in redundancy of EEG data features, which affects the accuracy of EEG emotion recognition models. Existing scientific theories show that there is important intrinsic structural information in the distribution of real EEG data, that is, by exploring the inherent subclass information in EEG data, more prior knowledge can be provided for the EEG emotion recognition model, thereby improving the accuracy of the EEG emotion recognition model. The present invention proposes a multi-task emotion EEG feature extraction and recognition method based on clustering. We explore multiple emotion subclasses under the original emotion category, use multi-task learning to perform emotion recognition on the subclasses, and use the emotion recognition task of each subclass as a subtask. By sharing information between tasks, the performance of each subtask is improved, thereby more effectively mining the data feature weight distribution in the EEG emotion recognition task. Through the feature weight distribution, more discriminative features in the EEG emotion data are extracted. Using the EEG features after feature extraction to train the EEG emotion recognition model can effectively improve the accuracy of the EEG emotion recognition model. Summary of the invention

[0004] The purpose of the present invention is to provide a multi-task emotion EEG feature extraction and recognition method based on clustering. This method can realize EEG emotion data feature extraction and EEG emotion recognition.

[0005] The specific steps of the present invention are as follows:

[0006] Step 1: Conduct an EEG data collection experiment on multiple subjects; provide external stimulation to the subjects to cause them to have emotional changes, and collect the EEG signals of the subjects at the same time.

[0007] Step 2: Preprocess all the EEG data obtained in Step 1. The processed data of each EEG sample of a subject is used as a sample matrix X; each sample matrix X corresponds to a label vector y; the label vector y corresponds to the emotion category of the subject.

[0008] Step 3: Cluster the processed EEG data to explore the emotion subcategories in the original emotion categories of the EEG data, and obtain an emotion subcategory label matrix Y.

[0009] Step 4: Establish a clustering multi-task feature extraction model to solve the feature weight matrix.

[0010] Step 4-1: Establish the objective function of the clustering multi-task feature extraction algorithm model as shown in Equation (1):

[0011]

[0012] In the formula, is the subcategory emotion label matrix after clustering, where n represents the number of samples and c represents the number of subcategory emotion categories; is the EEG emotion data, where n represents the number of samples and d represents the data dimension; represents the feature weight matrix of the EEG emotion data, where d represents the data dimension and c represents the number of subcategory emotion categories; represents the Laplacian matrix of the EEG emotion data. The specific calculation method is L = D - S, where is a similarity matrix describing the subcategory relationship, and the specific definition is x i represents the i-th sample of the EEG emotion data X, and x j represents the j-th sample of the EEG emotion data X; is a diagonal matrix about S, and the specific definition is λ is a hyperparameter used to control the sparsity of the weight matrix W; β is a hyperparameter used to control the influence of the subcategory relationship on the objective function; represents the square of the F-norm of the matrix, and the specific calculation method is represents the square of the 2,1-norm of the matrix, and the specific calculation method is tr(·) represents the trace of the matrix, and the specific calculation method is the sum of the diagonal elements of the matrix.

[0013] Step 4-2: Solve Equation (1) to obtain the feature weight matrix W of the EEG emotion data in the EEG emotion recognition task.

[0014] Step 5: Extract features from the EEG emotion data according to the feature weight matrix W, and train a classification model based on the extracted features of the EEG emotion data to obtain an EEG emotion recognition model.

[0015] Preferably, in step 1, the emotional state of the subject is induced by having the subject watch movie clips of different emotional types, causing the subject to have emotional changes of happiness, sadness, fear, and neutrality.

[0016] Preferably, the label vector y described in step 2 is defined by the emotional type of the movie clip watched by the subject during the EEG data acquisition process. The element values in the label vector y are 1, 2, 3, and 4.

[0017] Preferably, the preprocessing process in step 2 is as follows.

[0018] Step 2-1. Downsample the EEG data to 200 Hz and perform band-pass filtering on it to the range of 1-50 Hz; according to the 5-band method, divide it into five bands: δ, θ, α, β, and γ.

[0019] Step 2-2. Perform short-time Fourier transform with a time window of 4 seconds and non-overlapping on the EEG data of these 5 bands respectively, and extract the differential entropy feature h(X) as shown in formula (2),

[0020] h(X) = -∫ x f(x)ln(f(x))dx, (2)

[0021] In formula (2), X is the input sample matrix, x is the element in the input sample matrix; f(X) is the probability density function. The updated differential entropy feature h(X) is as shown in formula (3):

[0022]

[0023] In formula (3), σ is the standard deviation of the probability density function; μ is the expectation of the probability density function.

[0024] Preferably, the EEG data acquisition uses 62 leads and selects 5 bands; the 5 bands are 1-4 Hz, 4-8 Hz, 8-14 Hz, 14-31 Hz, and 31-50 Hz respectively.

[0025] Preferably, the clustering method in step 3 uses the AP (Affinity Propagatio) clustering algorithm, and the maximum number of clustering iterations is 1000 times.

[0026] Preferably, the specific solution process for the weight matrix W in step 4 is as follows:

[0027] When the subclass label matrix Y, the EEG emotion data X, and the Laplacian matrix L are determined, formula (1) can be determined as:

[0028]

[0029] Since Theoretically, it exists and will make Equation (4) non-differentiable. Therefore, let where represents the square of the 2-norm of the vector, and its specific calculation method is ∈ is a very small constant close to 0. At this time, Equation (4) becomes:

[0030]

[0031] Find the Lagrangian function of Equation (5)

[0032]

[0033] For Derive with respect to W and set the derivative to 0 to obtain:

[0034]

[0035] In the formula, Q is a diagonal matrix related only to W, and its i-th diagonal element q ii is:

[0036]

[0037] Through Equation (7), the update formula of W can be obtained:

[0038] W = (X T X + λQ + βX T LX) -1 X T Y (9)

[0039] Preferably, in step 5, the weight matrix W obtained by training is processed, and a constant γ = 10 -5 is introduced. Let where w ij is the element in the i-th row and j-th column of the weight matrix W; according to the processed weight matrix Extract features from the EEG emotion data, and only retain the j-th dimensional feature of the EEG emotion data X corresponding to in the weight matrix where represents the j-th row feature weight in the weight matrix Train the features of the extracted EEG emotion data using a support vector machine classification model to obtain an EEG emotion recognition model, where the support vector machine uses a radial basis kernel function type.

[0040] The beneficial effects of the present invention are:

[0041] 1. The present invention clusters electroencephalogram (EEG) emotion data, expands four types of emotion recognition problems into multi-class emotion recognition problems, and makes full use of the underlying structural information of EEG emotion data in real situations, so that the finally extracted features are more discriminative in the EEG emotion recognition task.

[0042] 2. In the present invention, multi-task learning is introduced on the basis of clustering. The emotion recognition tasks of each subclass in the original emotion category are used as subtasks for learning, and the learning performance of the subtasks is improved by sharing the subtask parameters, making full use of the relationship between subclasses, so as to more effectively explore the underlying structural information of EEG emotion data.

[0043] 3. After extracting more discriminative EEG data features, the present invention performs model training on EEG emotion data. Compared with the EEG emotion recognition model without feature extraction, the EEG emotion recognition model obtained after feature extraction has higher accuracy in the EEG emotion recognition task. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 is a schematic flow chart of the present invention;

[0045] Figure 2 is a model framework diagram of a clustering-based multi-task emotion EEG feature extraction and recognition method. DETAILED DESCRIPTION OF THE INVENTION

[0046] The present invention will be further described below with reference to the accompanying drawings.

[0047] The present invention solves the two important problems of EEG emotion data feature extraction and EEG emotion recognition based on the following starting points:

[0048] In the EEG emotion recognition task, due to the non-stationary characteristics of EEG signals, the collected EEG signals have the characteristics of low signal-to-noise ratio, resulting in redundant EEG data features and affecting the accuracy of the EEG emotion recognition model. Based on the underlying structural information existing in real EEG emotion data, we believe that the information between EEG emotion subclasses can be used to explore more discriminative features in the EEG emotion recognition task and train a better-performing EEG emotion recognition model.

[0049] As Figure 1 and Figure 2 shown, a clustering-based multi-task emotion EEG feature extraction and recognition method is as follows:

[0050] Step 1: Use external stimuli (in this embodiment, a movie clip viewing scenario containing the induction of the subject's emotional state) to collect the EEG data of the subject's induced emotional state.

[0051] EEG data of N subjects were collected M times in the same EEG data acquisition experimental environment, obtaining N·M sets of EEG data. The data volume of each set of data is n*d, where n is the number of EEG data samples related to time obtained in a single acquisition, and d is the dimension of each set of data. Each set of data is used as a sample matrix X. Each sample matrix X corresponds to a label y; the label y corresponds to the emotional type of the movie clip watched by the subject. Before the experiment, the movie clips were edited. Each movie clip was about two minutes long. Through the emotional category evaluation of the movie clips, the movie clips of the four most evocative target emotions (happy, sad, neutral, and fear) were selected for the experiment. Before each EEG emotional data collection, the subjects needed to be informed of the purpose and steps of the experiment and the harmlessness of the entire experiment. There would be a 5-second start prompt before the subjects watched the movie clips that induced emotional states, and a 45-second self-assessment after watching the movie clips. The subjects were required to self-assess their true feelings. If the subjects failed to generate the correct emotion or the emotion was not strong enough, the EEG emotional data of the subjects were regarded as invalid.

[0052] Step 2. Preprocess all the EEG data obtained in Step 1. The present invention is based on 62 leads and 5 frequency bands (Delta (1 - 4Hz), Theta (4 - 8Hz), Alpha (8 - 14Hz), Beta (14 - 31Hz), and Gamma (31 - 50Hz)), and differential entropy features are extracted for this purpose. In practical applications, the number of leads depends on the EEG cap worn by the subject during data collection; the division of frequency bands also follows the 5-frequency-band division with physiological significance; the most commonly used features of EEG signals are power spectral density and differential entropy. The EEG signals of humans are very weak, which means that EEG signals are vulnerable to interference, and the collected results are difficult to directly conduct experiments on, which poses requirements for EEG signal preprocessing:

[0053] The preprocessing process is as follows:

[0054] Step 2-1. Downsample the EEG data to 200Hz and then band-pass filter it to the range of 1 - 50Hz. According to the 5-frequency-band method, it is divided into five frequency bands: δ, θ, α, β, and γ.

[0055] Step 2-2. Take the EEG data of these 5 frequency bands as sample matrices respectively, and perform short-time Fourier transform with a time window of 4 seconds and non-overlapping, and extract differential entropy features. The differential entropy feature h(X) is defined as:

[0056] h(X) = ∫ x f(x)ln(f(x)dx, (10)

[0057] In Equation (10), X is the input sample matrix, x is the element in the input sample matrix; f(x) is the probability density function. The updated differential entropy feature h(X) is shown in Equation (11):

[0058]

[0059] In Equation (11), σ is the standard deviation of the probability density function; μ is the expectation of the probability density function.

[0060] It can be seen that essentially the differential entropy feature is the logarithmic form of the power spectral density feature, that is

[0061] Step 3: We perform a clustering algorithm on the preprocessed EEG emotion data to explore the subclasses in the original categories of the EEG emotion data so as to utilize the inner structure information under real EEG data and solve the problem of redundant data features in the EEG emotion recognition task. Specifically, we use the AP clustering algorithm to divide the four-class EEG data into multiple subclasses, encode each subclass with a unique label, convert the label vector y of the original EEG emotion category into the label matrix Y of the EEG emotion subclass, and transform the four-class problem into a multi-class problem.

[0062] The AP clustering algorithm is a clustering algorithm based on the information transfer of data points. This algorithm regards all data samples as network nodes and calculates the clustering centers of each node through the information transfer among the nodes in the network. During the clustering process, the information of "attractiveness" and "ownership" is continuously transmitted among the nodes, and finally the clustering centers are selected to complete the clustering. This clustering algorithm does not need to determine the final number of categories, ensuring that the inner structure information of the EEG emotion data is mined to the greatest extent. In actual use, it is necessary to control the convergence effect of the AP clustering algorithm. In this embodiment, the maximum number of iterations of the AP clustering is set to 1000 times, ensuring the effectiveness of the EEG emotion subclasses.

[0063] Step 4: By utilizing the relationship between the EEG emotion subclasses, more discriminative features in the EEG emotion data are explored. Specifically, we use the clustering multi-task feature extraction algorithm. In the EEG emotion data, the emotion recognition task of each subclass is regarded as a sub-task, and the performance of each sub-task is improved by sharing the parameters among the sub-tasks. Through this algorithm, the feature weight distribution of the EEG data can be obtained.

[0064] Step 4-1. Establish the objective function of the clustering multi-task feature extraction algorithm model as shown in Equation (12).

[0065]

[0066] In the formula, is the sub-class sentiment label matrix after clustering, where n represents the number of samples and c represents the number of sub-class sentiment categories; is the EEG emotion data, where n represents the number of samples and d represents the data dimension; represents the EEG emotion data feature weight matrix, where d represents the data dimension and c represents the number of sub-class sentiment categories; represents the Laplacian matrix of the EEG emotion data. Its specific calculation method is L = D - S, where is a similarity matrix describing the sub-class relationship, specifically defined as x i represents the i-th sample of the EEG emotion data X, x j represents the j-th sample of the EEG emotion data X; is a diagonal matrix about S, specifically defined as λ is a hyperparameter used to control the sparsity of the weight matrix W; β is a hyperparameter used to control the influence of the sub-class relationship on the objective function; represents the square of the F-norm of the matrix, and its specific calculation method is represents the square of the 2,1-norm of the matrix, and its specific calculation method is tr(·) represents the trace of the matrix, and its specific calculation method is the sum of the diagonal elements of the matrix.

[0067] Step - 2. Optimize the model based on the Lagrange method and solve for the feature weight matrix W.

[0068] When the sub-class label matrix Y, the EEG emotion data X, and the Laplacian matrix L are determined, equation (12) can be determined as:

[0069]

[0070] Since theoretically exists and will make equation (13) non-differentiable, so let where represents the square of the 2-norm of the vector, and its specific calculation method is ∈ is a very small constant close to 0. At this time, equation (13) becomes:

[0071]

[0072] Find the Lagrangian function of equation (14)

[0073]

[0074] For Take the derivative with respect to W and set the derivative to 0 to get:

[0075]

[0076] In the formula, Q is a diagonal matrix related only to W, and its i-th diagonal element q ii is:

[0077]

[0078] Through formula (16), the update formula of W can be obtained:

[0079] W = (X T X + λQ + βX T LX) -1 X T Y (18)

[0080] Step 5. Analyze the weight matrix W of the sub-category emotion recognition task obtained in Step 4. We believe that more discriminative features have larger weight values in the sub-category EEG emotion recognition task. On the contrary, the features corresponding to smaller weight values in the sub-category EEG emotion recognition task are less discriminative. Therefore, more discriminative features can be extracted through the weight matrix in the sub-category emotion recognition task. Since a constant ∈ is introduced in formula (14), the weight matrix W obtained by solving in Step 4 is infinitely close to zero rather than zero. Therefore, we introduce a constant γ = 10 -5 , and let where w ij is the element in the i-th row and j-th column of the weight matrix W. According to the processed weight matrix perform feature extraction on the EEG emotion data, and only retain the j-th dimensional feature of the EEG emotion data X corresponding to in the weight matrix , where represents the feature weight of the j-th row in the weight matrix . Train the extracted EEG emotion features and the original EEG emotion category label vector to obtain an EEG emotion recognition model. In this embodiment, a support vector machine is used for classification model training, and the support vector machine uses a radial basis kernel function.

Claims

1. A method for extracting and recognizing multi-task emotion EEG features based on clustering, characterized in that, it includes the following steps: Step 1: Conduct EEG data collection experiments on multiple subjects; apply external stimuli to the subjects to cause emotional changes, and at the same time collect the EEG signals of the subjects; Step 2: Preprocess all the EEG data obtained in Step 1; the processed data of each subject's EEG sample is used as a sample matrix X; each sample matrix X corresponds to a label vector y; the label vector y corresponds to the emotional category of the subject; Step 3: Cluster the processed EEG data, retrieve the emotional subcategories in the original emotional category of the EEG data, and obtain an emotional subcategory label matrix Y; Step 4: Establish a clustering multi-task feature extraction model to solve the feature weight matrix; Step 4-1: Establish the objective function of the clustering multi-task feature extraction algorithm model as shown in Equation (1): In the formula, is the sub-class sentiment label matrix after clustering, where n represents the number of samples and c represents the number of sub-class sentiment categories; is electroencephalogram emotion data, where n represents the number of samples and d represents the data dimension; Represents the EEG emotion data feature weight matrix, where d represents the data dimension and c represents the number of subclass emotion categories; The Laplacian matrix representing electroencephalogram (EEG) emotion data is calculated as L = D - S, where is a similarity matrix describing subclass relationships and is defined as x i represents the i-th sample of the EEG emotion data X, and x j represents the j-th sample of the EEG emotion data X; is a diagonal matrix about S, defined as λ is a hyperparameter used to control the sparsity of the weight matrix W; β is a hyperparameter used to control the impact of subclass relationships on the objective function; denotes the square of the F-norm of a matrix, and its calculation method is denotes the square of the 2,1-norm of a matrix, and its calculation method is tr(·) denotes the trace of a matrix, and its calculation method is the sum of the diagonal elements of the matrix; Step 4-2: Solve Equation (1) to obtain the feature weight matrix W of the EEG emotion data in the EEG emotion recognition task; Step 5: Extract features from the EEG emotion data according to the feature weight matrix W, and train a classification model based on the extracted features of the EEG emotion data to obtain an EEG emotion recognition model.

2. A method for extracting and recognizing multi-task emotion EEG features based on clustering according to Claim 1, characterized in that, in Step 1, the emotional state of the subject is induced by letting the subject watch movie clips of different emotional types, so that the subject has emotional changes of happiness, sadness, fear and neutrality.

3. A method for extracting and recognizing multi-task emotion EEG features based on clustering according to Claim 2, characterized in that, the label vector y in Step 2 is defined by the emotional type of the movie clips watched by the subject during the EEG data collection process; the elements in the label vector y take values of 1, 2, 3, and 4.

4. A method for extracting and recognizing multi-task emotion EEG features based on clustering according to Claim 1, characterized in that, the preprocessing process in Step 2 includes the following sub-steps: Step 2-1: Sample the EEG data to 200Hz and perform band-pass filtering on it to the range of 1-50Hz; according to the 5-band method, divide it into five bands: δ, θ, α, β and γ Step 2-2: Perform short-time Fourier transform with a time window of 4 seconds and non-overlapping on the EEG data of these five bands respectively, and extract the differential entropy feature h(X) as shown in Equation (2), h(X) = -∫ x f(x)ln(f(x))dx, (2) in Equation (2), X is the input sample matrix, x is the element in the input sample matrix; f(x) is the probability density function; the updated differential entropy feature h(X) is as shown in Equation (3): in Equation (3), σ is the standard deviation of the probability density function; μ is the expectation of the probability density function.

5. A method for extracting and recognizing multi-task emotion EEG features based on clustering according to Claim 1, characterized in that: the EEG data collection uses 62 leads and selects 5 bands; the 5 bands are 1-4Hz, 4-8Hz, 8-14Hz, 14-31Hz, and 31-50Hz respectively.

6. A method for extracting and recognizing multi-task emotion EEG features based on clustering according to claim 1, wherein, in step 3, the AP clustering algorithm is used as the clustering method, and the maximum number of clustering iterations is 1000 times.

7. A method for extracting and recognizing multi-task emotion EEG features based on clustering according to claim 1, wherein, In step 5, the weight matrix W obtained by training is processed, and a constant γ = 10 is set -5 , and let where w ij is the element in the i-th row and j-th column of the weight matrix W; according to the processed weight matrix feature extraction is performed on the EEG emotion data; the j-th dimensional feature of the EEG emotion data X corresponding to in the weight matrix is retained, where represents the feature weight of the j-th row in the weight matrix ; a support vector machine classification model is used to train the features of the extracted EEG emotion data to obtain an EEG emotion recognition model, where the support vector machine uses the radial basis kernel function type.

Images