A method for generating EEG emotion classification model based on mutual information drive

Abstract

The present application relates to a method for generating an EEG emotion classification model driven by mutual information, which estimates the upper and lower bounds of the mutual information between two features, and tries to divide the EEG features into personalized and classification-independent features, as well as common and classification-related features. The model is generalized by using a commonality-related feature extractor, thereby improving the accuracy of EEG emotion classification in unknown fields.

Description

Technical Field

[0001] The present application relates to the technical field of EEG emotion recognition, and in particular to a method, system and storage medium for generating an EEG emotion classification model driven by mutual information. Background Art

[0002] Emotions are the result of the coordinated activities of the cerebral cortex and subcortical neural processes, and are highly transient. EEG is a spontaneous discharge activity that is not controlled by humans. It has the advantages of high temporal resolution and simplicity. Therefore, it is feasible to use EEG to identify emotions and reveal their complex neural mechanisms. EEG-based emotion research has received increasing attention in recent years. In the future, it can be used for virtual reality technology, as well as emotional communication between companion robots and service robots. In the future, virtual reality technology can be used for social interaction, online learning, large-scale technology exhibitions, and the treatment of mental illness. When people's emotions based on wearable devices can interact with other elements in real time, it can provide patients with quantitative results, especially in the treatment of emotional disorders or the quality of depression.

[0003] At present, researchers at home and abroad have used facial expressions, voices, and gestures to identify human emotions. Using a variety of pattern recognition methods, some results have been achieved. However, due to the controllability and disguiseability of the signals, the results cannot exclude the influence of the subjects' subjective factors, and sometimes the potential and true emotional state cannot be observed. At present, due to individual differences, the accuracy of emotion classification based on EEG signals in unknown areas is not high. Summary of the invention

[0004] In response to the above-mentioned problem that the accuracy of individual emotion classification in unknown fields is low due to individual differences in the current EEG signal emotion classification, this application proposes a method, system and storage medium for generating an EEG emotion classification model driven by mutual information.

[0005] In the first aspect, the present application proposes a method for generating an EEG emotion classification model driven by mutual information, comprising the following steps:

[0006] S101: Collect EEG emotion training data;

[0007] S102: performing data preprocessing on the EEG emotion training data, and then filtering the preprocessed EEG emotion training data in four frequency bands of α, β, θ and γ, and combining the filtered EEG emotion training data with the original EEG emotion training data to form EEG input data;

[0008] S103: using a two-dimensional convolutional network to perform two-dimensional convolution on the EEG input data to obtain initial features of the EEG signal;

[0009] S104: inputting the initial features of the EEG signal into a feature extractor based on the self-attention mechanism, and outputting the attention-based EEG features;

[0010] S105: Input the attention-based EEG features into one-dimensional convolution and pooling for low-dimensional feature mapping, thereby obtaining key features based on time dimension and channel;

[0011] S106: performing feature segmentation on the key features based on the time dimension and the channel, and then further reducing the dimension of the segmented features using a global feature extractor, thereby obtaining global features;

[0012] S107: Use a fully connected neural network to perform label prediction on global features to obtain an initial training model;

[0013] S108: Optimize the initial training model using the loss function to obtain an EEG emotion classification model.

[0014] By adopting the above technical solution, the present application estimates the upper and lower bounds of the mutual information between two features, and tries to divide the entangled EEG features into personalized and classification-independent features, and common and classification-related features. The model is generalized by the feature extractor related to commonality, thereby improving the accuracy of EEG emotion classification in the unknown domain.

[0015] Preferably, in S103, the two-dimensional convolution of the EEG input data using a two-dimensional convolutional network is specifically:

[0016]

[0017] Among them, Em_or is the initial feature of the EEG signal, is the EEG input data, and

[0018] Preferably, in S104, the feature extractor based on the self-attention mechanism includes a plurality of attention blocks using a multi-head attention mechanism, and the multi-head attention mechanism is specifically:

[0019]

[0020]

[0021] Em_a=A*Em_or;

[0022] Among them, M is the number of two-dimensional convolution kernels, K represents the key vector, V represents the value vector, and Q is the query vector. is a coefficient matrix containing M features and of the same size as q, L k is the length of EEG data, qi represents the i-th query vector, kj represents the j-th key value, Represents the magnitude of vector qi and vector kj.

[0023] Preferably, in 106, the feature segmentation based on the key features of the time dimension and the channel is specifically as follows:

[0024] [F re , F ir ] = split[Cov1d(Layer)];

[0025] Among them, F re is the class-related feature, F ir is a class-independent feature, split represents the segmentation function, and Covld represents one-dimensional convolution;

[0026] The further dimensionality reduction of the segmented features using the global feature extractor is specifically as follows:

[0027] Specifically:

[0028] Among them, l represents different neural network layers, w represents the deformation parameter, b represents the bias parameter, and Fg represents the gobal feature extractor.

[0029] Preferably, in S107, the label prediction of the global feature using the fully connected neural network is specifically:

[0030] Pre=Softmax((F g (f re ))·W+b);

[0031] Among them, Softmax is the normalization function, Fg is the gobal feature extractor, and f re is the category-related feature, W is the mapping parameter, and b is the bias parameter;

[0032] Preferably, in S108, the loss function is specifically:

[0033] loss=α·L1+β·L2+γ·L3;

[0034]

[0035] Among them, α, β, and γ are weight coefficients, xs and ys represent different domain data from different topics and corresponding labels, K represents different types of labels, E represents expectation, and pre k represents the predicted value;

[0036] L2=min E1，E2，C，D1 max M1 I θ (f ir ;fre );

[0037] I θ (X; Y) = E p [-sp(T θ (x,y))]-E q [spT θ (x, y′)];

[0038] sp(z)=log(1+e (z) );

[0039] Among them, I θ is the mutual information, x and y are the corresponding variables that need to be estimated for mutual information, T θ is a neural network composed of θ, and y′ is a variable randomly shuffled along the batch data axis y. p is the joint distribution of pq, and E q represents the marginal distribution, min represents the minimum mutual information of the optimized network parameters, and max represents the maximum mutual information of the optimized network parameters;

[0040] L3=loss local +loss global ;

[0041]

[0042]

[0043] Among them, F2 is the output of the last layer of neural network before classification, F re is the classification-related feature, loss global The representative introduces the M2 network to convert both into one-dimensional features and then calculate the mutual information. local The M3 network is introduced to copy and map the one-dimensional data of F2 into two-dimensional data and then map it to F re Perform fine-grained mutual information calculation, where N and h represent the length and width of the feature respectively.

[0044] In a second aspect, the present application also proposes a method for classifying EEG emotions, the method comprising the following steps:

[0045] S201: Obtaining EEG emotion target data that needs to be classified;

[0046] S202: Inputting the EEG emotion target data into an EEG emotion classification model, wherein the EEG emotion classification model is pre-trained based on the method described in the first aspect;

[0047] S203: Output the classification result of the EEG emotion classification model.

[0048] In a third aspect, the present application also proposes a device for generating an EEG emotion classification model driven by mutual information, the device comprising:

[0049] An acquisition module, which is used to acquire EEG emotion training data;

[0050] A preprocessing module, which is used to preprocess the EEG emotion training data;

[0051] A training module, wherein the training module is used to train the pre-processed EEG emotion training data into an EEG emotion classification model through the method described in the first aspect.

[0052] In a fourth aspect, the present application further proposes an EEG emotion classification device, the device comprising:

[0053] An acquisition module, the acquisition module is used to acquire EEG emotion target data collected by an EEG acquisition device;

[0054] A testing module, wherein the testing module is used to input EEG emotion target data into an EEG emotion classification model, wherein the EEG emotion classification model is pre-trained based on the method described in the first aspect, and output a classification result of the EEG emotion classification model.

[0055] In a fifth aspect, the present application further proposes a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in the first aspect.

[0056] In summary, this application at least includes the following beneficial technical effects:

[0057] 1. This application provides a new perspective for extracting domain-specific label-dependent features and shared domain-invariant label-dependent features by removing domain-specific label-independent features;

[0058] 2. The sparse multi-head attention neural network of this application is a device for basic feature extraction, which solves the limitation of convolution as a filter that only pays attention to adjacent channels and releases limited computational memory;

[0059] 3. The proposed method achieves state-of-the-art performance in extensive experiments on various benchmarks for domain-agnostic learning. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] The accompanying drawings are included to provide a further understanding of the embodiments and are incorporated into and constitute a part of this specification. The accompanying drawings illustrate the embodiments and are used together with the description to explain the principles of the present application. It will be easy to recognize other embodiments and many expected advantages of the embodiments because they become better understood by reference to the following detailed description. The elements of the drawings are not necessarily to scale with each other. The same reference numerals refer to corresponding similar parts.

[0061] Figure 1 This is a structural diagram of an EEG emotion classification model driven by mutual information in one embodiment of the present application.

[0062] Figure 2 It is a flowchart of a method for generating an EEG emotion classification model driven by mutual information in one embodiment of the present application.

[0063] Figure 3 It is a data flow diagram for performing feature segmentation based on the key features of the time dimension and the channel in one embodiment of the present application.

[0064] Figure 4 It is a visual schematic diagram of three features in one embodiment of the present application.

[0065] Figure 5 It is a flowchart of an EEG emotion classification method in one embodiment of the present application.

[0066] Figure 6 It is a schematic diagram of the module structure of an apparatus for generating an EEG emotion classification model driven by mutual information in one embodiment of the present application.

[0067] Figure 7 It is a schematic diagram of the module structure of an EEG emotion classification device in one embodiment of the present application.

[0068] Figure 8 It is a structural diagram of a computer system suitable for implementing an electronic device of an embodiment of the present application. DETAILED DESCRIPTION

[0069] The present application will be further described in detail below in conjunction with the accompanying drawings and embodiments. It will be appreciated that the specific embodiments described herein are only used to explain the relevant inventions, rather than to limit the inventions. It should also be noted that, for ease of description, only portions related to the relevant inventions are shown in the accompanying drawings. It should be noted that, in the absence of conflict, the embodiments in the present application and the features in the embodiments may be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in conjunction with the embodiments.

[0070] In order to make the embodiments of the present application clearer, mutual information and multi-source domain adaptive learning are described below:

[0071] Mutual information is an estimate of the degree of dependence between two random variables x and z, which measures the coexistence information of two random variables. It can also be understood as the uncertainty of X based on Z:

[0072] I(x;z): =H(x)-H(x|Z);

[0073] Among them, H(x) is Shannon entropy, which represents the uncertainty of event x. H(x|Z) represents the conditional entropy of z at x. When x and z are exactly the same, H(x|z) is 0, and the mutual information is maximum at this time H(x). When x and z are completely independent, the value of H(x|z) increases and the mutual information decreases. Mutual information is non-negative, and mutual information can also be expressed based on probability distribution theory as:

[0074]

[0075] Where P(xz) represents the joint distribution of x and z and p(x), p(z) represents the marginal distribution of x and z, D KL Represents the Kullback-Leibler divergence. In an autoencoder neural network, the mutual information between features and inputs is minimized, and the mutual information between features and labels is maximized. In this application, I(x;z) can be regarded as an encoder, which hopes to compress the source as much as possible to obtain a good representation; I(y;z) can be regarded as a decoder, which hopes to ensure that the compressed representation and the label are consistent, that is, this application cannot lose the information of the source X about Y. Mutual information can be used to strengthen the correlation between features in different layers of the neural network or reduce the correlation between different layers.

[0076] Most methods adopt multi-source domain adaptation learning. The generalization learning of multi-source domain data refers to relying on datasets distributed in different domains, aiming to learn a generalizable model. In the training process, there is no need to access the target domain data, and a shared feature extractor is used to learn domain-invariant features. However, domain invariance may impair semantic discrimination ability. Another method uses non-universal feature extractors and classifiers. Although non-shared feature extractors can better align the features of the target and source domains in the latent space, this greatly increases the number of parameters in the model. A large number of feature extractors and classifiers are required. Domain adaptation aims to transfer knowledge learning from the source domain to the target domain where labeled data is sparse or non-existent. Domain adaptation usually involves domain-invariant representation. Maximum mean difference (MMD) is used in most methods to minimize the difference in feature distribution between different domains; many adversarial strategies, such as gradient reversal layers and domain adversarial classifiers, have been successfully used for domain adaptation tasks. A one-one adaptation method is proposed in the current technology, which entangles the image embedding into three features through an autoencoder network: category-independent features, domain-specific category-related features, and domain-invariant category-related features based on mutual information. Compared with DAL, MutualIS simplifies the embedding learning process by omitting the decoder process and obtains enhanced semantic features by maximizing the mutual information between global and local features.

[0077] This application mainly provides a framework that tries to divide EEG features into personalized and classification-independent features and common and classification-related features by estimating the upper and lower bounds of the mutual information between two features. The model is generalized by using the feature extractor related to commonality, thereby improving the accuracy of EEG emotion classification in the unknown domain.

[0078] Each subject's data {x i ,y i} is considered as a field data, where {x i} is a multi-channel time series, {y i} is corresponding Label. Represents the kth source test data. The purpose of this study is based on multi-source data Train the shared feature extraction model F c and personalized feature extraction model F s And the classifier C based on these two features c and C s According to these two feature extractors and classifiers, {x t} label to predict, assuming that {x t} is an unlabeled data based on time series. To solve this problem, this method removes useless features from the extracted features by minimizing the mutual information between class-related and class-irrelevant related features.

[0079] Figure 1 The structure of the EEG emotion classification model driven by mutual information is shown in this application: E1 is the basic feature extractor, F1 is the output of E1, which is a highly entangled feature. Disentanglement D1 is responsible for decomposing the feature F that is not related to the class. ir and class-related features F re From basic features F1 to class-related F re It is input to the global feature extractor E2, the purpose of which is to convert high-dimensional features into low-dimensional features, and finally use the low-dimensional features for the emotion classifier C. The whole process consists of two data streams. The first data stream is from raw data to class-related features, and the second data stream is from raw data to predicted labels. The innovation of this application is to apply mutual information minimization to the entanglement of class-related features and class-irrelevant features. Mutual information maximization is applied to class-related features and global features to retain as many details as possible that are conducive to classification.

[0080] Reference Figure 2 , Figure 2 A flowchart of a method for generating an EEG emotion classification model driven by mutual information is shown in the present application, referring to Figure 2 , the method specifically comprises the following steps:

[0081] S101: Collect EEG emotion training data;

[0082] In a specific embodiment, S101 specifically includes: providing 32 participants (half of them are male), all of whom are composers, with an average age of 23.27 and a variance of 2.37. Each participant watched 40 one-minute music clips and evaluated his or her emotional state after watching for one minute each time. The evaluation dimensions include valence, arousal, preference, dominance and familiarity (valence, arousal, preference, dominance and familiarity). The evaluation value is between 1 and 9. The evaluation value is between 1-9, and the numerical value represents the attribute of the evaluation dimension from weak to strong, such as low price means sadness and high price means happiness. During the data collection process, when the subject closes his or her eyes for three seconds before watching the video, a physiological signal called a baseline signal is also recorded. The EEG acquisition device is the BiosemiActiveTwo system, and the EEG signal is collected through 32 channels of BioSemi with a sampling frequency of 512Hz. Other physiological signals include OMG, electromyography, and EOG signals. The available EEG data were processed (EEG and EOG were removed, and 4.0-45.0 Hz filtering was performed) and sampled at 128 Hz. Thirty-one of them were used as source data, and the remaining one was used as target data.

[0083] S102: performing data preprocessing on the EEG emotion training data, and then filtering the preprocessed EEG emotion training data in four frequency bands of α, β, θ and γ, and combining the filtered EEG emotion training data with the original EEG emotion training data to form EEG input data;

[0084] In a specific embodiment, the data preprocessing for the EEG emotion training data is specifically to filter the data at 4-54 Hz to eliminate irrelevant features.

[0085] The EEG signal is a chaotic and disordered electrical frequency signal. It is difficult to obtain sufficient frequency domain information through a simple one-dimensional convolution method. To maximize the acquisition of frequency domain information, time domain information, and spatial information, the EEG signal needs to be re-represented. This application filters the input signal in the following four frequency bands θ, α, β, γ, and combines it with the original signal. So the input data will become T×C×5, where C represents the number of channels and T represents the length of the time series. The input data is specifically:

[0086]

[0087] S103: using a two-dimensional convolutional network to perform two-dimensional convolution on the EEG input data to obtain initial features of the EEG signal;

[0088] In a specific embodiment, in S103, the two-dimensional convolution of the EEG input data using the two-dimensional convolution network is specifically:

[0089]

[0090] Among them, Em_or is the initial feature of the EEG signal, Input data for EEG.

[0091] In order to better preserve the association between channels and retain the information of time series, unlike natural language processing (NLP) embedding, EEG embedding uses two-dimensional convolution, where the convolution kernel is (1, 5c). In order to input the EEG representation into the attention model, this application designs a special convolution kernel, and then reduces the number of channels to 1 to ensure that the dimension of the feature is 2, and the final dimension of the EEG representation is (M, T).

[0092] S104: inputting the initial features of the EEG signal into a feature extractor based on the self-attention mechanism, and outputting the attention-based EEG features;

[0093] In S104, the feature extractor based on the self-attention mechanism includes a plurality of attention blocks using a multi-head attention mechanism, and the multi-head attention mechanism is specifically:

[0094]

[0095]

[0096] Em_a=A*Em_or;

[0097] Among them, M is the number of two-dimensional convolution kernels, K represents the key vector, V represents the value vector, and Q is the query vector. is a coefficient matrix containing M features and of the same size as q, L k is the length of EEG data, qi represents the i-th query vector, kj represents the j-th key value, Represents the magnitude of vector qi and vector kj.

[0098] The following is a detailed description of the feature extractor based on the self-attention mechanism:

[0099] After obtaining the initial features of the EEG signal, the next step is to design a basic feature extractor. The feature extractor hopes to condense the EEG length by emphasizing the important time points of the event sequence and reduce the length of the EEG signal as quickly as possible. The basic feature extractor consists of five layers of attention blocks. The attention block uses a multi-head attention module and a convolutional layer and pooling layer based on the time dimension. In order to reduce memory pressure, a sparse attention method is used.

[0100] The sparse attention mechanism is an improvement on the traditional attention mechanism. The traditional method performs matrix multiplication by inputting feature X and three parameters to obtain query vector Q, key vector K, and value vector V. The query vector and the key vector are multiplied to obtain the attention value, and then the value vector is weighted and summed according to the attention value.

[0101]

[0102]

[0103] Where X is the input feature, attention is the normalized value, and W is the parameter to be trained. The distribution of Attention is coefficient, and the sparsity of each Q needs to be calculated. Here, the distribution of Attention is used in the average distribution. The KL divergence between . The sparsity of Q attention is:

[0104]

[0105] Where L k is the length of the EEG data, where Yes i For the log-sum-exp of the inner product of all key values, the second term is q iThe arithmetic mean of the inner products of all key values. However, this is still a lot of computation, so a certain number of random sampling For an approximate estimate, since the dimension of the EEG data feature is 64, the number of random numbers set here is 16.

[0106] A certain number of Qs are estimated based on the sparsity of all Qs, and the top quries with low sparsity are selected for attention calculation, where It contains M features and is a coefficient matrix of the same size as q, where M<d.

[0107]

[0108] The remaining qury’s attention is no longer calculated, and the mean is used directly. Finally, the attention is merged into A and dot-multiplied with the original feature.

[0109] Em_a=A*Em_or;

[0110] S105: Input the attention-based EEG features into one-dimensional convolution and pooling for low-dimensional feature mapping, thereby obtaining key features based on time dimension and channel;

[0111] In a specific embodiment, the low-dimensional feature map of S105 is specifically:

[0112] atten j =MaxPool(Relu(Conv1d[Em_a]));

[0113] Among them, j represents the jth layer of the attention block. So Em_a is a channel matrix. Maxpool is the maximum pooling in one dimension, and its goal is to reduce the length of the sample. The feature length will be T / 2 5 , relu represents the activation function, and Convld represents one-dimensional convolution.

[0114] S106: performing feature segmentation on the key features based on the time dimension and the channel, and then further reducing the dimension of the segmented features using a global feature extractor, thereby obtaining global features;

[0115] In the step 106, the feature segmentation based on the key features of the time dimension and the channel is specifically as follows:

[0116] [F re , F ir ] = split[Cov1d(Layer)];

[0117] Among them, F re is the class-related feature, F iris a class-independent feature, split represents the segmentation function, and Covld represents one-dimensional convolution;

[0118] The disentangler applies 1D convolution to the base features, maps the feature size from D×T to 2D×T, and then clips the high-dimensional features into two features of the same size (class-related features F re Class-independent feature F ir ), where the specific data flow can be seen in Figure 3 .

[0119] The further dimensionality reduction of the segmented features using the global feature extractor is specifically as follows:

[0120]

[0121] Among them, l represents different neural network layers, w represents the deformation parameter, b represents the bias parameter, and Fg represents the gobal feature extractor.

[0122] The global features further reduce the dimensionality of the features. This application reduces the width and length of the features in two steps, and finally reshapes the two-dimensional features into one-dimensional data.

[0123] S107: Use a fully connected neural network to predict labels for global features to obtain an initial training model, thereby normalizing all labels.

[0124] In S107, the label prediction of the global feature using the fully connected neural network is specifically as follows:

[0125] Pre=Softmax((F g (f re ))·W+b);

[0126] Among them, softmax is the normalization function, Fg is the gobal feature extractor, f re is the category-related feature, W is the mapping parameter, and b is the bias parameter.

[0127] S108: Optimize the initial training model using the loss function to obtain an EEG emotion classification model.

[0128] In S108, the loss function is specifically:

[0129] loss=α·L1+β·L2+γ·L3;

[0130]

[0131] Among them, α, β, and γ are weight coefficients, xs and ys represent different domain data from different topics and corresponding labels, K represents different types of labels, E represents expectation, and pre k represents the predicted value;

[0132] L2=min E1，E2，C，D1 max M1 I θ (f ir ;f re );

[0133] I θ (X; Y) = E p [-sp(T θ (x,y))]-E q [spT θ (x, y′)];

[0134] sp(z)=log(1+e (z) );

[0135] Among them, I θ is the mutual information, x and y are the corresponding variables that need to be estimated for mutual information, T θ is a neural network composed of θ, and y′ is a variable randomly shuffled along the batch data axis y, Ep is the joint distribution of pq, and E q represents the marginal distribution, min represents the minimum mutual information of optimizing network parameters, and max represents the maximum mutual information of optimizing network parameters.

[0136] L3=loss local +loss global ;

[0137]

[0138]

[0139] Among them, F2 is the output of the last layer of neural network before classification, F re is the classification-related feature, loss global By introducing the M2 network, both are converted into one-dimensional features and then the mutual information is calculated. local The M3 network is introduced to copy and map the one-dimensional data of F2 into two-dimensional data and then map it to F re Perform fine-grained mutual information calculation, where N and h represent the length and width of the feature respectively.

[0140] The loss function loss of this application is described in detail below:

[0141] In order to solve the problem of EEG emotion recognition, all the classifications in this application are binary, and cross entropy is used to retain semantic information:

[0142]

[0143] where f re is a property associated with the class, x s ,y s represents different domain data from different topics and corresponding labels, K represents different types of labels, E represents expectations, and k Represents the predicted value.

[0144] Since the mutual information of variables x and y can be expressed as the Kullback-Leibler (KL) divergence of the joint distribution P and the marginal distribution Q, the mutual information lower bound can be reformulated as:

[0145] I(x, y) = D kl (p||Q)=sup E P [T θ (x, y)]-log E Q [e Tθ(x，y′) ];

[0146] Among them, P represents the joint distribution of two variables; Q represents the marginal distribution of two variables, T θ is a neural network with parameters θ. Here, the product of the marginal distributions Q is obtained by transforming the samples of the joint distribution along the batch axis. Since this application only focuses on maximizing the mutual information, the exact value of mutual information in training neural networks is unknown. The deep information model uses the Jensen-Shannon (JS) divergence as an alternative to the above formula, so the above formula can be rewritten as follows:

[0147] I θ (X; Y) = E p [-sp(T θ (x,y))]-E q [spT θ (x, y′)];

[0148] Where sp(z) = log(1+e (z) ), sp(z) is a softplus function. From the perspective of features, in order to obtain joint distribution and marginal distribution, a network M containing two variables is introduced to estimate f ir and f re The mutual information between them, and penalize the large mutual information value of M, so that the two factors f ir and f re Decomposition with minimal information overlap:

[0149] L2=min E1，E2，C，D1 max M1 I θ (f ir ;f re );

[0150] In this process, a two-layer fully connected network M1 is introduced to calculate the mutual information of relevant features and irrelevant features. θ (X; Y), maximize the mutual information of the two variables. The next valuation can maximize the mutual information of the two variables, but this patent aims to minimize the mutual information between the two features, so gradient reversal is used in M1. By maximizing M1, the model feature extractor E1, feature extractor E2, and classifier C are optimized to obtain the minimum mutual information of the two variables.

[0151] In order to enrich feature representation learning, the proposed model introduces multi-dimensional maximization of mutual information to reduce the information loss between global features and basic features. Specifically, based on the Deep infomax (DIM) algorithm, two sub-networks need to be designed, denoted as M loc and M glb exist Figure 1 M2 and M3 are in the two sub-networks. These two sub-networks realize the class-related features f re The mutual information between the two features M and the final feature f2 is maximized. loc The network estimates the feature f re The mutual information between each block of the segmentation and F2 at a fine-grained level:

[0152]

[0153] in the formula The jth local feature block of the class-related features is shown; F2 is mapped to each feature block by copying. glb Enhance the feature f on the coarse particles re Mutual information with F2 features:

[0154]

[0155] L3=loss local +loss global ;

[0156] Among them, F2 is the output of the last layer of neural network before classification, F re is the classification-related feature, loss global By introducing the M2 network, both are converted into one-dimensional features and then the mutual information is calculated. local The M3 network is introduced to copy and map the one-dimensional data of F2 into two-dimensional data and then map it to F reFine-grained mutual information calculation is performed, where N and h represent the length and width of the feature respectively. Its main purpose is to optimize E1, D1, E2, M2, and M3 to maximize the mutual information lower bound of F2 and Fre.

[0157] So the final loss function can be expressed as:

[0158] loss=α·L1+β·L2+γ·L3;

[0159] Among them, α, β, and γ are the weight coefficients of the three losses.

[0160] The embodiment of the present application also conducted a performance verification experiment on the EEG emotion classification model driven by mutual information.

[0161] The performance verification experiment used two data sets, one of which was the EEG emotion training data set in step S1 of this application, referred to as the DEAP data set in the following performance verification experiment, and the other was referred to as the Amigos data set in the following performance verification experiment, and the details of the Amigos data set are as follows:

[0162] The Amigos dataset includes 40 participants. All participants watched 17 short videos and 3 long videos. 37 participants watched all videos, 17 participants watched them alone, and the remaining 20 watched them in groups of 5. The data collection paradigm is the same as the DEAP dataset. First, the participants watched the induced video. After watching the video, the participants rated it according to their feelings. The data was collected using the Emotive Epoc, with a sampling frequency of 14 channels.

[0163] At the same time, after the end of this experiment, the subjects cut the corresponding viewing video into 20-second samples and asked another group of people to label them. The 20-second EEG label is not a self-evaluation, but an external label, called an external label. The purpose of external labeling is to verify whether the self-evaluation and external evaluation are consistent. The results show that people are more likely to express their emotional state in a social state than in a lonely state. At the same time, the results show that there is a high correlation between external labels and internal labels. This experiment uses internal labels as test labels.

[0164] In the validation experiment, both datasets used 1s sliding windows for non-overlapping sampling to avoid leaking test information. Valence and arousal are the most commonly used validation dimensions. The label threshold of the two datasets is set to 5, and emotion classification becomes a binary classification problem. During preprocessing, all data are subtracted from the average of the baseline. In the DEAP dataset, the baseline data is the first three seconds of EEG signals; in the Amigos dataset, the baseline data is the first 5s of each EEG data segment. The two datasets are normalized according to the channel. There are three validation schemes:

[0165] Scenario 1 (cross-subject learning): In this paradigm, all subject data, including source data and target data, are considered, and ten cross-validations are applied. This paradigm is mainly to demonstrate the feature extraction capability of the design framework.

[0166] Scenario 2 (ZERO-SHOT learning): Zero-shot learning is also called leave-one-out validation. Among all the subjects, one is selected as the target data, and the rest are used as test data. This paradigm is suitable for new users using wearable devices. Although a steady stream of target data can be obtained, there is no label.

[0167] Scenario 3 (Subject-dependent Learning): This is designed for individual EEG emotion classification, with few labels for model calibration. This application will compare the training time of model-based transfer with the training time without a base model.

[0168] The experiments were run on a 1080Ti GPU under Linux. The hyperparameters in the three cases were the same, with α, β, and γ being 0.3, 0.4, and 0.3, respectively. The entire model has three attention modules and two 2D convolutional layers. The training data for both scenario 1 and scenario 3 reached about 200 epochs.

[0169] Performance verification results on the DEAP dataset:

[0170] In the performance verification experiment of the DEAP dataset, it will be compared with the following benchmark methods: MDFV starts from the time domain, frequency and space features, and selects a set of effective hybrid features through carefully designed experiments to improve the accuracy of emotion classification. DANN-EEG is inspired by DANN. Its framework includes feature extractors, classifiers and discriminators. It hopes to train the feature extractor through the discriminator so that the features extracted by the feature extractor in different fields can deceive the discriminator, making it impossible for the discriminator to distinguish between the target domain and the source domain. MutualIS applies the de-entanglement framework, uses the minimization of mutual information to separate label-related features and label-irrelevant features, and applies the EEGnet model as the basic framework.

[0171] In verification scenario 1, the proposed method has a higher accuracy based on cross-test verification, which is better than other methods, proving that the proposed method is sufficient to extract common features; in the second verification scenario, the accuracy of the proposed method is improved by about 2% in valence and about 4% in arousal. The main reason is overfitting, and parameters need to be set to suppress overfitting. The final result is also better than other methods.

[0172] For scenario 3, this application is compared with state-of-the-art methods related to EEG emotion classification that rely on subjects or few-label learning. Samarth used deep and simple convolutional neural networks to improve the accuracy of emotion classification in the DEAP dataset. TLDR improved and proposed a Sincnet-based classifier Sincnet-R, which consists of three convolutional layers and three deep neural network (DNN) layers to test the classification accuracy and robustness of emotional EEG signals. SFA-FSL proposed a single-source domain adaptive few-shot learning network (SDA-FSL). A CBAM-based feature mapping module was designed to extract common features of the two domains, and the domain adaptation module was used to align the data distributions of the two domains. Wu et al. used the functional connectivity between brain regions to explore the strength, clustering coefficient and eigenvector centrality to improve the accuracy of EEG emotion recognition.

[0173]

[0174]

[0175] Table 6.1: Cross-person EEG emotion classification based on scenario 1

[0176]

[0177] Table 6.2: Individual-independent emotional EEG classification based on scenario 2

[0178]

[0179] Table 6.3: Comparison results of EEG emotion classification for test scenario 3

[0180] In general, the method proposed in this application achieves the best results in all three cases. In particular, in the third case, even with only 2 minutes of data as training data, the classification accuracy is very high (0.973% and 0.925%).

[0181] Performance verification results of the Amigos dataset:

[0182] There are 40 subjects in the Amigos dataset, only 37 subjects completed all the experiments, 9 subjects had abnormal data during the filtering process, and the final test data was 28 subjects. In the sample usage, this application only used the last 60 seconds of data, and the sample labels used in this application are internal labels. The experimental results of scenario 1 on the Amigos dataset are shown in Table 6.4. DCNN uses a 2d-cnn-based method that performs quite low in interdisciplinary studies, and even the training and test data can last for 20 seconds, with the final highest accuracy of 74.65%. All methods were trained 500 times. The results of scenario 2 are shown in Table 4:

[0183]

[0184] Table 6.4: Comparison results of Amigos across subjects in scenario 1

[0185]

[0186] Table 6.5: Comparison results based on scenario 2 independent of the subject Amigos

[0187]

[0188] Table 6.6: Comparison results of individual-dependent Amigos sentiment classification for test scenario 3

[0189] Reference Figure 4 , in order to prove that the disentanglement model obtains domain invariance. This application uses t-SNE to visualize three types of features: category-dependent features, category-independent features, and global features. Note that in the comparison of category-dependent features and global features distribution, global features are enriched by maximizing local and global mutual information.

[0190] The present invention discloses a method for generating an EEG emotion classification model based on mutual information drive, which has the following beneficial effects:

[0191] (1) By removing domain-specific label-irrelevant features, a new perspective is provided for extracting domain-specific label-dependent features and shared domain-invariant label-dependent features. At the same time, mutual information is used to replace adversarial learning in the feature separation process.

[0192] (2) Sparse multi-head attention neural network is a device for basic feature extraction, which solves the limitation of convolution as a filter that only pays attention to adjacent channels and releases limited computational memory.

[0193] (3) The proposed method achieves state-of-the-art performance in extensive experiments on various benchmarks for domain-agnostic learning.

[0194] Reference Figure 5 In the second aspect, the present application also proposes a method for classifying EEG emotions, the method comprising the following steps:

[0195] S201: Obtaining EEG emotion target data that needs to be classified;

[0196] S202: Inputting the EEG emotion target data into an EEG emotion classification model, wherein the EEG emotion classification model is pre-trained based on the method described in the first aspect;

[0197] S203: Output the classification result of the EEG emotion classification model.

[0198] Third, further reference Figure 6 As an implementation of the above method, the present application provides an embodiment of a device for generating an EEG emotion classification model driven by mutual information. Figure 1 Corresponding to the method embodiment shown, the system can be specifically applied to various electronic devices.

[0199] The present application proposes a device for generating an EEG emotion classification model based on mutual information drive, which specifically includes:

[0200] A collection module 301, which is used to obtain EEG emotion training data;

[0201] A preprocessing module 302, which is used to preprocess the EEG emotion training data;

[0202] The training module 303 is used to train the pre-processed EEG emotion training data into an EEG emotion classification model by using the method described in the first aspect.

[0203] Fourthly, reference Figure 7 , the present application also proposes an EEG emotion classification device, the device comprising:

[0204] An acquisition module 401 is used to acquire EEG emotion target data collected by an EEG acquisition device;

[0205] The testing module 402 is used to input EEG emotion target data into an EEG emotion classification model, the EEG emotion classification model is pre-trained based on the method described in the first aspect, and output a classification result of the EEG emotion classification model.

[0206] Reference below Figure 8 , which shows a schematic diagram of the structure of a computer system 500 suitable for implementing an electronic device of an embodiment of the present application. Figure 8The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0207] like Figure 8 As shown, the computer system 100 includes a central processing unit (CPU) 101, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 102 or a program loaded from a storage part 108 into a random access memory (RAM) 103. In the RAM 103, various programs and data required for the operation of the system 100 are also stored. The CPU 101, the ROM 102, and the RAM 103 are connected to each other through a bus 104. An input / output (I / O) interface 105 is also connected to the bus 104.

[0208] The following components are connected to the I / O interface 105: an input section 106 including a keyboard, a mouse, etc.; an output section 107 including a liquid crystal display (LCD), etc. and a speaker, etc.; a storage section 108 including a hard disk, etc.; and a communication section 109 including a network interface card such as a LAN card, a modem, etc. The communication section 109 performs communication processing via a network such as the Internet. A drive 110 is also connected to the I / O interface 105 as needed. A removable medium 111, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 110 as needed, so that a computer program read therefrom is installed into the storage section 108 as needed.

[0209] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication part 109, and / or installed from a removable medium 111. When the computer program is executed by a central processing unit (CPU) 101, the above-mentioned functions defined in the method of the present application are executed.

[0210] As another aspect, the present application also provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiment; or may exist independently without being assembled into the electronic device. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed by the electronic device, the electronic device implements the following when executing. Figure 1 The method shown in .

[0211] It should be noted that the computer-readable storage medium described in the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer-readable signal medium may also be any computer-readable storage medium other than a computer-readable storage medium that can send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0212] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0213] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0214] The above describes the specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0215] In the description of the present application, it should be understood that the terms "upper", "lower", "inside", "outside", etc. indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, which are only for the convenience of describing the present application and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present application. The wording 'comprising' does not exclude the presence of elements or steps not listed in the claims. The wording 'one' or 'an' preceding an element does not exclude the presence of a plurality of such elements. The simple fact that certain measures are recorded in mutually different dependent claims does not indicate that a combination of these measures cannot be used for improvement. Any reference symbols in the claims should not be interpreted as limiting the scope.

Claims

1. A method for generating an EEG emotion classification model driven by mutual information, characterized in that: The method comprises the following steps: S101: Collect EEG emotion training data; S102: performing data preprocessing on the EEG emotion training data, and then filtering the preprocessed EEG emotion training data in four frequency bands of α, β, θ and γ, and combining the filtered EEG emotion training data with the original EEG emotion training data to form EEG input data; S103: Use a two-dimensional convolutional network to perform two-dimensional convolution on the EEG input data to obtain the initial features of the EEG signal: Among them, Em_or is the initial feature of the EEG signal, Input data for EEG; S104: Input the initial features of the EEG signal into a feature extractor based on the self-attention mechanism, and output the EEG features based on attention. The feature extractor based on the self-attention mechanism includes a plurality of attention blocks using a multi-head attention mechanism. The algorithm of the multi-head attention mechanism is specifically as follows: Em_a=A*Em_or; Among them, M is the number of two-dimensional convolution kernels, K represents the key vector, V represents the value vector, and Q is the query vector. is a coefficient matrix containing M features and of the same size as q, L k is the length of EEG data, qi represents the i-th query vector, kj represents the j-th key value, Represents the magnitude of vector qi and vector kj; S105: Input the attention-based EEG features into one-dimensional convolution and pooling for low-dimensional feature mapping, thereby obtaining key features based on time dimension and channel; S106: Segment the key features based on the time dimension and the channel: [F re ,F ir ]=split[Cov1d(Em_a)]; Among them, F re is the class-related feature, F ir It is a class-independent feature, split represents the segmentation function, and Cov1d represents one-dimensional convolution; Then the global feature extractor is used to further reduce the dimension of the segmented features to obtain the global features: Among them, l represents different neural network layers, f re That is F re , is the class-related feature, w represents the deformation parameter, b represents the bias parameter, and Fg represents the gobal feature extractor; S107: Use a fully connected neural network to predict labels for global features to obtain an initial training model: Pre=Softmax((Fg(f re ))·W+b); Among them, Softmax is the normalization function, Fg is the gobal feature extractor, W is the mapping parameter, and b is the bias parameter; S108: Optimizing the initial training model using a loss function to obtain an EEG emotion classification model, wherein the loss function is: loss=α·L1+β·L2+γ·L3; Among them, α, β, and γ are weight coefficients, xs and ys represent different domain data from different topics and corresponding labels, K represents different types of labels, E represents expectation, and prek represents predicted value; L2=min E1，E2，C，D1 max M1 I θ (f ir ;f re ); I θ (X;Y)=E p [-sp(T θ (x,y))]-E q [sp(T θ (x,y′))]; sp(z)=log(1+e (z) ); Among them, I θ is the mutual information, x and y are the corresponding variables that need to be estimated for mutual information, T θ is a neural network composed of θ, and y′ is a variable randomly shuffled along the batch data axis y. p is the joint distribution of pq, and E q represents the marginal distribution, min represents the minimum mutual information of the optimized network parameters, max represents the maximum mutual information of the optimized network parameters, and f ir That is F ir , is a class-independent feature; L3=loss local +loss global ; Among them, E1 is the basic feature extractor, E2 is the global feature extractor, C is the emotion classifier, D1 is the disentanglement, Z is the variable, and M2 is the sub-network M loc , F2 is the output of the last layer of neural network before classification, N and h represent the length and width of the feature respectively, and loss local By introducing the M2 network, both are converted into one-dimensional features and then the mutual information is calculated. globall By introducing the M3 network, the one-dimensional data of F2 is copied and mapped into two-dimensional data and then combined with F re Perform fine-grained mutual information calculation.

2. The method for generating an EEG emotion classification model based on mutual information drive according to claim 1, characterized in that: In said S103, Among them, x θ is the EEG emotion training data filtered by the θ frequency band, x α is the EEG emotion training data filtered by α frequency band, x β is the EEG emotion training data filtered by β frequency band, x γ is the EEG emotion training data filtered by the γ frequency band, and x is the original EEG emotion training data.

3. A method for classifying EEG emotions, characterized by: The method comprises the following steps: S201: Obtaining EEG emotion target data that needs to be classified; S202: Inputting the EEG emotion target data into an EEG emotion classification model, wherein the EEG emotion classification model is pre-trained based on the method described in any one of claims 1-2; S203: Output the classification result of the EEG emotion classification model.

4. A device for generating an EEG emotion classification model driven by mutual information, characterized in that: The device comprises: An acquisition module, which is used to acquire EEG emotion training data; A preprocessing module, which is used to preprocess the EEG emotion training data; A training module, wherein the training module is used to train the pre-processed EEG emotion training data into an EEG emotion classification model through the method described in any one of claims 1-2.

5. An EEG emotion classification device, characterized in that: The device comprises: An acquisition module, the acquisition module is used to acquire EEG emotion target data collected by an EEG acquisition device; A test module, wherein the test module is used to input EEG emotion target data into an EEG emotion classification model, wherein the EEG emotion classification model is pre-trained based on the method described in any one of claims 1-2, and output a classification result of the EEG emotion classification model.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 3 is implemented.

Images