Cross-domain electroencephalogram emotion recognition method and device based on multi-scale relation learning

Abstract

The invention relates to a cross-domain electroencephalogram emotion recognition method and device based on multi-scale relation learning, and the method comprises the steps: inputting an electroencephalogram signal into a multi-scale relation learning network, and obtaining a fused emotion feature through a multi-scale separation module, a connection relation learning module and a multi-scale fusion module which are connected in sequence in the multi-scale relation learning network; wherein the fused emotional features comprise information of a plurality of source domains; and inputting the fused emotion features into a general feature induction network, eliminating interference of the personalized emotion features on emotion recognition by using a Dirichlet priori network in the general feature induction network, and obtaining an emotion recognition result. According to the technical scheme, the multi-scale relation learning network is used for capturing the emotional characteristics, the brain connection relation better conforming to cranial neuroscience can be learned, emotional information of multiple source domains is comprehensively considered, the fused emotional characteristics are concluded, personalized emotional interference is eliminated, and the generalization ability is improved.

Description

Technical Field

[0001] The present invention relates to the field of emotion recognition technology, and in particular to a cross-domain EEG emotion recognition method and device based on multi-scale relationship learning. Background Art

[0002] Emotions are the reactions of organisms to specific stimuli (people, situations or events), reflecting the current physiological and psychological state of the organism, and are crucial in various human experiences. It is easy for humans to distinguish the emotions of others and give appropriate responses (for example, not laughing when others are sad), but it is difficult for machines to do so, which makes machines appear not smart enough in many situations. In order to make machines serve humans better, it is very meaningful to develop machines that can recognize human emotions.

[0003] Human body signals (e.g., voice, facial expressions, postures, etc.) are widely used in emotion recognition due to their easy collection advantages, and have been developed for many years. However, this method has a certain degree of subjectivity, and people can disguise their emotions by controlling their body signals. In contrast, physiological signals such as electroencephalogram (EEG), electromyography, heart rate, temperature, etc. are generated autonomously by the nervous system and are difficult to be consciously controlled, so they are more reliable. Among them, EEG signals are generated by the central nervous system, respond to emotional changes more promptly, and have been proven to provide important features for emotion recognition.

[0004] EEG signals contain very rich and complex information, but not all information is related to emotions. How to extract emotion-related information from complex EEG signals has always been a difficult problem that researchers have been trying to solve. Some scholars have tried to extract emotion-related manual features from the time domain, frequency domain, and time-frequency domain, but these features may be too low-level to provide sufficient information for emotion recognition. Deep learning methods based on recurrent neural networks and convolutional neural networks can automatically learn high-level features from EEG signals, but they often focus more on the time-frequency domain and do not have enough ability to learn the spatial relationship between multiple EEG channels. Studies have shown that emotions may be related to certain circuits in the brain, so researchers should consider the connection between various brain regions rather than considering certain brain regions in isolation. However, the human brain is composed of tens of billions of neurons, and the connection between various brain regions is extremely complex. A channel in an EEG signal cannot represent a basic unit of the human brain, so the simple graph structure between EEG channels cannot fully reflect the complex connection between various brain regions.

[0005] Even if the model has the ability to extract richer EEG emotional information, it is equally important to ensure that the extracted emotional features are universal and applicable to most people, rather than relying on the personal emotional features of specific subjects. Since the cortical features from the same person are stable, many EEG-based emotion recognition works perform well when they rely on subjects. However, they often perform poorly when they are independent of subjects. This is because there are large individual differences in EEG signals. In EEG emotion recognition independent of subjects, some information with personal attributes will become interference factors, causing great obstacles to emotion recognition. Methods to solve the problem of emotion recognition independent of subjects can be divided into domain adaptation and domain generalization. Domain adaptation methods often require the target domain data to participate in training or fine-tuning the model, which is often difficult to meet in practice. At the same time, since the model is only adapted to the data of a specific target domain, the resulting model only performs well on the target domain, so what is learned is not a universal emotional feature. Summary of the invention

[0006] In order to solve the above problems existing in the prior art, the present invention provides a cross-domain EEG emotion recognition method and device based on multi-scale relationship learning.

[0007] According to a first aspect of an embodiment of the present invention, a cross-domain EEG emotion recognition method based on multi-scale relationship learning is provided, the method comprising:

[0008] Inputting the EEG signal into a multi-scale relational learning network, and using a multi-scale separation module, a connection relational learning module and a multi-scale fusion module sequentially connected in the multi-scale relational learning network to obtain a fused emotion feature; wherein the fused emotion feature includes information from multiple source domains;

[0009] The fused emotion feature is input into a universal feature induction network, and the Dirichlet prior network in the universal feature induction network is used to eliminate the interference of personalized emotion feature on emotion recognition, so as to obtain the emotion recognition result.

[0010] Optionally, the step of inputting the EEG signal into a multi-scale relational learning network and obtaining a fused emotion feature by using a multi-scale separation module, a connection relational learning module and a multi-scale fusion module sequentially connected in the multi-scale relational learning network includes:

[0011] Inputting the electroencephalogram signal into the multi-scale separation module to obtain the electroencephalogram signals of multiple different scales;

[0012] Inputting the EEG signals of multiple different scales into the connection relationship learning module to obtain multi-scale emotion features;

[0013] The multi-scale emotion feature is input into the multi-scale fusion module to obtain the fused emotion feature.

[0014] Optionally, inputting the electroencephalogram signal into the multi-scale separation module to obtain the electroencephalogram signals of multiple different scales includes:

[0015] Initialize the first weight matrix;

[0016] Obtaining a scale transformation matrix according to the first weight matrix;

[0017] The electroencephalogram signal is processed according to the scale transformation matrix to obtain the electroencephalogram signals of multiple different scales.

[0018] Optionally, the method further comprises:

[0019] Constraining the scale transformation matrix by a first loss function;

[0020] The first loss function is expressed as follows:

[0021]

[0022] Among them, l q Denotes the first loss function, Q j represents the scale transformation matrix of the jth scale, j = 1, 2, ..., J; T represents the matrix transpose, Indicates n j The unit matrix of order n j represents the number of nodes at the jth scale, ‖·‖ F represents the Frobenius norm.

[0023] Optionally, the step of inputting the electroencephalogram signals of multiple different scales into the connection relationship learning module to obtain multi-scale emotion features includes:

[0024] Initialize the second weight matrix;

[0025] Obtain an edge matrix according to the second weight matrix;

[0026] Obtaining a sparse edge matrix according to the edge matrix and the mask matrix;

[0027] Processing the sparse edge matrix to obtain a Laplacian matrix;

[0028] A multi-scale emotion feature is obtained according to the Laplacian matrix and the electroencephalogram signals of multiple different scales.

[0029] Optionally, the Laplacian matrix is ​​expressed as follows:

[0030]

[0031] Among them, Lj represents the Laplace matrix of the jth scale, j = 1, 2, ..., J; Indicates n j The unit matrix of order n j represents the number of nodes at the jth scale, T represents the matrix transpose, represents the unmasked sparse edge matrix at the jth scale, represents the edge mask matrix of the jth scale, mask represents the mask matrix, E j represents the edge matrix of the jth scale;

[0032] The multi-scale emotion feature is expressed as follows:

[0033]

[0034] in, represents the emotion feature of the jth scale, sigmoid(·) represents the sigmoid activation function, represents the k-th order Chebyshev parameter vector, Represents the Laplacian operator at the scale j The k-th order Chebyshev polynomial calculated at , λ j max represents the Laplacian matrix L of the jth scale j The maximum eigenvalue of Indicates n j ×n j The set of real matrices of dimension X j represents the EEG signal of the jth scale;

[0035] The second loss function is expressed as follows:

[0036]

[0037] Among them, l e represents the second loss function, which is used to Orthogonalize, ‖·‖ F represents the Frobenius norm.

[0038] Optionally, the fused emotion feature is expressed as follows:

[0039]

[0040] in, represents the fused emotion feature, Q 1 represents the scale transformation matrix of the first scale, Represents the emotional features of the first scale after feature dimension alignment.

[0041] Optionally, the general feature induction network further includes a plurality of classifiers; input ends of the plurality of classifiers receive the fused emotion features, and output ends of the plurality of classifiers are connected to input ends of the Dirichlet prior network; the fused emotion features are input into the general feature induction network, and the Dirichlet prior network in the general feature induction network is used to eliminate interference of personalized emotion features on emotion recognition, so as to obtain emotion recognition results, including:

[0042] Inputting the fused emotion features into the multiple classifiers respectively, and constraining the multiple classifiers through a third loss function to obtain multiple emotion distribution predictions;

[0043] The multiple emotion distribution predictions are input into the Dirichlet prior network, and the Dirichlet prior network is optimized by a fourth loss function to eliminate the interference of personalized emotion features on emotion recognition, thereby obtaining the emotion recognition result.

[0044] Optionally, the method further comprises:

[0045] The multiple classifiers are constrained by a third loss function, and the third loss function is expressed as follows:

[0046]

[0047] Among them, l c represents the loss function, l cross (·) represents the cross entropy function, μ ( i) represents the prediction of the sentiment distribution corresponding to the data of the i-th source domain in the training set, i=1,2,…,s, s represents the number of data of multiple source domains into which the training set is divided, Represents the label corresponding to the data of the i-th source domain in the training set;

[0048] The Dirichlet prior network is optimized by a fourth loss function, and the fourth loss function is expressed as follows:

[0049]

[0050] Among them, l d represents the fourth loss function, λ is a scalar parameter, l dpn represents the maximum likelihood loss of the Dirichlet prior network branch, Indicates the emotion recognition result corresponding to the training set.

[0051] According to a second aspect of an embodiment of the present invention, a cross-domain EEG emotion recognition device based on multi-scale relationship learning is provided, the device comprising:

[0052] A feature extraction module, used for inputting the EEG signal into a multi-scale relational learning network, and obtaining a fused emotion feature by using a multi-scale separation module, a connection relational learning module and a multi-scale fusion module sequentially connected in the multi-scale relational learning network; wherein the fused emotion feature includes information of multiple source domains;

[0053] The emotion recognition module is used to input the fused emotion feature into a universal feature induction network, and utilize the Dirichlet prior network in the universal feature induction network to eliminate the interference of personalized emotion feature on emotion recognition, so as to obtain the emotion recognition result.

[0054] The technical solution provided by the present invention may include the following beneficial effects:

[0055] Through the above technical solutions, the multi-scale relational learning network can learn brain connection relationships at different levels of the brain from multiple different scales to capture richer emotional information. The connection relationship learning module obtained based on the idea of ​​graph convolution can learn brain connection relationships that are more in line with brain neuroscience; the multi-scale relational learning network re-integrates EEG signals at multiple scales to obtain fused emotional features to comprehensively consider emotional information at different levels; the general feature induction network summarizes the fused emotional features obtained from emotional features of multiple source domains, and uses the Dirichlet prior network in the general feature induction network to eliminate the interference of personalized emotional features on emotion recognition, thereby obtaining purer emotional information, which can greatly improve the generalization ability.

[0056] Other features and advantages of the present invention will be described in detail in the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the following specific embodiments, they are used to explain the present invention but do not constitute a limitation of the present invention. In the accompanying drawings:

[0058] Figure 1 It is a flowchart of a cross-domain EEG emotion recognition method based on multi-scale relationship learning according to an exemplary embodiment.

[0059] Figure 2 It is a schematic diagram of the structure of a multi-scale relationship learning network and a general feature induction network according to an exemplary embodiment.

[0060] Figure 3 It is a schematic diagram of a correlation matrix according to an exemplary embodiment.

[0061] Figure 4 is a schematic diagram showing an edge matrix learning process according to an exemplary embodiment.

[0062] Figure 5a It is a schematic diagram of a confusion matrix on a SEED-V dataset according to an exemplary embodiment.

[0063] Figure 5b is a schematic diagram of a confusion matrix on an MPED data set according to an exemplary embodiment.

[0064] Figure 6a is a schematic diagram of a confusion matrix of a Chinese→French experiment according to an exemplary embodiment.

[0065] Figure 6b is a schematic diagram of a confusion matrix of a Chinese→German experiment according to an exemplary embodiment.

[0066] Figure 6c is a schematic diagram of a confusion matrix of a French→Chinese experiment according to an exemplary embodiment.

[0067] Figure 6d is a schematic diagram of a confusion matrix of a French→German experiment according to an exemplary embodiment.

[0068] Figure 6e is a schematic diagram of a confusion matrix of a German→Chinese experiment according to an exemplary embodiment.

[0069] Figure 6f is a schematic diagram of a confusion matrix of a German→French experiment according to an exemplary embodiment.

[0070] Figure 7a It is a schematic diagram of an adjacency matrix according to an exemplary embodiment.

[0071] Figure 7b is a schematic diagram of yet another adjacency matrix according to an exemplary embodiment.

[0072] Figure 8a is a schematic diagram of a result of parameter analysis according to an exemplary embodiment.

[0073] Figure 8b is a schematic diagram of another result of parameter analysis according to an exemplary embodiment.

[0074] Figure 8c is a schematic diagram of another result of parameter analysis according to an exemplary embodiment.

[0075] Figure 8d is a schematic diagram of another result of parameter analysis according to an exemplary embodiment.

[0076] Figure 8e is a schematic diagram of another parameter analysis result according to an exemplary embodiment.

[0077] Fig. 9 It is a block diagram of a cross-domain EEG emotion recognition device based on multi-scale relationship learning according to an exemplary embodiment. DETAILED DESCRIPTION

[0078] Figure 1 is a flowchart of a cross-domain EEG emotion recognition method based on multi-scale relationship learning according to an exemplary embodiment. Figure 1 As shown, the method includes the following steps.

[0079] S101, inputting the EEG signal into a multi-scale relational learning network, and using a multi-scale separation module, a connection relational learning module and a multi-scale fusion module sequentially connected in the multi-scale relational learning network to obtain a fused emotion feature; wherein the fused emotion feature includes information from multiple source domains;

[0080] It is understandable that, just as the human body is composed of cells, the numerous cells that make up the human body are gradually organized into multiple levels such as tissues, organs, and systems according to their morphology, structure, and function. The various components of each level of the system have detailed division of labor and cooperate with each other to form a more advanced system. The brain is a complex system composed of tens of billions of neurons, and in theory there should be different levels of division of labor and cooperation. Therefore, he divides the EEG signals into multiple scales to explore the division of labor and cooperation within the brain.

[0081] Optionally, Figure 2 is a schematic diagram of a structure of a multi-scale relationship learning network and a general feature induction network according to an exemplary embodiment. Figure 2 As shown, the multi-scale relationship learning network includes a multi-scale separation module (Multi-Scale Separation Module, MSS), a connection relationship learning module (Connection Relationship Learning Module, CRL) and a multi-scale fusion module (Multi-Scale Fusion Module, MSF) connected in sequence; S101 may include:

[0082] Input the EEG signal into the multi-scale separation module to obtain multiple EEG signals of different scales;

[0083] Input multiple EEG signals of different scales into the connection relationship learning module to obtain multi-scale emotion features;

[0084] The multi-scale emotion features are input into the multi-scale fusion module to obtain the fused emotion features.

[0085] It is understandable that in order to better explore the division of labor of different components in the brain, the fine-scale nodes are grouped to obtain the coarse scale in the process of scale coarsening using the multi-scale separation module. Specifically, the fine scale is grouped and coarsened by the multi-scale separation module to obtain the coarse-scale features containing higher-level information. The input data is recorded as n 0 represents the number of nodes of the input data, d represents the feature dimension, and the scale transformation matrix is The scale coarsening process can be expressed as follows:

[0086]

[0087] in, represents the jth (j=1,2,…,J) scale data obtained by coarsening, n j represents the number of nodes at the jth scale, d represents the feature dimension, and n 1 =n 0 ,n j <n j-1 (j=2,…,J).

[0088] Optionally, the EEG signal is input into a multi-scale separation module to obtain multiple EEG signals of different scales, including:

[0089] Initialize the first weight matrix;

[0090] Obtaining a scale transformation matrix according to the first weight matrix;

[0091] The electroencephalogram signal is processed according to the scale transformation matrix to obtain multiple electroencephalogram signals of different scales.

[0092] It is worth mentioning that during the grouping process, it can be found that if each group contains too many points, it will bring too much pressure to the model learning, thus learning too much redundant information. j The matrix is ​​constrained so that a node cannot be divided into multiple groups at the same time, that is, the scale transformation matrix Q j Each column of has only one element that is not 0.

[0093] Specifically, during the training process, the weight matrix can be randomly initialized And take the scale transformation matrix Then, through the first loss function l q Q j To constrain.

[0094] Optionally, the method may further include:

[0095] The scale transformation matrix is ​​constrained by the first loss function;

[0096] The first loss function is expressed as follows:

[0097]

[0098] Among them, l q represents the first loss function, Q j represents the scale transformation matrix of the jth scale, j = 1, 2, ..., J; T represents the matrix transpose, Indicates n j The unit matrix of order n j represents the number of nodes at the jth scale, ‖·‖ F represents the Frobenius norm. When j The row vectors are orthogonal, and since Q j ≥0, so there exists a permutation matrix P such that PQ j is the simplest form of the row, namely Q j Each column of has only one element that is not 0.

[0099] Optionally, multiple EEG signals of different scales are input into the connection relationship learning module to obtain multi-scale emotion features, including:

[0100] Initialize the second weight matrix;

[0101] Obtain an edge matrix according to the second weight matrix;

[0102] Obtain a sparse edge matrix according to the edge matrix and the mask matrix;

[0103] Process the sparse edge matrix to obtain the Laplacian matrix;

[0104] Multi-scale emotion features are obtained based on the Laplacian matrix and EEG signals of multiple different scales.

[0105] It is understandable that the idea of ​​graph convolution is adopted to extend CNN to high-dimensional irregular domains to learn the connection relationship between brain functional units at various scales and further extract the spatial domain information of EEG signals. At the same time, K-order Chebyshev polynomials are used as filters to simplify calculations, and sigmoid is used as the activation function to obtain the multi-scale emotion feature representation as follows:

[0106]

[0107] in, represents the emotion feature of the jth scale, sigmoid(·) represents the sigmoid activation function, represents the k-th order Chebyshev parameter vector, k = 1, 2, ..., K; Represents the Laplacian operator at the scale j The k-th order Chebyshev polynomial calculated at , λ j max represents the Laplacian matrix L of the jth scale j The maximum eigenvalue of Indicates n j ×n j The set of real matrices of dimension X j represents the EEG signal at the jth scale.

[0108] In the process of implementing graph convolution, considering that it is difficult to manually construct a sufficiently reasonable adjacency matrix, a new dynamic adjacency matrix constraint is designed to allow the network to adaptively learn more reasonable dynamic graph connections.

[0109] Figure 3 is a schematic diagram of a correlation matrix according to an exemplary embodiment. Figure 3 As shown, there is an undirected graph with n nodes and m edges. in is the set of nodes, ε=e 1 ,e 2 ,…,e m is a set of edges, then the graph Available correlation matrix express. Each column of has only two non-zero elements, representing the two vertices associated with each edge. Available graph The degree matrix Adjacency Matrix The operator ⊙ represents the Hadamard product, which is the multiplication of corresponding elements of the matrices.

[0110] Based on the above theory, the weight matrix can be randomly initialized And take As the edge matrix, through the edge matrix E j Constraints are imposed to simulate the association matrix of the brain connectivity graph at each scale. Figure 4 is a schematic diagram of an edge matrix learning process according to an exemplary embodiment. The specific process is as follows Figure 4 shown.

[0111] First, using the mask matrix E jSet the second largest number in each column to zero, and we get

[0112] Then, the second loss function Orthogonalize the rows so that Only one non-zero element is retained in each column of , and the second loss function is expressed as follows:

[0113]

[0114] Among them, l e represents the second loss function, ‖·‖ F represents the Frobenius norm.

[0115] Finally, Perform unmasking and get but represents the incidence matrix of the jth scale, and The Laplacian matrix is ​​expressed as follows:

[0116]

[0117] Among them, L j represents the Laplace matrix of the jth scale, j = 1, 2, ..., J; Indicates n j The unit matrix of order n j represents the number of nodes at the jth scale, T represents the matrix transpose, represents the unmasked sparse edge matrix at the jth scale, represents the edge mask matrix of the jth scale, mask represents the mask matrix, E j represents the edge matrix of the jth scale, A j and D j The correlation matrix The corresponding adjacency matrix and degree matrix.

[0118] Optionally, the fused sentiment feature is expressed as follows:

[0119]

[0120] in, Indicates the fusion of emotional features, Q 1 represents the scale transformation matrix of the first scale, Represents the emotional features of the first scale after feature dimension alignment.

[0121] S102, inputting the fused emotion features into the universal feature induction network, using the Dirichlet prior network in the universal feature induction network to eliminate the interference of the personalized emotion features on emotion recognition, and obtaining the emotion recognition result.

[0122] Optionally, refer to Figure 2 The general feature induction network (General Feature Induction, GFI) also includes multiple classifiers; the input ends of the multiple classifiers receive the fused emotion features, and the output ends of the multiple classifiers are connected to the input end of the Dirichlet prior network; S102 may include:

[0123] The fused emotion features are input into multiple classifiers respectively, and the multiple classifiers are constrained by a third loss function to obtain multiple emotion distribution predictions;

[0124] Multiple emotion distribution predictions are input into the Dirichlet prior network, and the Dirichlet prior network is optimized through the fourth loss function to eliminate the interference of personalized emotion characteristics on emotion recognition and obtain the emotion recognition result.

[0125] It can be understood that, assuming that the training set consists of data from s subjects, and each subject's data is regarded as a source domain, there are s source domains S 1 ,S 2 ,...,S s In the i-th source domain middle, represents the EEG signal sample of the i-th source domain, Represents the corresponding label. Using The universal emotion representation learned in the target domain is used to identify the emotion category of samples in the unknown target domain as accurately as possible.

[0126] In mathematical statistics, if the type of a distribution is known and a certain number of samples that all obey the distribution are given, the parameters of the distribution can be estimated through these samples, thereby deriving the distribution.

[0127] It is known that the Dirichlet distribution is the conjugate prior of the categorical distribution μ, that is, the categorical distribution μ can be regarded as a sample drawn from the Dirichlet distribution. Then, given multiple samples μ ( 1),μ (2) ,…,μ ( s), the Dirichlet distribution of the population can be estimated by the maximum likelihood method.

[0128] like Figure 2 As shown, GFI is divided into s+1 branches, including s classifiers and a Dirichlet prior network (DPN) to integrate emotional features Sent to all branches simultaneously.

[0129] On the other hand, s classifiers can be used to model the classification distribution of each subject's EEG signal, and s classification distributions can be obtained.

[0130]

[0131] in, It represents the classifier corresponding to the i-th source domain and represents the emotional experience of the i-th subject.

[0132] Optionally, the method further comprises:

[0133] Multiple classifiers are constrained by the third loss function, which is expressed as follows:

[0134]

[0135] Among them, l c represents the loss function, l cross (·) represents the cross entropy function, μ ( i) represents the predicted sentiment distribution corresponding to the data in the i-th source domain in the training set, i = 1, 2, ..., s, s represents the number of data in multiple source domains into which the training set is divided, Represents the label corresponding to the data of the i-th source domain in the training set;

[0136] The Dirichlet prior network is optimized by the fourth loss function, which is expressed as follows:

[0137]

[0138] Among them, l d represents the fourth loss function, λ is a scalar parameter, l dpn represents the maximum likelihood loss of the Dirichlet prior network branch, Represents the emotion recognition results corresponding to the training set.

[0139] On the other hand, DPN can be used to model the Dirichlet distribution that these classification distributions jointly obey, and use the maximum likelihood loss to guide the optimization of the DPN branch. The maximum likelihood loss of the Dirichlet prior network branch is l dpn It is expressed as follows:

[0140]

[0141] in, is the probability density function of the Dirichlet distribution, represents the C outputs of the Dirichlet prior network, represents the data of the i-th source domain in the training set, μ ( u) represents the predicted sentiment distribution corresponding to the data in the u-th source domain in the training set, and θ is the model parameter; in l dpn In the above example, since the object of optimization is the DPN branch, μ (j) Here it is regarded as a constant, and only α is back-propagated during training;

[0142] For any input X, μ (1) ,μ (2) ,…,μ (s) They represent the prediction of the emotional distribution of the current input based on the emotional experience of the s source subjects. As the overall distribution of these classification distributions, the Dirichlet distribution simulated by DPN is a summary of the emotional experience of all subjects and can reflect the true emotions contained in the current input. Under the Dirichlet distribution, the class label ω k The posterior probability is equal to the mathematical expectation of the distribution, and the emotion recognition results are expressed as follows:

[0143]

[0144] Among them, p(ω k |μ) represents the posterior probability of the class label corresponding to the kth order for the emotion distribution prediction μ, k = 1, 2, ..., K, p(μ|X; θ) represents the posterior probability of the emotion distribution prediction μ for the EEG signal X, k≤C. Therefore, the DPN branch is essentially a classifier, and the mean of its output is the final sentiment distribution prediction of the input X.

[0145] It is worth mentioning that during the training process, in order to prevent the optimization of multiple loss functions from being out of sync, the hyperparameter λ is set 1 , 2 To adjust the optimization speed to balance the above losses, the final loss function l total It can be expressed as:

[0146] l total = l c +l d +λ 1 ·l q +λ 2 ·l e ;

[0147] Among them, λ 1 , 2 is a scalar parameter, l qIt can reduce the redundant information in the scale transformation matrix, l e This allows for learning sparse brain connections. c This enables multiple classifiers in GFI to correctly simulate the sentiment distribution of different source domains. d This enables the DPN in GFI to summarize more general emotional information from these emotional distributions, so that it can also well identify the emotional category to which the EEG signals that do not belong to these emotional distributions belong.

[0148] In the above technical solution, the multi-scale relationship learning network can learn the brain connection relationships at different levels of the brain from multiple different scales to capture richer emotional information; the multi-scale separation module separates the EEG signals into multiple different scales to simulate brain functional units at different levels; the connection relationship learning module contains a new dynamic adjacency matrix constraint to learn brain connection relationships that are more in line with brain neuroscience; the multi-scale fusion module re-integrates information at multiple scales to comprehensively consider emotional information at different levels; the general feature induction network summarizes emotional features from multiple source domains to obtain purer emotional information, which can greatly improve the generalization ability of the model.

[0149] In one embodiment, an experiment is performed according to the present invention:

[0150] 1. Experimental configuration:

[0151] 1.1. Introduction to the dataset used in the experiment:

[0152] In order to fully evaluate the proposed GMSRLM method, experiments were conducted using the following 5 public EEG emotion datasets.

[0153] The SEED-V dataset contains EEG data from 16 subjects. Each subject's EEG signal was induced by 15 movie clips representing five emotions: happiness, sadness, fear, disgust, and neutrality (three clips for each emotion type). Each subject participated in the experiment three times. After extracting the DE features of the EEG signal, the EEG signal of each subject contained 1823 samples.

[0154] The MPED dataset consists of EEG signals from 30 subjects. Each subject participated in the experiment only once and needed to watch 28 videos (divided into seven emotions: happy, funny, angry, fearful, disgusted, sad, and neutral, with 4 videos for each emotion). After extracting the short-time Fourier transform (STFT) of the EEG signal, the EEG signal of each subject contained 3360 samples.

[0155] The SEED dataset uses 15 Chinese movie clips (including three emotions: happy, neutral, and sad) as stimulus materials. There are a total of 15 Chinese subjects, each of whom participated in three experiments. Only the data of 12 of the subjects were used, and the DE features of the EEG signal with a sliding window of 1s were extracted, resulting in 10,182 samples for each subject.

[0156] The SEED-GER dataset uses 18 movie clips of three emotion categories (i.e., happy, neutral, and sad) as stimulus materials. A total of 8 Germans participated in the experiment, of which 4 subjects completed 3 experiments and the other 4 subjects only completed 2 experiments. Using the DE features of the EEG signal with a sliding window of 1s, the subjects who participated in the 3 experiments each contained 10,530 samples, and the rest only contained 7,020 samples.

[0157] The SEED-FRA dataset uses 21 movie clips of three emotion categories (i.e., happy, neutral, and sad) as stimulus materials. Eight French subjects each participated in three experiments. The DE features of the EEG signal with a sliding window of 1s were also used, and 10,110 samples were obtained for each subject.

[0158] All the above datasets used a 62-channel ESI NeuroScan system to record EEG signals with a sampling rate of 1000 Hz.

[0159] 1.2 Experimental Protocol:

[0160] In order to verify the generalization of the present invention, two cross-domain experimental schemes were adopted: cross-subject and cross-cultural. For cross-subject experiments, the training data and test data came from different subjects in the same dataset. For cross-cultural experiments, the training data and test data came from subjects in different countries.

[0161] Cross-subject experiments were conducted on the SEED-V and MPED datasets. Specifically, the most commonly used leave-one-subject-out (LOSO) cross-validation strategy was adopted. That is, for each dataset, the EEG data of one subject was used as the test data in turn, and the EEG data of the remaining subjects constituted the training set.

[0162] Cross-cultural experiments were conducted on three datasets: SEED, SEED-GER, and SEED-FRA. Samples from one of the datasets were used as training data, and samples from the other two datasets were used as test data.

[0163] 1.3. Implementation details:

[0164] In the experiment, the number of scales is set to 4, and the number of nodes at each scale is n. 1 =62, n 2=31, n 3 =15, n 4 = 7. In order to ensure that the edge matrix Sufficient to represent the relationship between all nodes, the number of edges m must be met j The number of edges of the complete graph of size j is not less than Therefore, take m 1 =2048, m 2 =512, m 3 =128, m 4 =32. In addition, the order of the Chebyshev polynomial is K=5, λ=0.01, λ 1 =0.1,λ 2 =0.1.

[0165] In particular, the present invention uses the Adam optimizer for training, with a batch size of 64 and a learning rate of 10 -4 The average accuracy (ACC) and standard deviation (STD) are used as evaluation criteria in all datasets.

[0166] 2. Experimental results:

[0167] 2.1. Cross-subject experimental results within the dataset:

[0168] Cross-subject experiments were conducted on SEED-V and MPED datasets to evaluate the cross-subject generalization performance of the present invention. In order to demonstrate the superiority of the present invention, the results of 13 other emotion recognition methods were compared: support vector machine (SVM), A-LSTM, DANN, P-GCNN, BDGLS, ECLGCNN, DGCNN, IAG, GECNN, RGNN, BiDANN, BiHDM, GMSS, and PGCN. All of these 13 methods are excellent methods for emotion recognition, and the experimental protocols used are the same as those in this section. The average accuracy (ACC) and standard deviation (STD) were used as the evaluation criteria for all subjects in the dataset. The experimental results are shown in Table 1.

[0169] Table 1

[0170]

[0171]

[0172] According to the experimental results in Table 1, the performance of the present invention on both SEED-V and MPED datasets is better than other methods. On the SEED-V dataset, the average accuracy of the present invention for five emotion classifications is 72.8%, which is 1.4% better than the optimal method PGCN. On the MPED dataset, the average accuracy of the present invention for seven emotion classifications is 30.73%, which is 2.24% better than the optimal method GMSS. Obviously, the improvement of the present invention in the MPED dataset is more prominent than that in the SEED-V dataset, which shows that: when the dataset contains more EEG data of subjects, the model can summarize a wider range of general emotional features to improve the generalization ability of the model.

[0173] In order to more intuitively analyze the recognition ability of the present invention for different emotions, the confusion matrices of the two groups of experiments are plotted respectively. Figure 5a is a schematic diagram of a confusion matrix on a SEED-V data set according to an exemplary embodiment. Figure 5b FIG. 1 is a schematic diagram of a confusion matrix on an MPED data set according to an exemplary embodiment. Figure 5a and Figure 5b As shown. It can be seen that the performance of the model on SEED-V is significantly more stable and reliable than that on MPED, and it is rarely confused with other emotions. This is mainly because SEED-V has more data (each subject participated in 3 experiments) and fewer emotion categories. The results on MPED show that positive or negative emotions are more easily confused, such as "joy" and "funny", and "angry", "fear", "disgust", and "sad". In addition, most emotions will move closer to "neural" when they are difficult to identify. This may be because the emotion is not induced properly, resulting in no obvious fluctuations in the subject's emotions. In general, "disgust" is the emotion with the lowest recognition accuracy in both datasets. This may be because the induction of this emotion is difficult and it is easy to be confused with other emotions.

[0174] 2.2. Cross-dataset experimental results:

[0175] Cross-cultural experiments were conducted on three datasets, SEED, SEED-GER and SEED-FRA, to evaluate the cross-cultural generalization performance of the present invention. Similarly, the present invention was compared with the following 10 methods: SVM, A-LSTM, BDGLS, ECLGCNN, P_GCNN, IAG, GECNN, PGCN, BiDANN, RGNN. All methods used the same experimental protocol and used the average accuracy (ACC) and standard deviation (STD) of all subjects in the dataset as evaluation criteria. The experimental results are shown in Table 2. The best results in each column in Table 2 are highlighted in bold (the second best or suboptimal results can be marked with underscores). Among them, Chinese→French means that the EEG data of Chinese subjects (i.e., SEED) is used as the training set, and the EEG data of French subjects (i.e., SEED-GER) is used as the test set (Chinese→German, French→Chinese, French→German, German→Chinese, German→French, etc. and so on).

[0176] Table 2

[0177]

[0178] According to the experimental results in Table 2, the present invention performed best in all six cross-cultural experiments. For the six experiments of Chinese→French, Chinese→German, French→Chinese, French→German, German→Chinese, and German→French, the present invention was 1.59%, 2.15%, 4.24%, 4.39%, 4.65%, and 3.49% higher than the other optimal results, respectively. According to the conclusion of the author of the data set, the EEG data of French and German subjects are suitable for use as training sets, indicating that their EEG data contain more general emotional information. The results in Table 2 further illustrate that the present invention has extracted this general emotional information very well, greatly improving the generalization performance.

[0179] Figure 6a is a schematic diagram of a confusion matrix of a Chinese→French experiment according to an exemplary embodiment. Figure 6b is a schematic diagram of a confusion matrix of a Chinese→German experiment according to an exemplary embodiment. Figure 6c is a schematic diagram of a confusion matrix of a French→Chinese experiment according to an exemplary embodiment. Figure 6d is a schematic diagram of a confusion matrix of a French→German experiment according to an exemplary embodiment. Figure 6eis a schematic diagram of a confusion matrix of a German→Chinese experiment according to an exemplary embodiment. Figure 6f is a schematic diagram of a confusion matrix of a German→French experiment according to an exemplary embodiment. Figure 6a-6f is the confusion matrix of the six experiments. In terms of positive emotions, Germans and Chinese are relatively close, while Germans and French are quite different. When the EEG data of Chinese subjects are used as a training set, the recognition effects of the three emotions are relatively poor. In general, in cross-cultural emotion recognition, the recognition effects of positive and negative emotions are good, but when recognizing neutral emotions, they are easily confused with negative emotions. Because negative emotions are often not as strong as positive emotions, but they last longer and are difficult to adjust back in time, the impact of negative emotions will continue until the next video is watched. When these three data sets induce the emotions of the subjects, most of the negative emotion video clips are followed by neutral emotion video clips, so when inducing neutral emotions, they are easily affected by residual negative emotions.

[0180] 3. Discussion:

[0181] 3.1. Visualization:

[0182] To verify the edge matrix constraint loss l e The adjacency matrix learned by the present invention on the SEED-V dataset is visualized. Figure 7a is a schematic diagram of an adjacency matrix according to an exemplary embodiment. Figure 7b is a schematic diagram of another adjacency matrix according to an exemplary embodiment. Figure 7a and 7b The results are shown with and without the second loss function l. e By observing Figure 7a , it is not difficult to find that the connection weights between most points are much less than 0.1, so they can be ignored, and only a few connections are effective. Figure 7b The adjacency matrix shown in the figure has a connection weight between most points between 0.1 and 0.2, which is full of redundant information. This proves that the edge matrix constraint loss l e Under the influence of , the model does learn a relatively sparse adjacency matrix.

[0183] 3.2. Ablation study:

[0184] In order to evaluate the contribution of each module to the model performance, four groups of ablation experiments were conducted: (1) Let the scale number s = 1 to evaluate the contribution of the multi-scale separation and fusion module; (2) Let λ 1 = 0 to evaluate the scale transformation constraint loss lq Contribution; (3). Let λ 2 = 0 to evaluate the edge matrix constraint loss l e (4) Let λ = 0 to evaluate the contribution of the GFI module. Table 3 shows the results of the four groups of experiments on the SEED-V and MPED datasets.

[0185] In λ 2 = 0, the performance of the present invention on both data sets drops significantly, indicating that the edge matrix constraint loss l e The contribution to the model is very large. e This allows the model to learn a more sparse adjacency matrix, which is not only conducive to learning brain connection relationships that are more in line with brain neuroscience, but also greatly reduces the redundant information in the model. In addition, the results of the other three groups of experiments also declined compared to the original model, indicating that these modules are indispensable and helpful in improving generalization capabilities. Among them, when λ = 0, the performance on the MPED dataset dropped by 1.45%, which is much more obvious than the 0.66% of SEED-V. This further proves that EEG data containing more subjects is conducive to the GFI module to summarize more general emotional features.

[0186] Table 3

[0187]

[0188]

[0189] 3.3. Sensitivity analysis:

[0190] In order to find the most suitable value for each parameter, the influence of different parameters on the present invention can be evaluated. Figure 8a is a schematic diagram showing a result of parameter analysis according to an exemplary embodiment. Figure 8b is a schematic diagram showing another result of parameter analysis according to an exemplary embodiment, Figure 8c is a schematic diagram showing another result of parameter analysis according to an exemplary embodiment, Figure 8d is a schematic diagram showing another result of parameter analysis according to an exemplary embodiment, Figure 8e is a schematic diagram showing another result of parameter analysis according to an exemplary embodiment. Figure 8a-8e As shown: 1) Figure 8a It shows that extracting information from different layers at multiple scales is indeed beneficial to improving model performance, but the number of scales should not be too large. Because in the scale separation method of this article, there are only 3 points at the 5th scale, and even only 1 point at the 6th scale, it is difficult to extract effective information, but it is more likely to bring noise. 2) If Figure 8bAs shown in , to a certain extent, the larger the order K of the Chebyshev polynomial, the richer the frequency domain information that can be extracted, and therefore the better the effect. However, too large an order will bring more redundant information, which will weaken the model performance and reduce the training speed. Therefore, setting K = 5 is a more appropriate experimental value obtained after many attempts. 3) As Figure 8c The parameter λ shown determines the DPN loss l dpn How much role does it play in the model training process? This determines how similar the final output of the DPN branch of the model is to the output of each classifier branch. In order to balance the model performance, λ should not be set too large, otherwise l dpn In the case of absolute dominance during training, cross entropy loss will be difficult to play a role, and the model performance will drop sharply. Figure 8d As shown, λ 1 Responsible for regulating the scale transformation constraint loss l q If it is set too small, it will be difficult to constrain the node grouping during the scale coarsening process. If it is set too large, the model potential will be reduced by solidifying the grouping too early. Therefore, λ 1 Too large or too small will reduce the model performance. 5) Figure 8e As shown, λ 2 Responsible for adjusting the edge matrix constraint loss l e , so that the learned adjacency matrix can converge to a sparse matrix at the appropriate time. e Play its due role, 2 cannot be set too small. Similarly, λ 2 It should not be set too large, otherwise the adjacency matrix will solidify prematurely and reduce model performance.

[0191] Fig. 9 is a block diagram of a cross-domain EEG emotion recognition device based on multi-scale relationship learning according to an exemplary embodiment. Fig. 9 , the cross-domain EEG emotion recognition device 900 based on multi-scale relationship learning may include.

[0192] The feature extraction module 901 is used to input the EEG signal into the multi-scale relational learning network, and obtain the fused emotion feature by using the multi-scale separation module, the connection relationship learning module and the multi-scale fusion module sequentially connected in the multi-scale relational learning network; wherein the fused emotion feature includes information of multiple source domains;

[0193] The emotion recognition module 902 is used to input the fused emotion features into the general feature induction network, and use the Dirichlet prior network in the general feature induction network to eliminate the interference of personalized emotion features on emotion recognition, so as to obtain the emotion recognition result.

[0194] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0195] The preferred embodiments of the present invention are described in detail above in conjunction with the accompanying drawings. However, the present invention is not limited to the specific details in the above embodiments. Within the technical concept of the present invention, a variety of simple modifications can be made to the technical solution of the present invention, and these simple modifications all belong to the protection scope of the present invention.

[0196] It should also be noted that the various specific technical features described in the above specific embodiments can be combined in any suitable manner without contradiction. In order to avoid unnecessary repetition, the present invention will not further describe various possible combinations.

[0197] In addition, various embodiments of the present invention may be arbitrarily combined, and as long as they do not violate the concept of the present invention, they should also be regarded as the contents disclosed by the present invention.

Claims

1. A cross-domain EEG emotion recognition method based on multi-scale relationship learning, characterized in that: The method comprises: Inputting the EEG signal into a multi-scale relational learning network, and using a multi-scale separation module, a connection relational learning module and a multi-scale fusion module sequentially connected in the multi-scale relational learning network to obtain a fused emotion feature; wherein the fused emotion feature includes information from multiple source domains; The fused emotion feature is input into a universal feature induction network, and the Dirichlet prior network in the universal feature induction network is used to eliminate the interference of personalized emotion feature on emotion recognition, so as to obtain the emotion recognition result.

2. The cross-domain EEG emotion recognition method based on multi-scale relationship learning according to claim 1 is characterized in that: The electroencephalogram signal is input into the multi-scale relational learning network, and the fused emotion feature is obtained by using the multi-scale separation module, the connection relation learning module and the multi-scale fusion module connected in sequence in the multi-scale relational learning network, including: Inputting the electroencephalogram signal into the multi-scale separation module to obtain the electroencephalogram signals of multiple different scales; Inputting the EEG signals of multiple different scales into the connection relationship learning module to obtain multi-scale emotion features; The multi-scale emotion feature is input into the multi-scale fusion module to obtain the fused emotion feature.

3. The cross-domain EEG emotion recognition method based on multi-scale relationship learning according to claim 2 is characterized in that: The step of inputting the electroencephalogram signal into the multi-scale separation module to obtain the electroencephalogram signals of multiple different scales includes: Initialize the first weight matrix; Obtaining a scale transformation matrix according to the first weight matrix; The electroencephalogram signal is processed according to the scale transformation matrix to obtain the electroencephalogram signals of multiple different scales.

4. The cross-domain EEG emotion recognition method based on multi-scale relationship learning according to claim 3 is characterized in that: The method further comprises: Constraining the scale transformation matrix by a first loss function; The first loss function is expressed as follows: Among them, l q represents the first loss function, Q j represents the scale transformation matrix of the jth scale, j = 1, 2, ..., J; T represents the matrix transpose, Indicates n j The unit matrix of order n j represents the number of nodes at the jth scale, ‖·‖ F represents the Frobenius norm.

5. The cross-domain EEG emotion recognition method based on multi-scale relationship learning according to claim 2 is characterized in that: The step of inputting the EEG signals of multiple different scales into the connection relationship learning module to obtain multi-scale emotion features includes: Initialize the second weight matrix; Obtain an edge matrix according to the second weight matrix; Obtaining a sparse edge matrix according to the edge matrix and the mask matrix; Processing the sparse edge matrix to obtain a Laplacian matrix; A multi-scale emotion feature is obtained according to the Laplacian matrix and the electroencephalogram signals of multiple different scales.

6. The cross-domain EEG emotion recognition method based on multi-scale relationship learning according to claim 5 is characterized in that: The Laplacian matrix is ​​expressed as follows: Among them, L j represents the Laplace matrix of the jth scale, j = 1, 2, ..., J; Indicates n j The unit matrix of order n j represents the number of nodes at the jth scale, T represents the matrix transpose, represents the unmasked sparse edge matrix at the jth scale, represents the edge mask matrix of the jth scale, mask represents the mask matrix, E j represents the edge matrix of the jth scale; The multi-scale emotion feature is expressed as follows: in, represents the emotion feature of the jth scale, sigmoid(·) represents the sigmoid activation function, represents the k-th order Chebyshev parameter vector, k = 1, 2, ..., K; Represents the Laplacian operator at the scale j The k-th order Chebyshev polynomial calculated at , λ j max represents the Laplacian matrix L of the jth scale j The maximum eigenvalue of Indicates n j ×n j The set of real matrices of dimension X j represents the EEG signal of the jth scale; The second loss function is expressed as follows: Among them, l e represents the second loss function, which is used to Orthogonalize, ‖·‖ F represents the Frobenius norm.

7. The cross-domain EEG emotion recognition method based on multi-scale relationship learning according to claim 6 is characterized in that: The fused emotion feature is expressed as follows: in, represents the fused emotion feature, Q 1 represents the scale transformation matrix of the first scale, Represents the emotional features of the first scale after feature dimension alignment.

8. The cross-domain EEG emotion recognition method based on multi-scale relationship learning according to claim 1 is characterized in that: The universal feature induction network also includes a plurality of classifiers; the input ends of the plurality of classifiers receive the fused emotion features, and the output ends of the plurality of classifiers are connected to the input end of the Dirichlet prior network; the fused emotion features are input into the universal feature induction network, and the Dirichlet prior network in the universal feature induction network is used to eliminate the interference of personalized emotion features on emotion recognition, so as to obtain the emotion recognition result, including: Inputting the fused emotion features into the multiple classifiers respectively to obtain multiple emotion distribution predictions; The multiple emotion distribution predictions are input into the Dirichlet prior network to obtain the emotion recognition result.

9. The cross-domain EEG emotion recognition method based on multi-scale relationship learning according to claim 8, characterized in that: The method further comprises: The multiple classifiers are constrained by a third loss function, and the third loss function is expressed as follows: Among them, l c represents the loss function, l cross (·) represents the cross entropy function, μ (i) represents the emotion distribution prediction corresponding to the data in the i-th source domain in the training set, i=1,2,…,s, s represents the number of data in multiple source domains into which the training set is divided, Represents the label corresponding to the data of the i-th source domain in the training set; The Dirichlet prior network is optimized by a fourth loss function, and the fourth loss function is expressed as follows: Among them, l d represents the fourth loss function, λ is a scalar parameter, l dpn represents the maximum likelihood loss of the Dirichlet prior network branch, Indicates the emotion recognition result corresponding to the training set.

10. A cross-domain EEG emotion recognition device based on multi-scale relationship learning, characterized in that: The device comprises: A feature extraction module, used for inputting the EEG signal into a multi-scale relational learning network, and obtaining a fused emotion feature by using a multi-scale separation module, a connection relational learning module and a multi-scale fusion module sequentially connected in the multi-scale relational learning network; wherein the fused emotion feature includes information of multiple source domains; The emotion recognition module is used to input the fused emotion feature into a universal feature induction network, and utilize the Dirichlet prior network in the universal feature induction network to eliminate the interference of personalized emotion feature on emotion recognition, so as to obtain the emotion recognition result.

Images