A multi-scale relationship learning cross-domain electroencephalogram emotion recognition method and device

Abstract

The application relates to a multi-scale relationship learning cross-domain electroencephalogram emotion recognition method and device, which comprises the following steps: inputting an electroencephalogram signal into a multi-scale relationship learning network; obtaining fused emotion features by using a multi-scale separation module, a connection relationship learning module and a multi-scale fusion module which are sequentially connected in the multi-scale relationship learning network; wherein the fused emotion features comprise information of multiple source domains; inputting the fused emotion features into a general feature induction network; and obtaining an emotion recognition result by using a Dirichlet prior network in the general feature induction network to exclude the interference of personalized emotion features on emotion recognition. The above technical scheme uses the multi-scale relationship learning network to capture emotion features, can learn brain connection relationships which are more in line with brain neuroscience, comprehensively considers emotion information of multiple source domains, and induces the fused emotion features to exclude personalized emotion interference, thereby improving the generalization capability.

Description

Technical Field

[0001] This invention relates to the field of emotion recognition technology, specifically to a cross-domain EEG emotion recognition method and device based on multi-scale relationship learning. Background Technology

[0002] Emotions, as responses of organisms to specific stimuli (people, situations, or events), reflect the organism's current physiological and psychological state and are crucial in various human experiences. Humans can easily discern the emotions of others and respond appropriately (e.g., not laughing when someone is sad), but machines struggle to do so, making them appear less intelligent in many situations. Therefore, developing machines capable of recognizing human emotions is of great significance in order to enable machines to better serve humanity.

[0003] Human bodily signals (e.g., voice, facial expressions, posture) are widely used in emotion recognition due to their ease of acquisition and have been developed for many years. However, this method has a degree of subjectivity; people can manipulate their bodily signals to fake their emotions. In contrast, physiological signals such as electroencephalograms (EEG), electromyography (EMG), heart rate, and temperature are generated autonomously by the nervous system and are difficult to consciously control, thus making them more reliable. Among these, EEG signals, generated by the central nervous system, respond more promptly to emotional changes and have been proven to provide important characteristics for emotion recognition.

[0004] Electroencephalogram (EEG) signals contain a wealth of complex information, but not all of it is related to emotion. Extracting emotion-related information from these intricate EEG signals has been a persistent challenge for researchers. Some scholars have attempted to extract handcrafted features related to emotion from the time, frequency, and time-frequency domains, but these features may be too rudimentary to provide sufficient information for emotion recognition. Deep learning methods based on recurrent neural networks and convolutional neural networks can automatically learn high-level features from EEG signals, but they often focus more on the time-frequency domain and lack the capacity to learn the spatial relationships between multiple EEG channels. Research suggests that emotions may be related to certain circuits in the brain; therefore, researchers should consider the connections between different brain regions rather than considering isolated areas. However, the human brain is composed of billions of neurons, and the connections between brain regions are extremely complex. A single channel in an EEG signal cannot represent a basic unit of the brain; therefore, simple graphical structures between EEG channels cannot fully reflect the complex connections between different brain regions.

[0005] Even if a model possesses the ability to extract richer EEG emotional information, it is equally important to ensure that the extracted emotional features are universal and applicable to most people, rather than relying on the individual emotional features of a specific subject. Because cortical features from the same individual are stable, many EEG-based emotion recognition studies perform well when dependent on the subject. However, they often perform poorly when independent of the subject. This is because there are significant individual differences in EEG signals; in subject-independent EEG emotion recognition, some information with personal attributes can become interfering factors, posing a significant obstacle to emotion recognition. Methods to address subject-independent emotion recognition can be divided into domain adaptation and domain generalization. Domain adaptation methods often require target domain data to participate in training or fine-tuning the model, which is often difficult to achieve in practice. Furthermore, since the model only adapts to data from a specific target domain, the resulting model only performs well in that target domain, and therefore does not learn universal emotional features. Summary of the Invention

[0006] To address the aforementioned problems in the existing technology, this invention provides a cross-domain EEG emotion recognition method and device based on multi-scale relationship learning.

[0007] According to a first aspect of the present invention, a cross-domain EEG emotion recognition method based on multi-scale relation learning is provided, the method comprising:

[0008] EEG signals are input into a multi-scale relation learning network. The multi-scale separation module, the connection relation learning module, and the multi-scale fusion module, which are sequentially connected in the multi-scale relation learning network, are used to obtain fused emotional features. The fused emotional features include information from multiple source domains.

[0009] The fused emotion features are input into a general feature induction network, and the Dirichlet prior network in the general feature induction network is used to eliminate the interference of personalized emotion features on emotion recognition, so as to obtain the emotion recognition result.

[0010] Optionally, the step of inputting the EEG signal into a multi-scale relationship learning network, and using the multi-scale separation module, connection relationship learning module, and multi-scale fusion module sequentially connected in the multi-scale relationship learning network to obtain fused emotional features, includes:

[0011] The EEG signal is input into the multi-scale separation module to obtain multiple EEG signals at different scales;

[0012] The EEG signals at multiple different scales are input into the connectivity learning module to obtain multi-scale emotional features;

[0013] The multi-scale emotion features are input into the multi-scale fusion module to obtain the fused emotion features.

[0014] Optionally, the step of inputting the EEG signal into the multi-scale separation module to obtain multiple EEG signals at different scales includes:

[0015] Initialize the first weight matrix;

[0016] The scaling transformation matrix is ​​obtained based on the first weight matrix;

[0017] The EEG signal is processed according to the scale transformation matrix to obtain multiple EEG signals at different scales.

[0018] Optionally, the method further includes:

[0019] The scaling transformation matrix is ​​constrained by a first loss function;

[0020] The first loss function is expressed as follows:

[0021]

[0022] Among them, l q Let Q represent the first loss function. j This represents the scaling transformation matrix at the j-th scale, where j = 1, 2, ..., J; T Indicates matrix transpose. n j n-order identity matrix j Represents the number of nodes at the j-th scale, ||·|| F This represents the Frobenius norm.

[0023] Optionally, the step of inputting multiple EEG signals at different scales into the connectivity learning module to obtain multi-scale emotion features includes:

[0024] Initialize the second weight matrix;

[0025] The edge matrix is ​​obtained based on the second weight matrix;

[0026] The sparse edge matrix is ​​obtained based on the edge matrix and the mask matrix;

[0027] The sparse edge matrix is ​​processed to obtain the Laplacian matrix;

[0028] Multiscale emotional features are obtained based on the Laplace matrix and the EEG signals at multiple different scales.

[0029] Optionally, the Laplace matrix is ​​represented as follows:

[0030]

[0031] Among them, Lj Let J represent the Laplacian matrix at the j-th scale, where j = 1, 2, ..., J; n j n-order identity matrix j This represents the number of nodes at the j-th scale. T Indicates matrix transpose. Let represent the sparse edge matrix at the j-th scale after inverse masking. Let represent the edge mask matrix at the j-th scale, and mask represent the mask matrix. E j Let represent the edge matrix at the j-th scale;

[0032] The multi-scale emotion features are represented as follows:

[0033]

[0034] in, Let represent the sentiment feature at the j-th scale, and sigmoid(·) represent the sigmoid activation function. This represents the k-th order Chebyshev parameter vector. This represents the Laplacian operator at the j-th scale. The k-th order Chebyshev polynomial is calculated at point k. λ j max L represents the Laplacian matrix at the j-th scale. j The largest eigenvalue, n j ×n j The set of real matrices of dimension X j This represents the EEG signal at the j-th scale;

[0035] The second loss function is expressed as follows:

[0036]

[0037] Among them, l e This represents the second loss function, which is used to... Perform orthogonalization, ||·|| F This represents the Frobenius norm.

[0038] Optionally, the fused emotional features are represented as follows:

[0039]

[0040] in, Q represents the fused emotional feature. 1 This represents the scaling transformation matrix for the first scale. This represents the sentiment feature at the first scale after feature dimension alignment.

[0041] Optionally, the general feature induction network further includes multiple classifiers; the inputs of the multiple classifiers receive the fused emotion features, and the outputs of the multiple classifiers are connected to the input of the Dirichlet prior network; the step of inputting the fused emotion features into the general feature induction network and using the Dirichlet prior network in the general feature induction network to eliminate the interference of personalized emotion features on emotion recognition, thereby obtaining the emotion recognition result, includes:

[0042] The fused emotion features are input into the multiple classifiers respectively, and the multiple classifiers are constrained by a third loss function to obtain multiple emotion distribution predictions;

[0043] The multiple emotion distribution predictions are input into the Dirichlet prior network, and the Dirichlet prior network is optimized by a fourth loss function to eliminate the interference of personalized emotion features on emotion recognition, thereby obtaining the emotion recognition result.

[0044] Optionally, the method further includes:

[0045] The multiple classifiers are constrained by a third loss function, which is expressed as follows:

[0046]

[0047] Among them, l c Let l represent the loss function. cross (·) denotes the cross-entropy function, μ ( i) represents the predicted sentiment distribution corresponding to the data in the i-th source domain in the training set, i = 1, 2, ..., s, where s represents the number of data in the multiple source domains into which the training set is divided. This represents the label corresponding to the data in the i-th source domain of the training set;

[0048] The Dirichlet prior network is optimized using a fourth loss function, which is expressed as follows:

[0049]

[0050] Among them, l d Let λ represent the fourth loss function, where λ is a scalar parameter, and l dpn This represents the maximum likelihood loss of the Dirichlet prior network branch. This represents the emotion recognition result corresponding to the training set.

[0051] According to a second aspect of the present invention, a cross-domain EEG emotion recognition device based on multi-scale relation learning is provided, the device comprising:

[0052] The feature extraction module is used to input EEG signals into a multi-scale relation learning network, and use the multi-scale separation module, the connection relation learning module and the multi-scale fusion module connected in sequence in the multi-scale relation learning network to obtain fused emotional features; wherein, the fused emotional features include information from multiple source domains;

[0053] The emotion recognition module is used to input the fused emotion features into a general feature induction network, and use the Dirichlet prior network in the general feature induction network to eliminate the interference of personalized emotion features on emotion recognition, so as to obtain the emotion recognition result.

[0054] The technical solution provided by this invention may include the following beneficial effects:

[0055] Through the above technical solutions, the multi-scale relation learning network can learn brain connectivity at different levels from multiple different scales to capture richer emotional information. The connectivity learning module obtained based on the graph convolution concept can learn brain connectivity that is more in line with neuroscience. The multi-scale relation learning network re-fused EEG signals from multiple scales to obtain fused emotional features, so as to comprehensively consider emotional information at different levels. The general feature induction network summarizes the fused emotional features obtained from emotional features from multiple source domains. By using the Dirichlet prior network in the general feature induction network to eliminate the interference of personalized emotional features on emotion recognition, more pure emotional information can be obtained, which can greatly improve the generalization ability.

[0056] Other features and advantages of the present invention will be described in detail in the following detailed description section. Attached Figure Description

[0057] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the following detailed description to explain the invention, but do not constitute a limitation thereof. In the drawings:

[0058] Figure 1 This is a flowchart illustrating a cross-domain EEG emotion recognition method based on a multi-scale relation learning according to an exemplary embodiment.

[0059] Figure 2 This is a schematic diagram illustrating the structure of a multi-scale relation learning network and a general feature induction network according to an exemplary embodiment.

[0060] Figure 3 This is a schematic diagram of an association matrix according to an exemplary embodiment.

[0061] Figure 4 This is a schematic diagram illustrating a side matrix learning process according to an exemplary embodiment.

[0062] Figure 5a This is a schematic diagram of a confusion matrix on a SEED-V dataset according to an exemplary embodiment.

[0063] Figure 5b This is a schematic diagram of a confusion matrix on an MPED dataset according to an exemplary embodiment.

[0064] Figure 6a This is a schematic diagram of a confusion matrix for a Chinese→French experiment according to an exemplary embodiment.

[0065] Figure 6b This is a schematic diagram of a confusion matrix for a Chinese→German experiment according to an exemplary embodiment.

[0066] Figure 6c This is a schematic diagram of a confusion matrix for a French→Chinese experiment according to an exemplary embodiment.

[0067] Figure 6d This is a schematic diagram of a confusion matrix for a French→German experiment according to an exemplary embodiment.

[0068] Figure 6e This is a schematic diagram of a confusion matrix for a German→Chinese experiment according to an exemplary embodiment.

[0069] Figure 6f This is a schematic diagram of a confusion matrix for a German→French experiment according to an exemplary embodiment.

[0070] Figure 7a This is a schematic diagram of an adjacency matrix according to an exemplary embodiment.

[0071] Figure 7b This is a schematic diagram illustrating yet another adjacency matrix according to an exemplary embodiment.

[0072] Figure 8a This is a schematic diagram illustrating the result of a parameter analysis according to an exemplary embodiment.

[0073] Figure 8b This is a schematic diagram illustrating the results of another parameter analysis according to an exemplary embodiment.

[0074] Figure 8c This is a schematic diagram illustrating the results of another parameter analysis according to an exemplary embodiment.

[0075] Figure 8d This is a schematic diagram illustrating the results of another parameter analysis according to an exemplary embodiment.

[0076] Figure 8e This is a schematic diagram illustrating the results of another parameter analysis according to an exemplary embodiment.

[0077] Figure 9 This is a block diagram illustrating a cross-domain EEG emotion recognition device for multi-scale relational learning according to an exemplary embodiment. Detailed Implementation

[0078] Figure 1 This is a flowchart illustrating a cross-domain EEG emotion recognition method based on multi-scale relation learning, according to an exemplary embodiment. Figure 1 As shown, the method includes the following steps.

[0079] S101. Input the EEG signal into the multi-scale relation learning network, and use the multi-scale separation module, the connection relation learning module and the multi-scale fusion module connected in sequence in the multi-scale relation learning network to obtain the fused emotional features; wherein, the fused emotional features include information from multiple source domains;

[0080] It's understandable that, just as the human body is composed of cells, these numerous cells, based on their morphology, structure, and function, progressively form tissues, organs, and systems at multiple levels. Each level of system has its own clearly defined functions, yet they also cooperate to create more advanced systems. The brain, a complex system composed of billions of neurons, should theoretically also have different levels of division of labor and cooperation. Therefore, this study examines how he divides brainwave signals into multiple scales to explore the division of labor and cooperation within the brain.

[0081] Optionally, Figure 2 This is a schematic diagram illustrating the structure of a multi-scale relation learning network and a general feature induction network according to an exemplary embodiment, such as... Figure 2 As shown, the multi-scale relationship learning network includes a multi-scale separation module (MSS), a connection relationship learning module (CRL), and a multi-scale fusion module (MSF) connected in sequence; S101 may include:

[0082] The EEG signal is input into the multi-scale separation module to obtain multiple EEG signals at different scales;

[0083] By inputting multiple EEG signals at different scales into the connectivity learning module, multi-scale emotional features are obtained.

[0084] Multi-scale emotion features are input into the multi-scale fusion module to obtain fused emotion features.

[0085] Understandably, to better explore the division of labor among different components within the brain, the fine-scale nodes are grouped into coarse-scale nodes during the scale coarsening process using a multi-scale separation module. Specifically, the fine-scale nodes are grouped and coarsened using the multi-scale separation module to obtain coarse-scale features containing higher-level information. Let the input data be... n0 represents the number of nodes in the input data, d represents the feature dimension, and the scaling transformation matrix is ​​given by... The process of scale coarsening can be expressed as follows:

[0086]

[0087] in, This represents the j-th scale data obtained after coarsening (j = 1, 2, ..., J), where n is the number of scales. j Let d represent the number of nodes at the j-th scale, and d represent the feature dimension, where n1 = n0, n... j <n j-1 (j = 2, ..., J).

[0088] Optionally, the EEG signal is input into a multi-scale separation module to obtain multiple EEG signals at different scales, including:

[0089] Initialize the first weight matrix;

[0090] The scaling transformation matrix is ​​obtained from the first weight matrix;

[0091] The EEG signals are processed using the scaling transformation matrix to obtain EEG signals at multiple different scales.

[0092] It's worth noting that during the grouping process, it can be observed that if each group contains too many points, it will put excessive pressure on the model's learning, resulting in the learning of too much redundant information. Therefore, it is possible to adjust the Q... j The matrix Q is used to constrain a node so that it cannot be assigned to multiple groups simultaneously. j Each column contains only one non-zero element.

[0093] Specifically, during training, the weight matrix can be randomly initialized. And take the scaling transformation matrix Then, through the first loss function l q Q j Apply constraints.

[0094] Optionally, the method may further include:

[0095] The scaling transformation matrix is ​​constrained by the first loss function;

[0096] The first loss function is expressed as follows:

[0097]

[0098] Among them, l q Let Q represent the first loss function. j This represents the scaling transformation matrix at the j-th scale, where j = 1, 2, ..., J; T Indicates matrix transpose. n j n-order identity matrix j Represents the number of nodes at the j-th scale, ||·|| F This represents the Frobenius norm. When... At that time, Q j The row vectors are orthogonal, and since Q j Since ≥0, there exists a permutation matrix P such that PQ j It is the simplest form of the row vector, i.e., Q. j Each column contains only one non-zero element.

[0099] Optionally, multiple EEG signals at different scales are input into the connectivity learning module to obtain multi-scale emotional features, including:

[0100] Initialize the second weight matrix;

[0101] The edge matrix is ​​obtained based on the second weight matrix;

[0102] The sparse edge matrix is ​​obtained from the edge matrix and the mask matrix;

[0103] The sparse edge matrix is ​​processed to obtain the Laplacian matrix;

[0104] Multiscale emotional features were obtained based on the Laplace matrix and EEG signals at multiple different scales.

[0105] Understandably, the approach of graph convolution extends CNNs to high-dimensional irregular domains to learn the connections between brain functional units at various scales, further extracting spatial information from EEG signals. Simultaneously, a K-order Chebyshev polynomial is used as a filter to simplify computation, and a sigmoid function is used as the activation function, resulting in the following multi-scale emotion feature representation:

[0106]

[0107] in, Let represent the sentiment feature at the j-th scale, and sigmoid(·) represent the sigmoid activation function. Let k represent the k-th order Chebyshev parameter vector, k = 1, 2, ..., K; This represents the Laplacian operator at the j-th scale. The k-th order Chebyshev polynomial is calculated at point k. λ j max L represents the Laplacian matrix at the j-th scale. j The largest eigenvalue, n j ×n j The set of real matrices of dimension X j This represents the EEG signal at the j-th scale.

[0108] In the implementation of graph convolution, considering the difficulty in manually constructing sufficiently reasonable adjacency matrices, a new dynamic adjacency matrix constraint was designed to allow the network to adaptively learn more reasonable dynamic graph connections.

[0109] Figure 3 This is a schematic diagram of an association matrix according to an exemplary embodiment, such as... Figure 3 As shown, there is an undirected graph with n nodes and m edges. in Let ε be the set of nodes, where ε = e1, e2, ..., e m Let be the set of edges, then the graph Adaptation matrix available express. Each column contains only two non-zero elements, representing the two vertices connected by each edge. The diagram can be obtained degree matrix Adjacency Matrix The operator ⊙ represents the Hadamard product, which is the element-wise multiplication of matrices.

[0110] Based on the above theory, the weight matrix can be randomly initialized. and take As the edge matrix, through the edge matrix E j Constraints are applied to simulate the association matrix of the brain connectivity graph at each scale. Figure 4 This is a schematic diagram illustrating a side matrix learning process according to an exemplary embodiment, the specific process as follows: Figure 4 As shown.

[0111] First, using the mask matrix E j Set the second largest number in each column to zero to obtain

[0112] Then, the second loss function is applied to... Perform row orthogonalization so that Each column retains only one non-zero element, and the second loss function is expressed as follows:

[0113]

[0114] Among them, l e Let ||·|| represent the second loss function. F This represents the Frobenius norm.

[0115] Finally, for Perform inverse masking to obtain but Let represent the correlation matrix at the j-th scale, and The Laplace matrix is ​​represented as follows:

[0116]

[0117] Among them, L j Let J represent the Laplacian matrix at the j-th scale, where j = 1, 2, ..., J; n j n-order identity matrix j This represents the number of nodes at the j-th scale. T Indicates matrix transpose. Let represent the sparse edge matrix at the j-th scale after inverse masking. Let represent the edge mask matrix at the j-th scale, and mask represent the mask matrix. E j Let A represent the side matrix at the j-th scale. j and D j These are the correlation matrices. The corresponding adjacency matrix and degree matrix.

[0118] Optionally, the fused emotional features are represented as follows:

[0119]

[0120] in, Q represents the integration of emotional characteristics. 1 This represents the scaling transformation matrix for the first scale. This represents the sentiment feature at the first scale after feature dimension alignment.

[0121] S102. Input the fused emotion features into the general feature induction network, and use the Dirichlet prior network in the general feature induction network to eliminate the interference of personalized emotion features on emotion recognition, and obtain the emotion recognition result.

[0122] Optionally, refer to Figure 2The General Feature Induction (GFI) network also includes multiple classifiers; the inputs of the multiple classifiers receive the fused sentiment features, and the outputs of the multiple classifiers are connected to the input of the Dirichlet prior network; S102 may include:

[0123] The fused emotion features are input into multiple classifiers, and the multiple classifiers are constrained by a third loss function to obtain multiple emotion distribution predictions.

[0124] Multiple emotion distribution predictions are input into the Dirichlet prior network, and the Dirichlet prior network is optimized by a fourth loss function to eliminate the interference of personalized emotion features on emotion recognition, thereby obtaining the emotion recognition result.

[0125] It is understandable that, assuming the training set consists of data from s subjects, and each subject's data is considered a source domain, then there are s source domains S1, S2, ..., S... s In the i-th source domain middle, This represents the EEG signal sample from the i-th source domain. This represents the corresponding tag. It utilizes multiple tag source domains. The general emotion representation learned in the process can be used to identify the emotion category of samples in the unknown target domain as accurately as possible.

[0126] In mathematical statistics, if the type of a distribution is known and a certain number of samples that commonly follow that distribution are given, the parameters of the distribution can be estimated from these samples, thereby deriving the distribution.

[0127] Given that the Dirichlet distribution is the conjugate prior of the categorical distribution μ, meaning the categorical distribution μ can be viewed as a sample drawn from the Dirichlet distribution, then given multiple samples μ... ( 1),μ (2) ,…,μ ( In the case of s), the Dirichlet distribution of the population can be estimated using the maximum likelihood method.

[0128] like Figure 2 As shown, GFI is divided into s+1 branches, including s classifiers. Integrate emotional features with a Dirichlet prior network (DPN). They were sent to all branches simultaneously.

[0129] On the other hand, s classifiers can be used to model the classification distribution of each subject's EEG signal, resulting in s classification distributions.

[0130]

[0131] in, Let represent the classifier corresponding to the i-th source domain, and represent the emotional experience of the i-th subject.

[0132] Optionally, the method further includes:

[0133] Multiple classifiers are constrained by a third loss function, which is expressed as follows:

[0134]

[0135] Among them, l c Denotes the loss function, l cross (·) denotes the cross-entropy function, μ ( i) represents the predicted sentiment distribution corresponding to the data from the i-th source domain in the training set, where i = 1, 2, ..., s, and s represents the number of data points in the training set divided into multiple source domains. This represents the label corresponding to the data in the i-th source domain of the training set;

[0136] The Dirichlet prior network is optimized using a fourth loss function, which is expressed as follows:

[0137]

[0138] Among them, l d This represents the fourth loss function, where λ is a scalar parameter, l dpn This represents the maximum likelihood loss of the Dirichlet prior network branch. This represents the emotion recognition results corresponding to the training set.

[0139] On the other hand, DPN can be used to model the Dirichlet distribution that these classification distributions commonly follow, and maximum likelihood loss can be used to guide the optimization of the DPN branches. The maximum likelihood loss of the Dirichlet prior network branches is... dpn It is expressed as follows:

[0140]

[0141] in, Let be the probability density function of the Dirichlet distribution. This represents the C outputs of the Dirichlet prior network. μ represents the data from the i-th source domain in the training set. ( u) represents the predicted sentiment distribution corresponding to the u-th source domain data in the training set, and θ is the model parameter; in l dpnIn the middle, since the optimization target is the DPN branch, μ (j) Here it is treated as a constant, and only backpropagation is performed on α during training;

[0142] For any input X, μ (1) ,μ (2) ,…,μ (s) These represent the predicted emotion distribution of the current input based on the emotional experiences of the s source subjects. As the overall distribution of these classification distributions, the Dirichlet distribution simulated by DPN is a summary of all subjects' emotional experiences and reflects the true emotion inherent in the current input. Under the Dirichlet distribution, the class label ω... k The posterior probability is equal to the mathematical expectation of the distribution. The emotion recognition result is represented as follows:

[0143]

[0144] Where p(ω) k |μ) represents the posterior probability of the class label corresponding to the k-th order for the emotion distribution prediction μ, k = 1, 2, ..., K, and p(μ|X;θ) represents the posterior probability of the emotion distribution prediction μ for the EEG signal X. k≤C. Therefore, the DPN branch is essentially also a classifier, and the mean of its output is the final sentiment distribution prediction of the input X.

[0145] It is worth mentioning that, during training, to prevent asynchronous optimization among multiple loss functions, hyperparameters λ1 and λ2 are set to adjust and balance the optimization speed of the various losses. Therefore, the final loss function l total It can be represented as:

[0146] l total =l c +l d +λ1·l q +λ2·l e ;

[0147] Where λ1 and λ2 are scalar parameters, l q It can reduce redundant information in the scaling transformation matrix, l e This allows for the learning of sparse brain connectivity relationships. c This enables multiple classifiers in GFI to correctly simulate the sentiment distribution of different source domains. d This allows the DPN in GFI to extract more general emotional information from these emotional distributions, thus enabling it to identify the emotional category of EEG signals that do not belong to these emotional distributions.

[0148] In the above technical solution, the multi-scale relation learning network can learn brain connectivity at different levels of the brain from multiple different scales to capture richer emotional information; the multi-scale separation module separates EEG signals into multiple different scales to simulate different levels of brain functional units; the connectivity learning module includes a new dynamic adjacency matrix constraint to learn brain connectivity that is more in line with neuroscience; the multi-scale fusion module re-fusions information from multiple scales to comprehensively consider emotional information at different levels; and the general feature induction network inductively summarizes emotional features from multiple source domains to obtain purer emotional information, which can greatly improve the model's generalization ability.

[0149] In one embodiment, an experiment is conducted according to the present invention:

[0150] 1. Experimental setup:

[0151] 1.1. Introduction to the dataset used in the experiment:

[0152] To fully evaluate the proposed GMSRLM method, experiments were conducted using the following five public EEG sentiment datasets.

[0153] The SEED-V dataset contains EEG data from 16 subjects. Each subject's EEG signals were induced by 15 film clips (3 clips per emotion type) representing five emotions: happiness, sadness, fear, disgust, and neutrality. Each subject participated in the experiment three times. After extracting the DE features of the EEG signals, 1823 samples of EEG signals were obtained for each subject.

[0154] The MPED dataset consists of EEG signals from 30 subjects. Each subject participated in the experiment only once and watched 28 videos (divided into seven emotions: happiness, humor, anger, fear, disgust, sadness, and neutral, with four videos for each emotion). After extracting the Short Time Fourier Transform (STFT) of the EEG signals, 3360 samples of EEG signals were obtained for each subject.

[0155] The SEED dataset used 15 Chinese film clips (containing three emotions: happy, neutral, and sad) as stimuli. A total of 15 Chinese subjects participated, each in 3 experiments. Data from only 12 of these subjects was used, and DE features were extracted from EEG signals with a 1-second sliding window, resulting in 10,182 samples per subject.

[0156] The SEED-GER dataset used 18 film clips from three emotion categories (happiness, neutrality, and sadness) as stimuli. A total of eight Germans participated in the experiment; four completed three trials, while the other four completed only two. DE features of EEG signals with a 1-second sliding window were used. The participants who completed three trials each had 10,530 samples, while the others had only 7,020 samples each.

[0157] The SEED-FRA dataset used 21 film clips from three emotion categories (happiness, neutrality, and sadness) as stimuli. Eight French participants each participated in three experiments. DE features of EEG signals with a 1-second sliding window were used to obtain 10,110 samples per participant.

[0158] All of the above datasets were recorded using a 62-channel ESI NeuroScan system with a sampling rate of 1000 Hz.

[0159] 1.2. Experimental Protocol:

[0160] To verify the generalizability of this invention, two cross-domain experimental schemes were employed: cross-subject and cross-cultural. In the cross-subject experiment, training and test data came from different subjects within the same dataset. In the cross-cultural experiment, training and test data came from subjects in different countries.

[0161] Cross-subject experiments were conducted on the SEED-V and MPED datasets. Specifically, the most common leave-one-out-of-subject (LOSO) cross-validation strategy was employed. That is, for each dataset, the EEG data of one subject was used alternately as the test data, while the EEG data of the remaining subjects constituted the training set.

[0162] The cross-cultural experiment was conducted on three datasets: SEED, SEED-GER, and SEED-FRA. Samples from one of the datasets were used alternately as training data, while samples from the other two datasets were used as test data.

[0163] 1.3. Implementation details:

[0164] In the experiment, the number of scales was set to 4, and the number of nodes at each scale were n1 = 62, n2 = 31, n3 = 15, and n4 = 7, respectively. This was done to ensure the edge matrix... To sufficiently represent the relationships between all nodes, the number of edges m must be sufficient. j The number of edges of a complete graph at scale j, i.e. Therefore, we take m1 = 2048, m2 = 512, m3 = 128, and m4 = 32. Furthermore, the order of the Chebyshev polynomial is K = 5, λ = 0.01, λ1 = 0.1, and λ2 = 0.1.

[0165] Specifically, this invention uses the Adam optimizer for training, with a batch size of 64 and a learning rate of 10. -4 Mean accuracy (ACC) and standard deviation (STD) were used as evaluation metrics for all datasets.

[0166] 2. Experimental Results:

[0167] 2.1. Results of cross-subject experiments within the dataset:

[0168] Cross-subject experiments were conducted on the SEED-V and MPED datasets to evaluate the cross-subject generalization performance of the present invention. To demonstrate the superiority of the present invention, results were compared with 13 other emotion recognition methods: Support Vector Machine (SVM), A-LSTM, DANN, P-GCNN, BDGLS, ECLGCNN, DGCNN, IAG, GECNN, RGNN, BiDANN, BiHDM, GMSS, and PGCN. All 13 methods are excellent methods for emotion recognition, and the experimental protocol used was the same as in this section. Mean accuracy (ACC) and standard deviation (STD) were used as evaluation metrics for all subjects in the dataset. Experimental results are shown in Table 1.

[0169] Table 1

[0170]

[0171]

[0172] Based on the experimental results in Table 1, the present invention outperforms other methods on both the SEED-V and MPED datasets. On the SEED-V dataset, the present invention achieves an average accuracy of 72.8% for the five emotion classifications, which is 1.4% better than the optimal method PGCN. On the MPED dataset, the present invention achieves an average accuracy of 30.73% for the seven emotion classifications, which is 2.24% better than the optimal method GMSS. Clearly, the improvement of the present invention on the MPED dataset is more significant than that on the SEED-V dataset, indicating that including more EEG data from subjects in the dataset allows the model to generalize broader and more universal emotion features, thereby improving the model's generalization ability.

[0173] To more intuitively analyze the ability of this invention to recognize different emotions, confusion matrices for two sets of experiments were plotted. Figure 5a This is a schematic diagram of a confusion matrix on a SEED-V dataset according to an exemplary embodiment. Figure 5b This is a schematic diagram of a confusion matrix on an MPED dataset according to an exemplary embodiment, such as... Figure 5a and Figure 5b As shown, the model performs significantly better on SEED-V than on MPED, with less confusion with other emotions. This is mainly because SEED-V has a larger dataset (each subject participated in three trials) and fewer emotion categories. The results on MPED show that emotions that are both positive or both negative are easily confused, such as between "joy" and "funny," and between "angry," "fear," "disgust," and "sad." Furthermore, most emotions tend to gravitate towards "neural" when difficult to identify, possibly because the emotion induction was inadequate, resulting in no significant emotional fluctuation in the subjects. Overall, "disgust" has the lowest recognition accuracy on both datasets, likely because it is difficult to incite and easily confused with other emotions.

[0174] 2.2. Experimental results across datasets:

[0175] Cross-cultural experiments were conducted on the SEED, SEED-GER, and SEED-FRA datasets to evaluate the cross-cultural generalization performance of this invention. Similarly, this invention was compared with the following 10 methods: SVM, A-LSTM, BDGLS, ECLGCNN, P_GCNN, IAG, GECNN, PGCN, BiDANN, and RGNN. All methods used the same experimental protocol, and the mean accuracy (ACC) and standard deviation (STD) of all subjects in the datasets were used as evaluation criteria. The experimental results are shown in Table 2. The best results in each column of Table 2 are highlighted in bold (the second best or second-best results are indicated by underscores). Here, Chinese→French indicates that the EEG data of Chinese subjects (i.e., SEED) was used as the training set, and the EEG data of French subjects (i.e., SEED-GER) was used as the test set (Chinese→German, French→Chinese, French→German, German→Chinese, German→French, etc. follow the same pattern).

[0176] Table 2

[0177]

[0178] According to the experimental results in Table 2, this invention performed best in all six cross-cultural experiments. For the six experiments (Chinese→French, Chinese→German, French→Chinese, French→German, German→Chinese, and German→French), this invention outperformed other best results by 1.59%, 2.15%, 4.24%, 4.39%, 4.65%, and 3.49%, respectively. According to the dataset authors, the EEG data from French and German subjects were suitable for use as a training set, indicating that their EEG data contained more general emotional information. The results in Table 2 further demonstrate that this invention effectively extracted this general emotional information, resulting in a significant improvement in generalization performance.

[0179] Figure 6a This is a schematic diagram of a confusion matrix for a Chinese→French experiment according to an exemplary embodiment. Figure 6b This is a schematic diagram of a confusion matrix for a Chinese→German experiment according to an exemplary embodiment. Figure 6c This is a schematic diagram of a confusion matrix for a French→Chinese experiment according to an exemplary embodiment. Figure 6d This is a schematic diagram of a confusion matrix for a French→German experiment according to an exemplary embodiment. Figure 6e This is a schematic diagram of a confusion matrix for a German→Chinese experiment according to an exemplary embodiment. Figure 6f This is a schematic diagram of a confusion matrix for a German→French experiment according to an exemplary embodiment. Figures 6a-6f This is the confusion matrix for the six experimental groups. Regarding positive emotions, Germans and Chinese were relatively similar, while Germans and French differed significantly. When the EEG data of Chinese subjects was used as the training set, the recognition performance for all three emotions was relatively poor. Overall, in cross-cultural emotion recognition, the recognition performance for both positive and negative emotions was good, but neutral emotions were easily confused with negative emotions. This is because negative emotions are often less intense than positive emotions, but they last longer and are harder to recover from quickly; therefore, the influence of negative emotions can persist until the next video is watched. In these three datasets, when inducing emotions in subjects, most negative emotion video clips were followed by neutral emotion video clips, thus the induction of neutral emotions was easily influenced by residual negative emotions.

[0180] 3. Discussion:

[0181] 3.1. Visualization:

[0182] To verify the edge matrix constraint loss l e The function of this method is to visualize the adjacency matrix learned by this invention on the SEED-V dataset. Figure 7a This is a schematic diagram illustrating an adjacency matrix according to an exemplary embodiment. Figure 7b This is a schematic diagram illustrating yet another adjacency matrix according to an exemplary embodiment. Figure 7a and 7b The examples show the results with and without the second loss function l. e The adjacency matrix learned under certain conditions. This is achieved through observation. Figure 7a It's easy to see that the connection weights between the vast majority of points are much less than 0.1, and therefore can be ignored; only a small number of connections are valid. And for... Figure 7b The adjacency matrix shown contains a large amount of redundant information, with most connection weights between 0.1 and 0.2. This proves that in the edge matrix constraint loss l e Under its influence, the model did indeed learn a relatively sparse adjacency matrix.

[0183] 3.2. Ablation Study:

[0184] To evaluate the contribution of each module to the model performance, four ablation experiments were conducted: (1) setting the scale number s = 1 to evaluate the contribution of the multi-scale separation and fusion modules; (2) setting λ1 = 0 to evaluate the scale transformation constraint loss l. q (3) Let λ2 = 0 to evaluate the edge matrix constraint loss l e (4) Let λ = 0 to evaluate the contribution of the GFI module. Table 3 shows the results of the four sets of experiments on the SEED-V and MPED datasets, respectively.

[0185] When λ² = 0, the performance of this invention drops significantly on both datasets, indicating that the edge matrix constraint loss l e It made a very significant contribution to the model. e This allows the model to learn a sparser adjacency matrix, which not only facilitates learning brain connectivity relationships more consistent with neuroscience but also significantly reduces redundant information in the model. Furthermore, the results of the other three experimental groups also showed a decline compared to the original model, indicating that these modules are indispensable and helpful in improving generalization ability. Specifically, when λ=0, the performance on the MPED dataset decreased by 1.45%, far more significant than the 0.66% decrease in SEED-V, further demonstrating that including EEG data from more subjects helps the GFI module to inductively summarize more general emotional features.

[0186] Table 3

[0187]

[0188]

[0189] 3.3. Sensitivity Analysis:

[0190] To find the optimal value for each parameter, the impact of different parameters on the present invention was evaluated. Figure 8a This is a schematic diagram illustrating the result of parameter analysis according to an exemplary embodiment. Figure 8b This is a schematic diagram illustrating the results of another parameter analysis according to an exemplary embodiment. Figure 8c This is a schematic diagram illustrating the results of another parameter analysis according to an exemplary embodiment. Figure 8d This is a schematic diagram illustrating the results of another parameter analysis according to an exemplary embodiment. Figure 8e This is a schematic diagram illustrating the results of another parameter analysis according to an exemplary embodiment, the results of which are as follows: Figures 8a-8e As shown: 1) Figure 8a This indicates that extracting information from different land layers at multiple scales does indeed improve model performance, but the number of scales should not be excessive. In the scale separation method presented in this paper, the fifth scale contains only three points, and the sixth scale even contains only one point, making it difficult to extract effective information and more likely to introduce noise. 2) For example... Figure 8b As shown, to a certain extent, the larger the order K of the Chebyshev polynomial, the richer the frequency domain information can be extracted, thus resulting in better performance. However, an excessively large order introduces more redundant information, which weakens model performance and reduces training speed. Therefore, setting K=5 is a more suitable experimental value obtained after multiple trials. 3) As Figure 8c The parameter λ shown determines the DPN loss l dpn The extent to which λ plays a role in model training determines the similarity between the final output of the model's DPN branch and the outputs of each classifier branch. To balance model performance, λ should not be set too large; otherwise, l... dpn When a model has an absolute advantage during training, the cross-entropy loss becomes ineffective, leading to a sharp decline in model performance. 4) For example... Figure 8d As shown, λ1 is responsible for regulating the scaling transformation constraint loss l q The effect of λ1 is that if its value is too small, it will be difficult to constrain node grouping during the scale coarsening process; if it is too large, it will reduce the model's potential by fixing the grouping too early. Therefore, both excessively large and excessively small λ1 will reduce model performance. 5) For example Figure 8e As shown, λ2 is responsible for adjusting the edge matrix constraint loss l e This allows the learned adjacency matrix to converge into a sparse matrix in a suitable time. To make l eTo ensure it functions properly, λ2 cannot be set too small. Similarly, λ2 cannot be set too large either, otherwise the adjacency matrix will become fixed too early, thus reducing model performance.

[0191] Figure 9 This is a block diagram illustrating a cross-domain EEG emotion recognition device based on multi-scale relation learning, according to an exemplary embodiment. (Refer to...) Figure 9 The multi-scale relational learning cross-domain EEG emotion recognition device 900 may include such a device.

[0192] The feature extraction module 901 is used to input the EEG signal into the multi-scale relation learning network, and use the multi-scale separation module, the connection relation learning module and the multi-scale fusion module connected in sequence in the multi-scale relation learning network to obtain fused emotion features; wherein, the fused emotion features include information from multiple source domains;

[0193] The emotion recognition module 902 is used to input the fused emotion features into the general feature induction network, and use the Dirichlet prior network in the general feature induction network to eliminate the interference of personalized emotion features on emotion recognition, so as to obtain the emotion recognition result.

[0194] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0195] The preferred embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the specific details of the above embodiments. Within the scope of the technical concept of the present invention, various simple modifications can be made to the technical solution of the present invention, and these simple modifications all fall within the protection scope of the present invention.

[0196] It should also be noted that the various specific technical features described in the above specific embodiments can be combined in any suitable manner without contradiction. In order to avoid unnecessary repetition, the present invention will not describe the various possible combinations separately.

[0197] Furthermore, various different embodiments of the present invention can be combined in any way, as long as they do not violate the spirit of the present invention, they should also be regarded as the content disclosed by the present invention.

Claims

1. A cross-domain EEG emotion recognition method based on multi-scale relation learning, characterized in that, The method includes: EEG signals are input into a multi-scale relation learning network. The multi-scale separation module, the connection relation learning module, and the multi-scale fusion module, which are sequentially connected in the multi-scale relation learning network, are used to obtain fused emotional features. The fused emotional features include information from multiple source domains. The fused emotion features are input into a general feature induction network, and the Dirichlet prior network in the general feature induction network is used to eliminate the interference of personalized emotion features on emotion recognition, so as to obtain the emotion recognition result. The step of inputting EEG signals into a multi-scale relation learning network, and using a multi-scale separation module, a connection relation learning module, and a multi-scale fusion module sequentially connected in the multi-scale relation learning network to obtain fused emotional features includes: The EEG signal is input into the multi-scale separation module to obtain multiple EEG signals at different scales; The EEG signals at multiple different scales are input into the connectivity learning module to obtain multi-scale emotional features; The multi-scale emotion features are input into the multi-scale fusion module to obtain the fused emotion features; The step of inputting the EEG signal into the multi-scale separation module to obtain multiple EEG signals at different scales includes: Initialize the first weight matrix; The scaling transformation matrix is ​​obtained based on the first weight matrix; The EEG signal is processed according to the scale transformation matrix to obtain multiple EEG signals at different scales; The method further includes: The scaling transformation matrix is ​​constrained by a first loss function; The first loss function is expressed as follows: ； in, This represents the first loss function. Indicates the first A scaling transformation matrix for each scale. ; Indicates matrix transpose. express Unit matrix of order, Indicates the first The number of nodes at each scale Denotes the Frobenius norm; The process involves inputting multiple EEG signals at different scales into the connectivity learning module to obtain multi-scale emotional features, including: Initialize the second weight matrix; The edge matrix is ​​obtained based on the second weight matrix; The sparse edge matrix is ​​obtained based on the edge matrix and the mask matrix; The sparse edge matrix is ​​processed to obtain the Laplacian matrix; Multi-scale emotional features are obtained based on the Laplace matrix and the EEG signals at multiple different scales; The Laplace matrix is ​​represented as follows: ； in, Indicates the first Laplace matrix of scale , ; express Unit matrix of order, Indicates the first The number of nodes at each scale Indicates matrix transpose. Indicates the first A sparse edge matrix of several scales that has been demasked. , Indicates the first A side mask matrix of scales. Represents the mask matrix, , Indicates the first A side matrix of several scales; The multi-scale emotion features are represented as follows: ； in, Indicates the first Emotional characteristics at various scales This represents the sigmoid activation function. Indicates the first Chebyshev parameter vector, ; Indicates scaling the number of... Laplace operator at each scale Calculated at the location Chebyshev polynomials , Indicates the first Laplace matrix of scale The largest eigenvalue, express The set of dimensional real matrices Indicates the first Electroencephalogram (EEG) signals at various scales; The second loss function is expressed as follows: ； in, This represents the second loss function, which is used to... Perform orthogonalization. This represents the Frobenius norm.

2. The cross-domain EEG emotion recognition method based on multi-scale relation learning according to claim 1, characterized in that, The fused emotional features are represented as follows: ； in, This indicates the fused emotional features. This represents the scaling transformation matrix for the first scale. This represents the sentiment feature at the first scale after feature dimension alignment.

3. The cross-domain EEG emotion recognition method based on multi-scale relation learning according to claim 1, characterized in that, The general feature induction network further includes multiple classifiers; the input terminals of the multiple classifiers receive the fused emotion features, and the output terminals of the multiple classifiers are connected to the input terminal of the Dirichlet prior network; the process of inputting the fused emotion features into the general feature induction network, and using the Dirichlet prior network in the general feature induction network to eliminate the interference of personalized emotion features on emotion recognition, to obtain the emotion recognition result, includes: The fused emotion features are input into the multiple classifiers respectively to obtain multiple emotion distribution predictions; The multiple emotion distribution predictions are input into the Dirichlet prior network to obtain the emotion recognition results.

4. The cross-domain EEG emotion recognition method based on multi-scale relation learning according to claim 3, characterized in that, The method further includes: The multiple classifiers are constrained by a third loss function, which is expressed as follows: ； in, Denotes the loss function, Represents the cross-entropy function. Indicates the first training set Sentiment distribution prediction corresponding to data from each source domain , This indicates the number of data points in the multiple source domains into which the training set is divided. Indicates the first training set The labels corresponding to the data in each source domain; The Dirichlet prior network is optimized using a fourth loss function, which is expressed as follows: ； in, This represents the fourth loss function. For scalar parameters, This represents the maximum likelihood loss of the Dirichlet prior network branch. This represents the emotion recognition result corresponding to the training set.

5. A cross-domain EEG emotion recognition device based on multi-scale relation learning, characterized in that, The device includes: The feature extraction module is used to input EEG signals into a multi-scale relation learning network, and use the multi-scale separation module, the connection relation learning module and the multi-scale fusion module connected in sequence in the multi-scale relation learning network to obtain fused emotion features; wherein, the fused emotion features include information from multiple source domains; The emotion recognition module is used to input the fused emotion features into a general feature induction network, and use the Dirichlet prior network in the general feature induction network to eliminate the interference of personalized emotion features on emotion recognition, so as to obtain the emotion recognition result. The step of inputting EEG signals into a multi-scale relation learning network, and using a multi-scale separation module, a connection relation learning module, and a multi-scale fusion module sequentially connected in the multi-scale relation learning network to obtain fused emotional features includes: The EEG signal is input into the multi-scale separation module to obtain multiple EEG signals at different scales; The EEG signals at multiple different scales are input into the connectivity learning module to obtain multi-scale emotional features; The multi-scale emotion features are input into the multi-scale fusion module to obtain the fused emotion features; The step of inputting the EEG signal into the multi-scale separation module to obtain multiple EEG signals at different scales includes: Initialize the first weight matrix; The scaling transformation matrix is ​​obtained based on the first weight matrix; The EEG signal is processed according to the scale transformation matrix to obtain multiple EEG signals at different scales; Also includes: The scaling transformation matrix is ​​constrained by a first loss function; The first loss function is expressed as follows: ； in, This represents the first loss function. Indicates the first A scaling transformation matrix for each scale. ; Indicates matrix transpose. express Unit matrix of order, Indicates the first The number of nodes at each scale Denotes the Frobenius norm; The process involves inputting multiple EEG signals at different scales into the connectivity learning module to obtain multi-scale emotional features, including: Initialize the second weight matrix; The edge matrix is ​​obtained based on the second weight matrix; The sparse edge matrix is ​​obtained based on the edge matrix and the mask matrix; The sparse edge matrix is ​​processed to obtain the Laplacian matrix; Multi-scale emotional features are obtained based on the Laplace matrix and the EEG signals at multiple different scales; The Laplace matrix is ​​represented as follows: ； in, Indicates the first Laplace matrix of scale , ; express Unit matrix of order, Indicates the first The number of nodes at each scale Indicates matrix transpose. Indicates the first A sparse edge matrix of several scales that has been demasked. , Indicates the first A side mask matrix of scales. Represents the mask matrix, , Indicates the first A side matrix of several scales; The multi-scale emotion features are represented as follows: ； in, Indicates the first Emotional characteristics at various scales This represents the sigmoid activation function. Indicates the first Chebyshev parameter vector, ; Indicates scaling the number of... Laplace operator at each scale Calculated at the location Chebyshev polynomials , Indicates the first Laplace matrix of scale The largest eigenvalue, express The set of dimensional real matrices Indicates the first Electroencephalogram (EEG) signals at various scales; The second loss function is expressed as follows: ； in, This represents the second loss function, which is used to... Perform orthogonalization. This represents the Frobenius norm.

Images