EEG-based emotion recognition methods, devices, equipment, and media across subjects

Abstract

This application relates to a method, apparatus, device, and medium for emotion recognition across subjects using electroencephalography (EEG). The method includes: acquiring EEG data; extracting differential entropy features of multiple frequency bands from the EEG data to obtain initial features; performing feature extraction on the initial features based on gradient inversion to obtain unstructured features; performing data augmentation and structured feature extraction on the initial features to obtain structured features; fusing the unstructured features and structured features based on a multi-head self-attention mechanism to obtain fused features; training a classifier on EEG data of labeled source domains using the fused features to obtain a trained classifier; and outputting emotion recognition results based on the classifier. This invention overcomes the limitations in adapting to unlabeled source domains and capturing rich unstructured and structured information, thus improving the accuracy of emotion recognition.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method, device, equipment and medium for cross-subject emotion recognition via electroencephalography. Background Technology

[0002] The field of affective computing is rapidly developing EEG-based emotion recognition technologies. However, current research in this area primarily employs supervised learning methods, which heavily rely on high-quality labeled data for model training. This process can be time-consuming, expensive, and difficult to obtain. In contrast, the abundance of available unlabeled data offers an opportunity for semi-supervised learning (SSL), where a small amount of labeled data can be combined with a large amount of unlabeled data to build models with enhanced generalization capabilities while alleviating the burden of extensive labeling work.

[0003] A significant challenge in current semi-supervised EEG emotion recognition is developing algorithms that can effectively utilize both labeled and unlabeled data to improve model learning. However, previous semi-supervised EEG emotion recognition methods only utilized labeled data from the source domain during model training and developed domain classifiers to distinguish between labeled source and unknown target domains. These methods do not adapt well to unlabeled source domains, resulting in poor performance, especially when the source data is predominantly unlabeled. Furthermore, these methods extract features from isolated EEG channels, ignoring the complex feature representations between different EEG channels. This limitation leads to a lack of rich structural information representation during the modeling process. Summary of the Invention

[0004] The purpose of this application is to provide a method, apparatus, device, and medium for EEG-based emotion recognition across subjects, in order to overcome the limitations in adapting to unlabeled source domains and capturing rich unstructured and structured information, and to improve the accuracy of emotion recognition.

[0005] To address the aforementioned technical problems, embodiments of this application provide a cross-subject EEG emotion recognition method, comprising:

[0006] Acquire electroencephalogram (EEG) data, wherein the EEG data includes: EEG data with labeled source domains, EEG data with unlabeled source domains, and EEG data with unknown target domains;

[0007] Differential entropy features of multiple frequency bands are extracted from the electroencephalogram data to obtain initial features;

[0008] Based on gradient inversion, feature extraction is performed on the initial features to obtain unstructured features;

[0009] The initial features are augmented and structured features are extracted to obtain structured features;

[0010] Based on the multi-head self-attention mechanism, the unstructured features and the structured features are fused to obtain fused features;

[0011] The classifier is trained on the electroencephalogram data of the labeled source domain using the fusion features, and the trained classifier is obtained. The emotion recognition result is then output based on the classifier.

[0012] To address the aforementioned technical problems, embodiments of this application provide an EEG-based emotion recognition device for cross-subject responses, comprising:

[0013] An electroencephalogram (EEG) data acquisition unit is used to acquire EEG data, wherein the EEG data includes: EEG data with labeled source domains, EEG data with unlabeled source domains, and EEG data with unknown target domains;

[0014] The differential entropy feature extraction unit is used to extract differential entropy features of multiple frequency bands from the electroencephalogram data to obtain initial features;

[0015] An unstructured feature extraction unit is used to extract features from the initial features based on gradient inversion to obtain unstructured features;

[0016] The structured feature extraction unit is used to perform data augmentation and structured feature extraction on the initial features to obtain structured features;

[0017] The feature fusion unit is used to fuse the unstructured features and the structured features based on a multi-head self-attention mechanism to obtain fused features;

[0018] The classifier training unit is used to train a classifier on the electroencephalogram data of the labeled source domain using the fusion features, obtain the trained classifier, and output the emotion recognition result based on the classifier.

[0019] To solve the above-mentioned technical problems, one technical solution adopted by the present invention is to provide a computer device, including one or more processors; and a memory for storing one or more programs, such that the one or more processors implement the EEG-based cross-subject emotion recognition method described in any one of the above-mentioned methods.

[0020] To solve the above-mentioned technical problems, one technical solution adopted by the present invention is: a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the EEG-based cross-subject emotion recognition method described in any one of the above-mentioned methods.

[0021] This invention provides a method, apparatus, device, and medium for emotion recognition across subjects using electroencephalography (EEG). The method includes: acquiring EEG data, wherein the EEG data includes: EEG data with labeled source domains, EEG data with unlabeled source domains, and EEG data with unknown target domains; extracting differential entropy features of multiple frequency bands from the EEG data to obtain initial features; performing feature extraction on the initial features based on gradient inversion to obtain unstructured features; performing data augmentation and structured feature extraction on the initial features to obtain structured features; fusing the unstructured features and structured features based on a multi-head self-attention mechanism to obtain fused features; training a classifier on the labeled source domain EEG data using the fused features to obtain a trained classifier; and outputting an emotion recognition result based on the classifier. This invention overcomes limitations in adapting to unlabeled source domains and capturing rich unstructured and structured information, thus improving the accuracy of emotion recognition. Attached Figure Description

[0022] To more clearly illustrate the solutions in this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a flowchart of an implementation of the EEG-based cross-subject emotion recognition method provided in this application embodiment;

[0024] Figure 2 This is a schematic diagram illustrating the implementation process of the EEG-based cross-subject emotion recognition method provided in this application embodiment;

[0025] Figure 3 This is a sub-flowchart of the EEG-based cross-subject emotion recognition method provided in the embodiments of this application;

[0026] Figure 4 This is a sub-flowchart of the EEG-based cross-subject emotion recognition method provided in the embodiments of this application;

[0027] Figure 5 This is a sub-flowchart of the EEG-based cross-subject emotion recognition method provided in the embodiments of this application;

[0028] Figure 6 This is a sub-flowchart of the EEG-based cross-subject emotion recognition method provided in the embodiments of this application;

[0029] Figure 7 This is a schematic diagram of an emotion recognition embodiment provided in this application;

[0030] Figure 8 These are schematic diagrams illustrating the model performance under different iterations provided in the embodiments of this application;

[0031] Figure 9 This is a schematic diagram illustrating the information features provided in the embodiments of this application at different stages;

[0032] Figure 10 This is a schematic diagram of the classification results at different stages provided in the embodiments of this application;

[0033] Figure 11 This is a schematic diagram of the EEG-based emotion recognition device for cross-subjects provided in an embodiment of this application;

[0034] Figure 12 This is a schematic diagram of the computer device provided in the embodiments of this application. Detailed Implementation

[0035] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein in the specification of the application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application; the terms "comprising" and "having," and any variations thereof, in the specification, claims, and foregoing drawings of this application, are intended to cover non-exclusive inclusion. The terms "first," "second," etc., in the specification, claims, or foregoing drawings of this application are used to distinguish different objects, not to describe a particular order.

[0036] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0037] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0038] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0039] It should be noted that the EEG-based emotion recognition method across subjects provided in this application is generally executed by a computer device, and correspondingly, the EEG-based emotion recognition device across subjects is generally configured in the computer device.

[0040] Please see Figure 1 and Figure 2 , Figure 1This illustrates a specific implementation of an EEG-based emotion recognition method across subjects. Figure 2 This is a schematic diagram illustrating the implementation process of the EEG-based emotion recognition method for cross-subjects provided in this application embodiment.

[0041] It should be noted that if substantially the same result is obtained, the method of this invention is not based on... Figure 1 Limited to the order of the processes shown, this method includes the following steps:

[0042] S1: Acquire EEG data, which includes: EEG data with labeled source domains, EEG data with unlabeled source domains, and EEG data with unknown target domains.

[0043] This application embodiment is an emotion recognition method based on electroencephalography (EEG), therefore, it is necessary to first acquire EEG data. The EEG data consists of data from labeled source domains, unlabeled source domains, and unknown target domains.

[0044] Among them, electroencephalogram data of labeled source domains Includes labeled samples with corresponding emotion tags. Unlabeled source domain EEG data. Includes unlabeled samples without emotion labels. EEG data with unknown target domains. It includes samples from unknown domains that require classification. and It consists of EEG data from three fields. It is in the source domain of the tag The given sentiment label. N S N u and N t This refers to the corresponding sample size. It should be noted that during model training, sentiment label information from unlabeled source and unknown target domains is unavailable.

[0045] S2: Extract differential entropy features from multiple frequency bands of EEG data to obtain initial features.

[0046] In this embodiment, it is necessary to extract differential entropy features of multiple frequency bands from the electroencephalogram (EEG) data. Specifically, differential entropy features of five frequency bands—Delta, Theta, Alpha, Beta, and Gamma—can be extracted.

[0047] In one specific embodiment, the EEG data is divided into several 1-second segments, and differential entropy features in five frequency bands (Delta, Theta, Alpha, Beta, and Gamma) are extracted from each 1-second segment. These differential entropy features are given by 62 channels. Then, a linear dynamic system method is used to smooth each 1-second segment, representing it as a 310-dimensional feature vector (5 frequency bands × 62 channels).

[0048] S3: Based on gradient inversion, feature extraction is performed on the initial features to obtain unstructured features.

[0049] In this embodiment, a feature extractor is used to process initial features from labeled source domains, unlabeled source domains, and unknown target domains. To promote domain invariance, a gradient inversion layer is used during feature extraction to invert gradients, enabling the model to learn domain-invariant features and ensuring that features extracted from the three domains are indistinguishable. The unstructured feature is a type of unstructured EEG feature.

[0050] Please see Figure 3 , Figure 3 A specific implementation of step S3 is shown below:

[0051] S31: Flatten the initial features into a one-dimensional feature vector.

[0052] S32: Input the one-dimensional feature vector into the feature extractor and extract sample features from the one-dimensional feature vector in a gradient reversal-based manner to obtain sample features.

[0053] S33: Align the distribution offsets of sample features in different domains to obtain unstructured features.

[0054] In this embodiment, unstructured EEG features are extracted from initial features in the unstructured flow to obtain unstructured features. To address the distribution offset problem among the three domains, a multi-domain adversarial neural network is used for feature adaptation, which can make the distribution of features more consistent and reliable across different domains. Specifically, this application first flattens the extracted initial features into a one-dimensional feature vector {f1, f2, ..., fm,}, where m is the dimension of the feature; then, this one-dimensional feature vector is input into a feature extractor for sample feature extraction. This produces the corresponding features. in It is the feature dimension obtained after extraction. Further, a specific implementation of step S33 is provided:

[0055] The discriminator aligns the distribution offsets of sample features in different domains to generate discriminator loss values.

[0056] Based on the discriminator loss function, the distribution differences of sample features in different domains are optimized according to the discriminator loss value to obtain unstructured features;

[0057] In this embodiment, to align the distribution offsets of the labeled source domain sample features, unlabeled source domain sample features, and unknown target domain sample features extracted from the labeled source domain, unlabeled source domain, and unknown target domain, a discriminator with parameter θ is introduced. d The discriminator loss function is used to distinguish the source domain of sample features. This application minimizes the distribution differences among the three domains by optimizing the discriminator loss function, thereby obtaining the final unstructured features.

[0058] The discriminator loss function is:

[0059]

[0060] Where, l(x) i F is the one-hot encoded label of the input sample data. NS (X i ) represents the sample features corresponding to the source domain, θ d For discriminator parameters, The discriminator loss value is given. In this embodiment, domain adversarial training aligns the feature representation distributions in the three domains, making unstructured features more robust to domain changes and more effective for downstream tasks.

[0061] S4: Perform data augmentation and extract structured features from the initial features to obtain structured features.

[0062] In this embodiment, structured EEG features are extracted from three defined domains as structured features. Based on the initial features extracted from different frequency bands, a graph convolutional network (GCN) is constructed for spatial feature representation, and corresponding positive samples are generated through data augmentation. Then, a feature extractor is defined to characterize the structured feature representation in the input, and a contrastive loss is used to ensure that the structured features extracted from the positive samples are consistent.

[0063] Please see Figure 4 , Figure 4 A specific implementation of step S4 is shown below:

[0064] S41: Construct a graph convolutional network and convert the initial features into a spatial feature representation based on the graph convolutional network to obtain graph convolutional features.

[0065] Furthermore, a specific implementation of step S41 is provided: constructing a channel-based graph convolutional network; constructing a feature matrix based on EEG data and initial features; aggregating neighbor information of the feature matrix based on the graph convolutional network to extract spatial features of the initial features and obtain graph convolutional features.

[0066] In the graph convolutional network, a dynamic adjacency matrix is ​​constructed, and node features are extracted from the adjacency matrix based on the Chebyshev expansion of the graph Laplacian matrix. The input EEG data in the adjacency matrix is ​​subjected to a polynomial transformation to obtain graph convolutional features, where the node features are the initial features extracted on each channel.

[0067] In this embodiment, structural features are extracted to capture the structural information of EEG signals collected from multiple EEG channels. By expressing the intrinsic relationships between channels, a deeper understanding of the complex interconnections and dependencies in the EEG network can be achieved. In this embodiment, an undirected graph G = (V G E G A G ), where V G E G and A G These represent nodes, edges, and the adjacency matrix, respectively. |V G | Equals the number of brainwave channels, denoted as N G For sample data x i The input is represented as

[0068] in, It is the extracted feature of the k-th node.

[0069] Furthermore, in this embodiment of the application, a graph convolutional network is constructed to aggregate the feature matrix Ψ. i The neighbor information is used for spatial feature extraction to obtain graph convolutional features G. Specifically, in this embodiment, a channel-based graph is constructed, where node V G Corresponding to the EEG channels, node features are the initial features extracted from each channel, represented as follows:

[0070] Furthermore, embodiments of this application define a dynamic adjacency matrix A. G As shown below:

[0071]

[0072] in, and These are the initial features extracted from the j-th and k-th channels. Here, the Rectified Linear Unit (ReLU) is used as the activation function to ensure that the output of the linear operation (i.e., the weight vector w and the node distance) is linear. The dot product between the weights is non-negative, which introduces a non-linear factor and improves the model's ability to learn complex patterns. The weight vector w is learned by minimizing the GCN loss function, expressed as:

[0073]

[0074] here, and It is the node feature vector. This is the reciprocal of the corresponding adjacency value, representing the distance between two nodes. A smaller distance indicates a stronger connection between the two nodes. This reflects that nodes with stronger connections are more likely to have similar characteristics. Due to the sparse connectivity of brain networks, this embodiment introduces sparsity constraints during graph learning. In Equation 3, the adjacency matrix A... G The Frobenius norm term is used to encourage sparsity in the adjacency matrix. λ≥0 is a regularization parameter used to control the trade-off between graph learning and sparsity.

[0075] Based on the obtained adjacency matrix A G Based on the Chebyshev expansion of the graph Laplacian matrix, node features are extracted. The Chebyshev graph convolution is defined as a polynomial of degree Φ-1, expressed as:

[0076]

[0077] in, It is the Chebyshev coefficient vector, x i This is the input sample data. It takes variable L as a parameter. The order Chebyshev polynomial, L~, is represented as:

[0078]

[0079] Where, λ max It is the largest eigenvalue of the Laplace matrix. L is the identity matrix, and L is the Laplace matrix calculated as follows:

[0080] L=DA G (6);

[0081] Where D is the degree matrix. Then, the input sample data x i By transforming these polynomials, we obtain Represented as:

[0082]

[0083] in, and In the final graph G, the corresponding node representation can capture the graph's first node very well. It provides information about the nodes and offers a richer and more comprehensive graphical perspective.

[0084] S42: Generate positive samples by performing data augmentation on graph convolutional features, and extract structured features from the positive samples by using contrastive loss to obtain structured features.

[0085] In this embodiment of the application, in order to improve the distinguishability of EEG signal feature representation, graph contrastive learning (GCL) is introduced to learn a representation that is robust to transformations or enhancements of certain data, ensuring that similar instances are closer in the representation space, while dissimilar instances are pushed apart.

[0086] Please see Figure 5 , Figure 5 A specific implementation of step S42 is shown below:

[0087] S421: Generate a first enhanced graph feature and a second enhanced graph based on graph convolution features, and use the first enhanced graph feature and the second enhanced graph as positive samples.

[0088] S422: Flatten the first enhanced image feature and the second enhanced image into one-dimensional feature vectors respectively to obtain the first flattened feature vector and the second flattened feature vector.

[0089] S423: Based on the feature extractor, feature extraction is performed on the first flattened feature vector and the second flattened feature vector respectively to generate the first high-level feature vector and the second high-level feature vector.

[0090] S424: The first high-level feature vector and the second high-level feature vector are reduced by the projector to obtain the first target enhancement map feature vector and the second target enhancement map feature vector.

[0091] S425: Adjust the contrastive learning loss of the feature vectors of the first target augmentation map and the feature vectors of the second target augmentation map to output structured features.

[0092] In this embodiment, a first enhanced graph feature is generated by randomly deleting nodes with a uniform loss probability ζ% from the graph convolutional feature G. Second Enhancement Map and the first enhanced map features Second Enhancement Map As positive samples, the first augmented map features are then used. Second Enhancement Map Flattened into a one-dimensional eigenvector, the first flattened eigenvector is obtained. Second flattened eigenvector Where n is the corresponding feature dimension. Then, a feature extractor is applied to generate a high-level feature representation, resulting in the first high-level feature vector. Second high-level feature vector in, This refers to the feature dimension obtained after feature extraction. Then, projection is used to further reduce the feature dimension, generating the z-axis of the first target augmentation map feature vector and the second target augmentation map feature vector. i Second target enhancement map feature vector z j To ensure that the feature representations of two augmentation maps from the same input are consistent, a contrastive learning loss is defined. The cross-entropy loss, calculated using normalized temperature scaling, is as follows:

[0093]

[0094] Where Sim represents cosine similarity. It is a temperature parameter that adjusts the feature learning performance. Encourage maximizing z i and z j The similarity between (positive samples) and z i and z j The similarity between (negative samples) is pushed apart, and B is the batch size.

[0095] S5: Based on the multi-head self-attention mechanism, unstructured features and structured features are fused to obtain fused features.

[0096] In this embodiment, the unstructured and structured features extracted from the two parallel streams are fused. First, the extracted unstructured and structured features are concatenated, and then input into a multi-head self-attention mechanism. This fusion process generates a new feature representation that emphasizes the most distinctive features related to emotion and suppresses irrelevant information.

[0097] Please see Figure 6 , Figure 6 A specific implementation of step S5 is shown below:

[0098] S51: Connect the unstructured features and structured features to generate initial fused features, and perform a linear transformation on the initial fused features to generate three matrix features.

[0099] S52: Based on the multi-head self-attention mechanism, each matrix feature is divided into multiple subspace features, and attention is calculated for each subspace feature to obtain attention features.

[0100] S53: Connect all attention features to obtain fused features.

[0101] In this embodiment, a self-attention fusion mechanism is introduced to effectively highlight important features and assign higher weights to source domain data that is closer to the target domain data, thereby obtaining more informative feature representations. Furthermore, to ensure that the extracted features are discriminative for emotion recognition, a supervised classification component is incorporated into the model learning process.

[0102] In one specific embodiment, the extracted unstructured features and the extracted structured features are concatenated into a new feature representation, denoted as: To effectively perform feature fusion, a multi-head self-attention mechanism is employed to highlight the most salient features in the input feature representation. For example... Figure 2 As shown, in this embodiment, a linear transformation is used to generate three matrices Q, K, and V from the input. The attention weights are defined as follows:

[0103]

[0104] Furthermore, the attention mechanism is extended to the heads of three matrices. Each matrix is ​​divided into H subspaces, denoted as Q = {Q...} 1 Q H}, K = {K 1 ,…,KH},V={V 1 , ..., V H In each subspace, the calculation is performed using the attention formula, as shown below:

[0105] A h =ATT(Q h K h V h (10);

[0106] Finally, all H representations are concatenated together to obtain the final output for classification, as shown below:

[0107] MHA(Q, K, V) = Concat(A 1 A H (11);

[0108] S6: Train a classifier on EEG data of labeled source domain by fusing features to obtain the trained classifier, and output emotion recognition results based on the classifier.

[0109] In this embodiment, to ensure the interpretability of the features, a classifier is trained on the labeled source data using fused features, employing a classification loss. The feature representation and classifier are optimized simultaneously to ensure that the final feature representation is sufficiently effective for sentiment recognition.

[0110] Furthermore, an attention mechanism is used to weight the contribution of each sample during model training, assigning higher weights to samples that are more informative for emotion recognition. Specifically, based on the obtained information feature representation MHA(·), a fully connected layer φ(·) is applied to the feature representations from the labeled source domain and the unknown target domain, as shown below:

[0111]

[0112] Where B represents the batch size. For each labeled source domain sample In this embodiment, the cosine similarity between the sample and all samples from the unknown target domain is calculated as follows:

[0113]

[0114] Then to The application is then normalized. Finally, the normalized similarity weights are used to adjust the sample contributions in the multi-class cross-entropy loss function, as shown below:

[0115]

[0116] in, and Let represent the actual sentiment label and the predicted sentiment label of the source domain sample with the b-th label, respectively, and represent the total number of sentiment categories.

[0117] In this embodiment of the application, unstructured features are extracted using a feature extractor F. NS (·) is used for extraction; the structured features are extracted using the feature extractor F. S Extraction is performed using (·).

[0118] Feature Extractor F NS The layer (·) is designed with 310 neurons (input layer), 64 neurons (hidden layer 1), ReLU activation, 64 neurons (hidden layer 2), ReLU activation, and 64 neurons (output feature layer). The probability of node loss is set to 0.2, leaving 49 channels.

[0119] Feature Extractor F S (·) was designed with 245 neurons (input layer, 64 neurons (hidden layer 1), ReLU activation, 64 neurons (hidden layer 2), ReLU activation, and 64 neurons (output feature layer).

[0120] The discriminator was designed with 64 neurons (input layer), 64 neurons (hidden layer 1), ReLU activation, a loss layer, 64 neurons (hidden layer 2), 2 neurons (output layer) or 3 neurons (output layer), and Softmax activation.

[0121] Please see Figures 7 to 10 , Figure 7 This is a schematic diagram of an emotion recognition embodiment provided in this application; Figure 8 These are schematic diagrams illustrating the model performance under different iterations provided in the embodiments of this application; Figure 9 This is a schematic diagram illustrating the information features provided in the embodiments of this application at different stages; Figure 10 This is a schematic diagram of the classification results at different stages provided in the embodiments of this application.

[0122] To evaluate the effectiveness of the embodiments of this application, experiments were conducted on two publicly available emotional EEG databases: the SEED database and the SEED-IV database. In the SEED database, 15 film clips were used to induce three emotions (negative, neutral, and positive), and simultaneous EEG signals of 15 subjects (7 men and 8 women) in different emotional states were recorded using a 62-channel ESI Neuroscan system. In the SEED-IV database, 24 film clips were used to induce four emotions (pleasure, sadness, fear, and neutral), and simultaneous EEG signals of 15 subjects (7 men and 8 women) in different emotional states were recorded using a 62-channel ESI Neuroscan system. Each subject participated in three different experiments, each including 24 different film clips.

[0123] To ensure fair comparison with other studies on these two benchmark databases, embodiments of this application also use pre-computed differential entropy features as input to the model. Specifically, for each trial, the EEG data is divided into several 1-second segments, and DE features in five frequency bands (62 channels) are extracted from each 1-second segment. Then, each 1-second segment is smoothed using linear dynamic systems methods [9],

[39] , representing it as a 310-dimensional feature vector (5 frequency bands × 62 channels).

[0124] The embodiments of this application employ a leave-one-subjects cross-validation protocol with incomplete labeling. and These represent labeled source domains, unlabeled source domains, and unknown target domains, respectively. For a total of 15 subjects, 14-N subjects belong to... And N subjects belong to For each subject, the total number of trials is called M. and These are the classification loss, GCN loss, GCL loss, and discriminator loss, given by Equations 14, 3, 8, and 1 above.

[0125] like Figure 7As shown, this application employs a leave-one-out crossover experimental protocol with incomplete labeling. Specifically, this application uses 14 participants as the source domain and the remaining participant as the target domain. For the source domain, there is one unlabeled participant (unlabeled source domain). ), and the remaining 14-N labeled subjects (labeled source domains) The data from the remaining subject was considered as the target domain for testing the model. This application repeats the experiment 15 times until each subject is considered as a target domain. For each iteration, the unlabeled source domain is selected based on the target subject's ID, and then the mean accuracy and standard deviation are calculated as the final model performance. It is worth noting that the label information for the unlabeled source and target domains is unknown during model training. Furthermore, to measure the model's stability under different incomplete labeling conditions, this application adjusts the value from 1 to 13.

[0126] During the model training process, unlabeled source domains were initially excluded. Only source domains with tags and unknown target domain Participating in the pre-E t The training proceeds in the next iteration. At this point, the discriminator's output layer has two neurons that perform binary classification on the unlabeled source domain and its surrounding domain. Once the model reaches a certain level of stability, the unlabeled source domain will be classified... Added during training, the discriminator's output layer has three neurons, for... and The domain is classified into three categories. It is worth noting that only labeled data from the source domain is used throughout the process to calculate the cross-entropy loss in the classifier.

[0127] During model training, the embodiments of this application, after a certain number of iterations (denoted as E), t Then, unlabeled source domains are introduced into the training process. An evaluation was conducted in this application embodiment to study different E... t The impact of the value on model performance. E t The value changes from 0 (starting) to 100 (maximum number of iterations), and the corresponding performance results on the SEED and SEED-IV databases are as follows: Figure 8 As shown in (a) and (b), the results clearly demonstrate that after the model reaches a certain level of stability, better performance can be achieved by introducing U during training, rather than including it at the beginning (E). t =0). This is because introducing noise at the beginning of the training process can disrupt the initialization phase of model learning.

[0128] To gain a deeper understanding of the learning process, embodiments of this application employ the t-distributed random neighbor embedding (t-SNE) algorithm to compare the obtained feature representations at different stages. This analysis provides valuable insights into the learning dynamics of the model. Specifically, embodiments of this application visualize the obtained fused features at different learning stages. Figure 9 ) and the final classification results ( Figure 10 The visualizations show the model before training, at the 30th training epoch, and the final model after training. By examining these visualizations, the embodiments of this application vividly depict the evolution and enhancement of feature representation and classification performance. These visualizations demonstrate a significant improvement in inter-class separability, meaning that the distinction between different classes is more pronounced. Simultaneously, intra-class variability is minimized, resulting in tighter clustering of samples belonging to the same class. Through this visual examination, the embodiments of this application observe a clear trend in the model's ability to distinguish different classes, with the learned feature representations becoming increasingly unique and discriminative. The reduction in intra-class variability ensures that samples within the same class are closer together, enhancing the model's ability to accurately classify them. This visual evidence highlights the model's ability to learn meaningful and discriminative features, enabling it to finely distinguish classes throughout the learning process and achieve enhanced classification performance. Specifically, in... Figure 9 and Figure 10 In the image, different shades of gray correspond to different emotions, such as... Figure 10 In (c), the topmost group corresponds to neutral emotions, the bottom left group corresponds to negative emotions, and the bottom right group corresponds to positive emotions.

[0129] In this embodiment, electroencephalogram (EEG) data is acquired, including: EEG data with labeled source domains, EEG data with unlabeled source domains, and EEG data with unknown target domains. Differential entropy features of multiple frequency bands are extracted from the EEG data to obtain initial features. Based on gradient inversion, feature extraction is performed on the initial features to obtain unstructured features. Data augmentation and structured feature extraction are performed on the initial features to obtain structured features. Based on a multi-head self-attention mechanism, the unstructured features and the structured features are fused to obtain fused features. A classifier is trained on the labeled source domain EEG data using the fused features to obtain a trained classifier, and an emotion recognition result is output based on the classifier. This embodiment overcomes the limitations in adapting to unlabeled source domains and capturing rich unstructured and structured information, thus improving the accuracy of emotion recognition.

[0130] Furthermore, the embodiments of this application can be applied to emotion recognition and emotion monitoring, psychological health auxiliary diagnosis, emotion-driven human-computer interaction, and user emotion analysis and market research.

[0131] Emotion Recognition and Sentiment Monitoring: Embodiments of this application can be applied to the fields of emotion recognition and emotion monitoring to help identify human emotional states and changes in emotion. For example, in applications such as human-computer interaction, virtual reality, and games, this method can be used to monitor the user's emotional state in real time to achieve emotionally intelligent interaction and emotion perception.

[0132] Assisted Diagnosis in Mental Health: Emotion recognition has important applications in the field of mental health. By analyzing electroencephalogram (EEG) signals and identifying emotional states, it can assist psychologists or clinicians in the diagnosis and treatment of mental illnesses. For example, in the diagnosis of mental disorders such as depression and anxiety, this method can provide objective biomarkers to assist doctors in assessment and treatment decisions.

[0133] Emotion-Driven Human-Computer Interaction: Emotion recognition technology can enable emotion-driven experiences and personalized services in human-computer interaction. By recognizing the user's emotional state in real time, the system can provide corresponding feedback and support based on the user's emotional needs and preferences. For example, in fields such as virtual assistants, smart speakers, and emotionally intelligent robots, this method can be used to improve user experience and provide personalized emotional support.

[0134] User Sentiment Analysis and Market Research: Sentiment recognition has wide applications in market research and user sentiment analysis. By analyzing users' emotional states in specific situations, it is possible to obtain emotional feedback on products, services, or advertisements. This information is of great significance for corporate decision-making and market strategy formulation. For example, in areas such as advertising evaluation, product improvement, and user research, this method can help companies understand user emotional needs and market feedback.

[0135] Please refer to Figure 11 As a response to the above Figure 1 The implementation of the method shown in this application provides an embodiment of an EEG-based cross-subject emotion recognition device, which is similar to... Figure 1 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0136] like Figure 11 As shown, the EEG-based emotion recognition device across subjects in this embodiment includes: an EEG data acquisition unit 71, a differential entropy feature extraction unit 72, an unstructured feature extraction unit 73, a structured feature extraction unit 74, a feature fusion unit 75, and a classifier training unit 76, wherein:

[0137] The electroencephalogram (EEG) data acquisition unit 71 is used to acquire EEG data, which includes: EEG data with labeled source domains, EEG data with unlabeled source domains, and EEG data with unknown target domains.

[0138] The differential entropy feature extraction unit 72 is used to extract differential entropy features of multiple frequency bands from EEG data to obtain initial features;

[0139] The unstructured feature extraction unit 73 is used to extract features from the initial features based on gradient inversion to obtain unstructured features;

[0140] The structured feature extraction unit 74 is used to perform data augmentation and structured feature extraction on the initial features to obtain structured features;

[0141] The feature fusion unit 75 is used to fuse unstructured features and structured features based on a multi-head self-attention mechanism to obtain fused features;

[0142] The classifier training unit 76 is used to train a classifier on EEG data of labeled source domain by fusing features, obtain the trained classifier, and output emotion recognition results based on the classifier.

[0143] Furthermore, the unstructured feature extraction unit 73 includes:

[0144] The initial feature flattening unit is used to flatten the initial features into a one-dimensional feature vector;

[0145] The sample feature extraction unit is used to input a one-dimensional feature vector into the feature extractor and extract sample features from the one-dimensional feature vector in a gradient inversion manner to obtain sample features.

[0146] Alignment units are used to align the distribution offsets of sample features in different domains to obtain unstructured features.

[0147] Furthermore, the sample features include labeled source domain sample features, unlabeled source domain sample features, and unknown target domain sample features, and the alignment unit includes:

[0148] The discriminator loss calculation unit is used to align the distribution offset of sample features in different domains using a discriminator and generate discriminator loss values.

[0149] The distribution difference optimization unit is used to optimize the distribution difference of sample features in different domains based on the discriminator loss value according to the discriminator loss function, so as to obtain unstructured features;

[0150] The discriminator loss function is:

[0151]

[0152] Where, l(x) i F is the one-hot encoded label of the input sample data. NS (X i ) represents the sample features corresponding to the source domain, θd For discriminator parameters, The discriminant loss value is given.

[0153] Furthermore, the structured feature extraction unit 74 includes:

[0154] The graph convolutional feature generation unit is used to construct a graph convolutional network and convert the initial features into a spatial feature representation based on the graph convolutional network to obtain graph convolutional features.

[0155] The structured feature generation unit is used to generate positive samples by performing data augmentation on graph convolutional features, and extract structured features from the positive samples by using contrastive loss to obtain structured features.

[0156] Furthermore, the graph convolutional feature generation unit includes:

[0157] Graph convolutional network building unit, used to build channel-based graph convolutional networks;

[0158] The feature matrix construction unit is used to construct a feature matrix based on EEG data and initial features;

[0159] The spatial feature extraction unit is used to extract the spatial features of the initial features based on the neighbor information of the aggregated feature matrix of the graph convolutional network, so as to obtain the graph convolutional features.

[0160] In the graph convolutional network, a dynamic adjacency matrix is ​​constructed, and node features are extracted from the adjacency matrix based on the Chebyshev expansion of the graph Laplacian matrix. The input EEG data in the adjacency matrix is ​​subjected to a polynomial transformation to obtain graph convolutional features, where the node features are the initial features extracted on each channel.

[0161] Furthermore, the structured feature generation unit includes:

[0162] The data augmentation unit is used to generate a first augmented graph feature and a second augmented graph based on graph convolution features, and uses the first augmented graph feature and the second augmented graph as positive samples.

[0163] The graph feature flattening unit is used to flatten the first enhanced graph feature and the second enhanced graph into one-dimensional feature vectors, respectively, to obtain the first flattened feature vector and the second flattened feature vector;

[0164] The flattening feature extraction unit is used to extract features from the first flattened feature vector and the second flattened feature vector based on the feature extractor, and generate the first high-level feature vector and the second high-level feature vector.

[0165] The feature vector reduction unit is used to reduce the first high-level feature vector and the second high-level feature vector respectively through the projector to obtain the first target enhancement map feature vector and the second target enhancement map feature vector.

[0166] The contrastive learning loss adjustment unit is used to adjust the contrastive learning loss of the feature vectors of the first target augmentation map and the feature vectors of the second target augmentation map to output structured features.

[0167] Furthermore, the feature fusion unit 75 includes:

[0168] The matrix feature generation unit is used to connect unstructured features and structured features to generate initial fused features, and then perform a linear transformation on the initial fused features to generate three matrix features.

[0169] The attention computation unit is used to divide each matrix feature into multiple subspace features based on a multi-head self-attention mechanism, and to perform attention computation on each subspace feature to obtain attention features.

[0170] The feature connection unit is used to connect all attention features to obtain fused features.

[0171] To address the aforementioned technical problems, embodiments of this application also provide a computer device. Please refer to [link / reference needed]. Figure 12 , Figure 12 This is a basic structural block diagram of the computer device in this embodiment.

[0172] Computer device 8 includes a memory 81, a processor 82, and a network interface 83 that are interconnected via a system bus. It should be noted that only a computer device 8 with these three components (memory 81, processor 82, and network interface 83) is shown in the figure; however, it should be understood that it is not required to implement all the shown components, and more or fewer components can be implemented alternatively. Those skilled in the art will understand that the computer device described here is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0173] Computer devices can include desktop computers, laptops, handheld computers, and cloud servers. These devices allow for human-computer interaction with users through keyboards, mice, remote controls, touchpads, or voice-activated devices.

[0174] The memory 81 includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 81 may be an internal storage unit of the computer device 8, such as the hard disk or memory of the computer device 8. In other embodiments, the memory 81 may also be an external storage device of the computer device 8, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device 8. Of course, the memory 81 may also include both the internal storage unit and its external storage device of the computer device 8. In this embodiment, the memory 81 is typically used to store the operating system and various application software installed on the computer device 8, such as the program code of the EEG-based emotion recognition method for subjects. In addition, the memory 81 can also be used to temporarily store various types of data that have been output or will be output.

[0175] In some embodiments, processor 82 may be a central processing unit (CPU), controller, microcontroller, microprocessor, or other data processing chip. This processor 82 is typically used to control the overall operation of the computer device 8. In this embodiment, processor 82 is used to run program code stored in memory 81 or process data, for example, to run the program code of the above-described EEG-based cross-subject emotion recognition method to implement various embodiments of the EEG-based cross-subject emotion recognition method.

[0176] The network interface 83 may include a wireless network interface or a wired network interface, which is typically used to establish a communication connection between the computer device 8 and other electronic devices.

[0177] This application also provides another embodiment, namely, providing a computer-readable storage medium storing a computer program that can be executed by at least one processor to cause the at least one processor to perform the steps of the above-described EEG-based cross-subject emotion recognition method.

[0178] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods of the various embodiments of this application.

[0179] Obviously, the embodiments described above are only some embodiments of this application, not all embodiments. The accompanying drawings show preferred embodiments of this application, but do not limit the patent scope of this application. This application can be implemented in many different forms; rather, the purpose of providing these embodiments is to provide a more thorough and comprehensive understanding of the disclosure of this application. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing specific embodiments, or make equivalent substitutions for some of the technical features. Any equivalent structures made using the content of this application's specification and drawings, directly or indirectly applied to other related technical fields, are similarly within the scope of patent protection of this application.

Claims

1. A cross-subject EEG emotion recognition method, characterized in that, include: Acquire electroencephalogram (EEG) data, wherein the EEG data includes: EEG data with labeled source domains, EEG data with unlabeled source domains, and EEG data with unknown target domains; Differential entropy features of multiple frequency bands are extracted from the electroencephalogram data to obtain initial features; Based on gradient inversion, feature extraction is performed on the initial features to obtain unstructured features; The initial features are augmented and structured features are extracted to obtain structured features; Based on the multi-head self-attention mechanism, the unstructured features and the structured features are fused to obtain fused features; A classifier is trained on the electroencephalogram data of the labeled source domain using the fusion features to obtain the trained classifier, and the emotion recognition result is output based on the classifier. The gradient inversion-based method extracts features from the initial features to obtain unstructured features, including: Flatten the initial features into a one-dimensional feature vector; The one-dimensional feature vector is input into the feature extractor, and sample features are extracted from the one-dimensional feature vector in a gradient inversion manner to obtain sample features; The distribution offsets of the sample features in different domains are aligned to obtain the unstructured features; The sample features include labeled source domain sample features, unlabeled source domain sample features, and unknown target domain sample features. Aligning the distribution offsets of the sample features in different domains to obtain the unstructured features includes: The discriminator aligns the distribution offsets of the sample features in different domains using a discriminator to generate a discriminator loss value. The unstructured features are obtained by optimizing the distribution differences of the sample features in different domains based on the discriminator loss function and the discriminator loss value. The discriminator loss function is as follows: Where, l(x) i F is the one-hot encoded label of the input sample data. NS (X i ) represents the sample features corresponding to the source domain, θ d For discriminator parameters, The discriminant loss value is given.

2. The EEG-based cross-subject emotion recognition method according to claim 1, characterized in that, The process of performing data augmentation and structured feature extraction on the initial features to obtain structured features includes: Construct a graph convolutional network, and based on the graph convolutional network, convert the initial features into a spatial feature representation to obtain graph convolutional features; Positive samples are generated by performing data augmentation on the graph convolutional features, and structured features are extracted from the positive samples by using contrastive loss to obtain the structured features.

3. The EEG-based emotion recognition method across subjects according to claim 2, characterized in that, The construction of a graph convolutional network, and the conversion of the initial features into a spatial feature representation based on the graph convolutional network to obtain graph convolutional features, includes: Construct the channel-based graph convolutional network; A feature matrix is ​​constructed based on the electroencephalogram data and the initial features; The graph convolutional network aggregates the neighbor information of the feature matrix to extract the spatial features of the initial features, thereby obtaining the graph convolutional features. In the graph convolutional network, a dynamic adjacency matrix is ​​constructed, and node features are extracted from the adjacency matrix based on the Chebyshev expansion of the graph Laplacian matrix. The input EEG data in the adjacency matrix is ​​subjected to a polynomial transformation to obtain the graph convolutional features, wherein the node features are the initial features extracted on each channel.

4. The EEG-based emotion recognition method across subjects according to claim 2, characterized in that, The process involves generating positive samples by performing data augmentation on the graph convolutional features, and extracting structured features from the positive samples using a contrastive loss method. The structured features include: A first enhanced graph feature and a second enhanced graph are generated based on the graph convolution features, and the first enhanced graph feature and the second enhanced graph are used as the positive samples. The first enhanced image feature and the second enhanced image are flattened into one-dimensional feature vectors respectively to obtain the first flattened feature vector and the second flattened feature vector; Based on the feature extractor, feature extraction is performed on the first flattened feature vector and the second flattened feature vector to generate a first high-level feature vector and a second high-level feature vector; The first high-level feature vector and the second high-level feature vector are reduced by the projector to obtain the first target enhancement map feature vector and the second target enhancement map feature vector. The contrastive learning loss of the first target augmentation map feature vector and the second target augmentation map feature vector is adjusted to output the structured features.

5. The EEG-based cross-subject emotion recognition method according to any one of claims 1 to 4, characterized in that, The multi-head self-attention mechanism fuses the unstructured features and the structured features to obtain fused features, including: The unstructured features and the structured features are connected to generate initial fused features, and the initial fused features are linearly transformed to generate three matrix features; Based on the multi-head self-attention mechanism, each matrix feature is divided into multiple subspace features, and attention is calculated for each subspace feature to obtain attention features. All the attention features are concatenated to obtain the fused features.

6. A brainwave-based emotion recognition device for subjects, characterized in that, include: An electroencephalogram (EEG) data acquisition unit is used to acquire EEG data, wherein the EEG data includes: EEG data with labeled source domains, EEG data with unlabeled source domains, and EEG data with unknown target domains; The differential entropy feature extraction unit is used to extract differential entropy features of multiple frequency bands from the electroencephalogram data to obtain initial features; An unstructured feature extraction unit is used to extract features from the initial features based on gradient inversion to obtain unstructured features; The structured feature extraction unit is used to perform data augmentation and structured feature extraction on the initial features to obtain structured features; The feature fusion unit is used to fuse the unstructured features and the structured features based on a multi-head self-attention mechanism to obtain fused features; The classifier training unit is used to train a classifier on the electroencephalogram data of the labeled source domain using the fusion features, obtain the trained classifier, and output the emotion recognition result based on the classifier. The unstructured feature extraction unit includes: An initial feature flattening unit is used to flatten the initial feature into a one-dimensional feature vector. The sample feature extraction unit is used to input the one-dimensional feature vector into the feature extractor and extract sample features from the one-dimensional feature vector in a gradient inversion manner to obtain sample features. Alignment units are used to align the distribution offsets of the sample features in different domains to obtain the unstructured features. The sample features include labeled source domain sample features, unlabeled source domain sample features, and unknown target domain sample features. The alignment unit includes: The discriminator loss calculation unit is used to align the distribution offset of the sample features in different domains using a discriminator to generate a discriminator loss value. The distribution difference optimization unit is used to optimize the distribution difference of the sample features in different domains based on the discriminator loss value according to the discriminator loss function, so as to obtain the unstructured features; The discriminator loss function is as follows: Where, l(x) i F is the one-hot encoded label of the input sample data. NS (X i ) represents the sample features corresponding to the source domain, θ d For discriminator parameters, The discriminant loss value is given.

7. A computer device, characterized in that, It includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the EEG-based emotion recognition method for subjects as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the EEG-based emotion recognition method for subjects as described in any one of claims 1 to 5.

Images