Cross-culture EEG emotion recognition method based on multi-source multi-target domain similarity
By adopting the multi-source and multi-objective domain similarity method in cross-cultural EEG emotion recognition, dynamically adjusting the weight of each domain, the problem of different cultural and individual differences in cross-cultural emotion recognition is solved, and higher adaptability and accuracy are achieved.
Patent Information
- Application Number
- CN202510164182.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-02-14
- Publication Date
- 2025-06-03
AI Technical Summary
Existing cross-cultural EEG emotion recognition technology is difficult to effectively deal with emotional expression differences between different cultures and individual differences within cultures, resulting in limited model performance.
A cross-cultural EEG emotion recognition method based on the similarity of multi-source and multi-target domains is adopted to treat different individuals in the source culture and target culture as multiple source domains and target domains. By training a model that includes feature extractors, multiple emotion classifiers, source domain classifiers and target domain classifiers, the similarity between the domains is calculated and the weights are dynamically adjusted to alleviate the distribution differences.
It significantly improves the adaptability and accuracy of cross-cultural EEG emotion recognition, can effectively deal with differences between different cultures and individuals, and improves the generalization ability of the model in cross-cultural scenarios.
Smart Images
Figure CN120086685A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a technology in the field of neural network applications, specifically a cross-cultural EEG emotion recognition method based on multi-source and multi-target domain similarity. Background Art
[0002] In cross-cultural emotion recognition, a key challenge in the emotion classification of electroencephalogram (EEG) signals is the difference in emotion expression between different cultures. Existing domain adaptation (DA) techniques partially address the problem of cross-cultural feature distribution differences by aligning the data features of the source domain and the target domain. However, the existing techniques do not sufficiently partition the domains and ignore the important influence of individual differences within the same culture, resulting in limited model performance. Summary of the Invention
[0003] In view of the above deficiencies of the existing technology, the present invention proposes a cross-cultural EEG emotion recognition method based on multi-source and multi-target domain similarity, regarding different individuals in the source culture as multiple source domains and different individuals in the target culture as multiple target domains, and fully considering cross-cultural and intra-cultural individual differences.
[0004] The present invention is implemented through the following technical solutions:
[0005] The present invention relates to a cross-cultural EEG emotion recognition method based on multi-source and multi-target domain similarity. By training a cross-cultural EEG emotion recognition model including a feature extractor G, multiple emotion category classifiers a source domain classifier C for calculating the similarity between each source domain and the target domain src and a target domain classifier C for calculating the similarity between target domains tgt in the offline stage, cross-cultural EEG emotion recognition is performed in real time in the online stage.
[0006] The method specifically includes:
[0007] Step 1: Obtain features through the feature extractor G according to the input source domain the main target domain T 0 and the auxiliary target domain .
[0008] Step 2: Apply the maximum mean discrepancy (MMD) loss to minimize the distribution difference between each source domain S i and the target domain T 0 .
[0009] The maximum mean discrepancy (MMD) loss is: Where: M S is the number of source domains, is the feature of the source domain S i . The main target domain T 0 Features; Maximum mean difference is a Gaussian kernel function used to measure the similarity between samples x and y.
[0010] Step 3: Use respective emotion classifiers Processing from source domain S i Features And get the corresponding label classification loss.
[0011] The classification loss of the label classifier is: in: The source domain S i Data and sentiment labels, M S is the number of source domains, The source domain S i feature, For sentiment classifier Emotion classification results of source domain features, E is the cross entropy loss.
[0012] Step 4: Pass all M S The label classifier processes the main target domain T 0 Features extracted from Generate M S Class label predictions.
[0013] The target domain alignment loss of the label classifier is: Among them: through predict The pseudo-label of a feature is defined as R N , N is the number of emotion label types, matrix for The connection is obtained, α is a hyperparameter.
[0014] Step 5: Train the source domain classifier C src And calculate the source domain classification loss Specifically: in: The source domain S i The samples and their source domain labels, The source domain S i feature, is the source domain classifier C src The domain classification result of the source domain features, E is the cross entropy loss.
[0015] Make Csrc Can identify the source domain of the given data. When features are input into C src , since these target domain features do not correspond to any specific source domain, the source domain label prediction generated by the classifier indicates which source domain the target features are closest to. These similarity predictions will be used to weight the contribution of each source domain during testing, thus reducing the distribution difference and improving the performance on the target domain.
[0016] Step 6: Adjust the weight of the target domain classifier C tgt according to the similarity between each auxiliary target domain T i and T 0 .
[0017] The target domain classification loss of the target domain classifier is: Where: is the sample from T i ∈T aux and its target domain label, is the feature of the auxiliary target domain T i , is the domain classification result of the target domain classifier C tgt for the features of the auxiliary target domain, and E is the cross-entropy loss.
[0018] Step 7: C tgt predicts the similarity of the features of T 0 reflecting the closest auxiliary target domain, and normalizes the prediction result into a probability distribution through the softmax layer to obtain Where: is the sample of T 0 , is the domain classification result of the target domain classifier C tgt for the features of the main target domain. is the weight of T i . The auxiliary target domain T aux supports training by calculating the auxiliary target domain loss. By analogy, calculate the 0 of T and Calculate the i ∈T aux of each auxiliary target domain T and Combine these individual target domain losses with their weights to adjust their contributions.
[0019] Step 8: Calculate the total loss function of the model: Where: β, λ, and γ are hyperparameters.
[0020] Step 9. During testing, aggregate the results of multiple emotion label classifiers trained on different source domains to obtain the most accurate prediction for T 0 The label classifiers trained on source domains with feature distributions similar to the target domain usually perform better. Therefore, calculate the weights of the source domains based on the predictions of C src for the target data. These weights represent the relative importance of each source domain and its corresponding classifier in testing. After passing through the softmax layer, these predictions are normalized to a probability distribution, resulting in where: is a sample of T 0 and is the domain classification result of the source domain classifier C src for the features of the main target domain. is the weight of S i . Meanwhile, obtain which is the label prediction generated from the features of T 0 . The final label prediction combines these individual label predictions with their weights:
[0021] The present invention relates to a system for implementing the above method, including: a label classification unit, a domain alignment unit, a source domain similarity unit, and a target domain similarity unit, where: the label classification unit trains a source domain classifier according to the source domain data and its corresponding label information, optimizes the performance of the classifier by minimizing the classification loss, and provides high-quality source domain classification results for subsequent modules; the domain alignment unit calculates the distribution difference according to the feature information of the target domain and the source domain, and aligns the features of the target domain and the source domain through the maximum mean discrepancy (MMD) loss to reduce the distribution deviation and improve the adaptation performance of the model; the source domain similarity unit calculates the weight distribution of the source domain according to the prediction results of the source domain classifier on the similarity between the target domain and each source domain. Combining the prediction results of the source domain label classifier, dynamically adjusts the contributions of each classifier to generate the final prediction result of the target domain; the target domain similarity unit calculates the weight distribution of the auxiliary target domain according to the prediction results of the target domain classifier on the similarity between the main target domain and each auxiliary target domain. By weighting the loss of the auxiliary target domain, further enhances the adaptability of the model to the target domain. Technical effects
[0022] The present invention uses a multi-source multi-target domain similarity network (MSMTDS) to treat different individuals as independent source domains or target domains, comprehensively considering cross-cultural and intra-cultural individual differences, and ensuring the adaptability and accuracy of the emotion recognition model when dealing with different cultures and individuals. By evaluating cross-cultural and intra-cultural similarities, the weights of each domain are dynamically adjusted, enabling similar domains to play a dominant role in training while reducing the adverse effects of dissimilar domains. MSMTDS makes full use of limited EEG data to maximize model performance. Compared with the prior art, the present invention significantly solves the problem of huge differences in emotion patterns between different cultures in cross-cultural EEG emotion recognition, while taking into account the differences in the feature distributions among individuals within a culture. Dynamically adjusting the weights of each source domain and target domain enables the model to be optimized according to the importance of each domain during training, thereby effectively reducing the interference of irrelevant or noisy domains on the model and improving the cross-cultural adaptability of the model. Experiments prove that the experimental results of the MSMTDS method on the SEED dataset (including EEG data of three cultures: Chinese, French, and German) show that this method has achieved the latest research results in cross-cultural EEG emotion recognition tasks, demonstrating excellent generalization ability. At the same time, this method reveals the emotion representation characteristics of EEG signals between different cultures, provides a new perspective for in-depth understanding of emotion expression under different cultural backgrounds, and provides valuable references for cross-cultural emotion computing and cross-cultural communication. BRIEF DESCRIPTION OF THE DRAWINGS
[0023] Figure 1 is a flowchart of the present invention;
[0024] Figure 2 is a schematic diagram of the model in the embodiment;
[0025] Figure 3 is a schematic diagram of the effect in the embodiment. DETAILED DESCRIPTION OF THE EMBODIMENTS
[0026] As Figure 1 shown, this embodiment relates to a cross-cultural EEG emotion recognition method based on multi-source multi-target domain similarity, including:
[0027] Step 1: This embodiment uses datasets of three cultures, namely SEED (Chinese), SEED-GER (German), and SEED-FRA (French), as cross-cultural EEG datasets.
[0028] Step 2: Segment the EEG signals using a 1-second Hamming window and extract 310-dimensional differential entropy features.
[0029] Step 3. In this embodiment, two experimental scenarios are designed: within-culture cross-individual experiment and cross-culture experiment. The within-culture cross-individual experiment is used as a control experiment to evaluate the cross-cultural influence and the effectiveness of the Domain Adaptation (DA) method.
[0030] The within-culture experiment adopts the "Leave-One-Subject-Out Cross-Validation" scheme. Each time, one subject in the same culture is selected as the target domain, and the remaining subjects are used as the source domain for training.
[0031] In the cross-culture experiment, each time one subject is selected as the test target domain, other subjects in the same culture are used as the auxiliary target domain, and subjects in other cultures are used as the source domain for training.
[0032] Step 4. Use the data obtained in Step 3 to train and test the cross-cultural EEG emotion recognition model (MSMTDS) shown as Figure 2 including the feature extractor G, multiple emotion category classifiers C i l , the source domain classifier C for calculating the similarity between each source domain and the target domain src and the target domain classifier C for calculating the similarity between target domains tgt in two scenarios: within-culture and cross-culture. Specifically: the RMSProp optimizer with a learning rate range of [0.00001, 0.01], and early stopping is used to prevent overfitting; the feature generator is based on a three-layer MLP with hidden layer sizes of [256, 128, 64, 32] and the activation function is ReLU; the number of self-attention heads in the Transformer model is adjusted within the range of [1, 2, 4, 8]. For all domain adaptation methods, the trade-off hyperparameter of the target loss ranges from [0.1, 2]; for the MSMTDS model, the hyperparameters α, β, λ, and γ are adjusted within the ranges of [0.1, 2], [0.001, 1], [0.001, 1], and [0.1, 5] respectively to ensure the robustness and superiority of the model in different scenarios.
[0033] Step 5. Analyze and visualize the test results obtained in Step 4, and compare them with multiple domain adaptation (DA) methods.
[0034] The described domain adaptation methods include: single-source domain adaptation methods (DDC, Deep CORAL, DANN, CDAN, Transformer-DANN) and multi-source domain adaptation methods (MFSAN and wMADA). The baseline model (Source-Only) is a multi-layer perceptron (MLP) without using any domain adaptation techniques. In the same-culture scenario, since individuals in the source domain and the target domain come from the same culture, it cannot be regarded as a multi-target domain problem, and the MSMTDS model is not applicable. Therefore, the target domain similarity module (abbreviated as MSDS) is removed for replacement.
[0035] Table 1 Comparison of experimental data
[0036] In the same-culture cross-individual experiment and the cross-culture experiment, the present invention shows significant advantages among all domain adaptation baseline methods, achieving the optimal classification accuracy. The present invention can effectively solve the problem of cross-cultural EEG data distribution differences, especially in the CHN-FRA and CHN-GER groups, by making full use of all the data in the target domain. The experimental results show that the present invention has significant practical application value in the cross-cultural EEG emotion recognition task.
[0037] As Figure 3 shown, the feature distributions of the existing comparison model (DANN, left figure) and the model of the present invention (MSMTDS, right figure) are visualized through t-SNE dimensionality reduction projection. The visualization results show that the existing comparison model achieves feature alignment between the source domain and the target domain to a certain extent, but the discrimination between different emotion categories is poor. In contrast, the present invention not only achieves robust feature alignment in the cross-cultural scenario, but also significantly improves the separation effect of different emotion categories, fully demonstrating the value of the multi-source multi-target domain structure in solving the cross-cultural emotion recognition task.
[0038] The above specific implementation can be locally adjusted by those skilled in the art in different ways without departing from the principles and purposes of the present invention. The protection scope of the present invention is subject to the claims and is not limited by the above specific implementation. All implementation solutions within its scope are subject to the present invention.
Claims
1. A cross-cultural EEG emotion recognition method based on multi-source and multi-target domain similarity, characterized in that: By training the feature extractor G and multiple emotion category classifiers in the offline stage The source domain classifier C is used to calculate the similarity between each source domain and the target domain. src and a target domain classifier C for calculating the similarity between target domains tgt The cross-cultural EEG emotion recognition model performs cross-cultural EEG emotion recognition in real time in the online stage.
2. The cross-cultural EEG emotion recognition method based on multi-source and multi-target domain similarity according to claim 1 is characterized in that: include: Step 1: According to the input source domain Main target domain T0, auxiliary target domain Obtain features through feature extractor G; Step 2: Apply the Maximum Mean Difference (MMD) loss to minimize each source domain S i The distribution difference between the target domain T0; Step 3: Use respective emotion classifiers Processing from source domain S i Features And get the corresponding label classification loss; Step 4: Pass all M S The label classifier processes the features extracted from the main target domain T0 Generate M S Class label predictions; Step 5: Train the source domain classifier C src And calculate the source domain classification loss Specifically: in: The source domain S i The samples and their source domain labels, The source domain S i feature, is the source domain classifier C src For the domain classification result of the source domain features, E is the cross entropy loss; Step 6: Use the target domain classifier C tgt According to each auxiliary target domain T i The similarity with T0 adjusts its weight; Step 7, C tgt The feature of predicted T0 reflects the similarity of the closest auxiliary target domain. The prediction result is normalized into a probability distribution through the softmax layer, and we get in: is the T0 sample, is the target domain classifier C tgt Domain classification results for the main target domain features, T i The weight of the auxiliary target domain T aux Training is supported by computing an auxiliary target domain loss, similar to T0 and Calculate each auxiliary target domain T i ∈T aux of and These individual target domain losses are combined with their weights to adjust their contributions; Step 8. Calculate the total loss function of the model: Where: β, λ and γ are hyperparameters; Step 9. At test time, aggregate the results of multiple emotion label classifiers trained on different source domains to obtain the most accurate prediction for T0. Label classifiers trained on source domains with similar feature distributions to the target domain usually perform better. Therefore, according to C src The weights of the source domains are calculated by predicting the target data. These weights represent the relative importance of each source domain and its corresponding classifier in the test. These predictions are normalized into probability distributions after passing through the softmax layer, and we get in: is the T0 sample, is the source domain classifier C src Domain classification results for the main target domain features, For S i The weight of Is Label prediction generated for T0 features, final label prediction Combine these individual label predictions with their weights:
3. The cross-cultural EEG emotion recognition method based on multi-source and multi-target domain similarity according to claim 2 is characterized in that: The maximum mean difference (MMD) loss is: Where: M S is the number of source domains, The source domain S i feature, is the main target domain T0 feature; the maximum mean difference is a Gaussian kernel function used to measure the similarity between samples x and y.
4. The cross-cultural EEG emotion recognition method based on multi-source and multi-target domain similarity according to claim 2 is characterized in that: The classification loss of the label classifier is: in: The source domain S i Data and sentiment labels, M S is the number of source domains, The source domain S i feature, For sentiment classifier Emotion classification results of source domain features, E is the cross entropy loss.
5. The cross-cultural EEG emotion recognition method based on multi-source and multi-target domain similarity according to claim 2 is characterized in that: The source domain classification loss of the source domain classifier is: in: The source domain S i The samples and their source domain labels, The source domain S i feature, is the source domain classifier C src The domain classification result of the source domain features, E is the cross entropy loss.
6. The cross-cultural EEG emotion recognition method based on multi-source and multi-target domain similarity according to claim 2 is characterized in that: The target domain classification loss of the target domain classifier is: in: For T i ∈T aux samples and their target domain labels, is the auxiliary target domain T i feature, is the target domain classifier C tgt The domain classification result of the auxiliary target domain features, E is the cross entropy loss.
7. A cross-cultural EEG emotion recognition system based on multi-source and multi-target domain similarity that implements any of the methods described in claims 1-6, characterized in that: include: The label classification unit, the domain alignment unit, the source domain similarity unit and the target domain similarity unit include: the label classification unit trains the source domain classifier according to the source domain data and its corresponding label information, optimizes the classifier performance by minimizing the classification loss, and provides high-quality source domain classification results for subsequent modules; the domain alignment unit calculates the distribution difference according to the feature information of the target domain and the source domain, and aligns the target domain and source domain features through the maximum mean difference (MMD) loss to reduce the distribution deviation and improve the adaptability of the model; the source domain similarity unit calculates the weight distribution of the source domain according to the prediction results of the source domain classifier for the similarity between the target domain and each source domain, and dynamically adjusts the contribution of each classifier in combination with the prediction results of the source domain label classifier to generate the final prediction results of the target domain; The target domain similarity unit calculates the weight distribution of the auxiliary target domain based on the prediction results of the target domain classifier on the similarity between the main target domain and each auxiliary target domain, and further enhances the adaptability of the model to the target domain by weighting the loss of the auxiliary target domain.