A Deep Transfer Learning Domain Adaptation Method for Emotion Recognition
By combining a three-layer network structure of joint distributional adaptation and deep convolutional neural networks, the problem of domain adaptation across subjects and trial batches in EEG emotion recognition was solved, achieving higher classification accuracy and transfer capability.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-03-27
- Publication Date
- 2026-04-03
AI Technical Summary
Existing technologies struggle to establish universal models for emotion recognition using EEG signals, exhibit low domain adaptation in cross-subject and cross-batch experiments, and cannot completely eliminate domain differences between the source and target domains.
We adopt a deep transfer learning domain adaptation method for emotion recognition, which combines joint distribution adaptation with deep convolutional neural networks. We use a three-layer network topology, consider the difference between marginal distribution and conditional distribution, and use a multi-kernel computing method for multi-layer deep domain adaptation.
It improves domain adaptability, effectively bridging the differences in marginal and conditional distributions between the source and target domains, and enhancing the classification accuracy and transferability of emotion recognition transfer learning.
Smart Images
Figure CN116796221B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of emotion recognition transfer learning, and in particular to a deep transfer learning domain adaptation method for emotion recognition. Background Technology
[0002] EEG-based emotion recognition technology is gaining increasing attention as an important tool for disease diagnosis and treatment. However, due to the complex characteristics of EEG signals, even the same person under the same stimuli will produce different EEG signals, making it difficult to establish a universal model applicable to different subjects or trial batches. To address this issue, some studies employ domain adaptation methods from transfer learning. The core of domain adaptation is to reduce the difference between two domains, thereby determining the label of the target domain data using labeled source domain data. Currently, maximum mean difference (MMD) is the most commonly used method to measure the difference between two domains, mapping source and target domain data to a regenerating kernel Hilbuter space (RKHS) for measurement. However, most research methods rely on marginal or conditional distributions to measure distributional differences, failing to completely eliminate the domain difference between the source and target domains, and exhibiting low domain adaptation in cross-subject and cross-trial batch experiments. Summary of the Invention
[0003] This invention aims to overcome the problems of domain differences between the source and target domains in existing emotion recognition transfer learning technologies, as well as the low degree of domain adaptation in cross-subject and cross-trial batch experiments, and provides a domain adaptation method for deep transfer learning in emotion recognition.
[0004] To achieve the above objectives, the present invention adopts the following technical solution:
[0005] A deep transfer learning domain adaptation method for emotion recognition includes the following steps: S1: Sampling the raw EEG signals and preprocessing them, then extracting differential entropy features from the processed EEG signals; S2: In a cross-subject experiment, selecting data from one subject as the target domain data, merging the remaining subject data as a unified source domain data, and using an MLP layer to perform feature dimensionality reduction on the target domain data and the source domain data; S3: Performing deep domain adaptation on the dimensionality-reduced source domain data and target domain data; the deep domain adaptation is a three-layer network topology model, where the first layer calculates the source domain data and target domain data... According to the JP-MMD value, JP-MMD considers both marginal and conditional distributions. The last two layers of the model calculate the MMD value. S4: Perform label prediction on the target domain data, compare the prediction results with the actual results, and obtain the accuracy of this cross-subject experiment. S5: Select different subjects as the target domain, repeat S2-S4, until all subjects and all batches have been cycled once, and the cross-subject experiment ends. S6: In the cross-batch experiment, select one batch of the test as the target domain data, and the remaining data as the source domain data, execute S3-S4, until all subjects and all batches have been cycled once, and the cross-batch experiment ends. This invention provides a deep transfer learning domain adaptation method for emotion recognition, which combines joint distribution adaptation with deep convolutional neural networks. The joint distribution adaptation method considers the differences between the marginal and conditional distributions of the source and target domain data, improving the domain adaptation capability. This invention uses a three-layer network for multi-layer deep domain adaptation. Through multi-layer adaptation, the domain differences under marginal and conditional distributions can be effectively bridged, improving the network's transfer capability.
[0006] As a preferred embodiment of the present invention, S3 specifically includes the following steps: S31: Inputting source domain data and target domain data into a deep domain adaptive model for feature transformation; S32: Performing edge distribution adaptation, calculating the distance between source domain data and target domain data samples in k-dimensional embedding; S33: Performing conditional distribution adaptation, obtaining pseudo-labels for unlabeled target domain data, matching class conditional distributions, and measuring the class conditional distribution distance; S34: After matching the edge distribution and conditional distribution, the source domain data and target domain data reach the last two layers of the network model for deep domain adaptation, and the last two layers of the network use maximum mean difference calculation and comparison of different distributions.
[0007] As a preferred embodiment of the present invention, the feature transformation formula in S31 is as follows:
[0008]
[0009] Where tr(·) represents the trace of the matrix, Represents the input data matrix. 1 is the central matrix, which is an n×n matrix of 1, and I = A T A, n are the source domain data n s With target domain data n t Total data volume, XHX T To calculate the covariance matrix.
[0010] As a preferred embodiment of the present invention, the formula for calculating the distance between the source domain data and the target domain data sample in the k-dimensional embedding in step S32 is as follows:
[0011]
[0012] Where M0 is the maximum mean difference matrix, calculated as follows:
[0013]
[0014] in, and These are source domain data and target domain data, respectively, n s With n t These represent the source domain data volume and the target domain data volume, respectively, x i ,x j ∈D s and x i ,x j ∈D t They represent x respectively i and x j Data from the source domain and data from the target domain.
[0015] As a preferred embodiment of the present invention, the formula for measuring the distribution distance of the measurement class conditions in S33 is as follows:
[0016]
[0017] in, y(x) represents the set of samples belonging to class c in the source domain data. i ) is x i Authentic labels, and This represents the set of samples belonging to class c in the target domain. For x j The pseudo-tags, and MMD matrix containing class tags c The calculation method is as follows:
[0018]
[0019] As a preferred embodiment of the present invention, the last two layers of the network in S34 use the maximum mean difference calculation and comparison of different distributions, and the calculation formula is the same as the formula for calculating the distance between the source domain data and the target domain data samples in the k-dimensional embedding in S32.
[0020] As a preferred embodiment of the present invention, the three-layer network topology model adopts a multi-core computing method, and its kernel function... The calculation method is as follows:
[0021]
[0022] Where k represents the obtained multi-core, h represents the number of defined power spectral density kernel convex combinations, and *η i} represents the coefficient.
[0023] Therefore, this invention combines joint distribution adaptation with deep convolutional neural networks to propose the Edge Conditional Distribution Deep Domain Adaptation (MCD-DDA) method, which has the following beneficial effects: 1. This invention adopts a joint distribution adaptation method, which simultaneously considers the differences between the edge distribution and conditional distribution of source domain data and target domain data, thus improving domain adaptability; 2. This invention is not a single-layer domain adaptation architecture, but uses a three-layer network for multi-layer deep domain adaptation. A single-layer network structure cannot eliminate the deviation between source domain data and target domain data. Through the multi-layer adaptation of this invention, the domain differences under edge distribution and conditional distribution can be effectively bridged, improving the network's transferability; 3. This invention adopts a multi-kernel computation method. A single kernel function in a single-layer network will have a large deviation in different models. The multi-layer network uses the optimal multi-kernel selection method for mean embedding matching, further reducing domain differences. Attached Figure Description
[0024] Figure 1 This is a flowchart of the method of the present invention;
[0025] Figure 2 This is a flowchart of step S3 of the present invention;
[0026] Figure 3 This is a flowchart of the three-layer network model of the present invention. Detailed Implementation
[0027] The present invention will now be further described with reference to the accompanying drawings and specific embodiments.
[0028] Example 1:
[0029] The purpose of this invention is to address the domain differences between the source and target domains in emotion recognition transfer learning, as well as the low degree of domain adaptation in cross-subject and cross-batch experiments. It proposes a deep transfer learning domain adaptation method for emotion recognition, such as... Figure 1 As shown, it includes the following steps:
[0030] S1: The original EEG signal is downsampled to 200Hz, and a 0-70Hz bandpass filter is used to remove noise and interference signals. Then, the differential entropy features are extracted from the processed EEG signal.
[0031] S2: In a cross-subject experiment, one subject's data is selected as the target domain data, and the remaining subjects' data are merged as a unified source domain; after determining the source domain data and the target domain data, an MLP layer is used to perform feature dimensionality reduction.
[0032] S3: Perform deep domain adaptation between the dimensionality-reduced source domain data and the target domain data, such as... Figure 3 As shown, deep domain adaptation is a three-layer network topology. The first layer calculates the joint maximum mean difference (JP-MMD) between the source and target domain data. JP-MMD considers both marginal and conditional distributions. The next two layers calculate the MMD value to reduce the difference between the source and target domain data in the latent space. Through deep domain adaptation, the model can transfer knowledge learned from the source domain data to the target domain data, enabling label prediction for the target domain data.
[0033] like Figure 2 As shown, the specific method of step S3 is as follows:
[0034] S31: Perform feature transformation on depth-domain adaptive input data:
[0035]
[0036] Where tr(·) represents the trace of the matrix, Represents the input data matrix. 1 is the central matrix, which is an n×n matrix of 1, and I = A T A, n are the source domain data n s With target domain data n t Total data volume, XHX T The goal of calculating the covariance matrix is to find an orthogonal transformation matrix. To maximize the variance of the embedded data, we can find the optimal k-dimensional representation Z = [z1, z2, ... z]. n ] = A T X.
[0037] S32: Marginal distribution adaptation is performed. To reduce the difference between the source and target domains, the maximum mean difference is used to compare different distributions. It calculates the distance between the source and target domain data samples in the k-dimensional embedding:
[0038]
[0039] Where M0 is the maximum mean difference matrix, calculated as follows:
[0040]
[0041] in, and n represents the data in the source and destination domains, respectively. s With n t x represents the amount of data in the source domain and the amount of data in the target domain, respectively. i ,x j ∈D s and x i ,x j ∈D t They represent x respectively i and x j From the source domain data and the target domain data, by minimizing formula (2), formula (1) is maximized, thus in the new representation Z = A T Under X, the edge distributions of the source and target domains will be closer.
[0042] S33: Perform conditional distribution adaptation. After obtaining pseudo-labels for the unlabeled target domain data, match the conditional distribution Q of the class. s =(x s |y s =c) and Q t =(x t |y t =c), where each class is c∈*1,2,…,C}, where x s and y s Indicates source domain data and source domain label, x t With y t The formula for representing the distance between the target domain data and the target domain label, and the distribution of measurement class conditions, is as follows:
[0043]
[0044] in y(x) represents the set of samples belonging to class c in the source domain data. i ) is x i Authentic labels, and Corresponding This represents the set of samples belonging to class c in the target domain. For x j The pseudo-tags, and MMD matrix containing class tags c The calculation method is as follows:
[0045]
[0046] By minimizing formula (4), formula (1) is maximized, thus in the new representation Z = A T Under X conditions, the conditional distributions of the source and target domains will be more similar.
[0047] S34: After matching the edge distribution and conditional distribution, the source domain data and target domain data reach the last two layers of the network for deep domain adaptation. To further reduce domain differences, the last two layers of the network use the maximum mean difference to compare different distributions. The calculation formula is shown in S32. The three-layer network adaptation uses a multi-core calculation method, and its kernel function calculation method is as follows. as follows;
[0048]
[0049] Where k represents the obtained multi-core, h represents the number of defined power spectral density kernel convex combinations, and {η i} represents the coefficient. For *η i The constraint} ensures that the obtained multi-kernel k is characteristic.
[0050] S4. Perform label prediction on the target domain data, compare the prediction results with the actual results, and obtain the accuracy of this cross-subject experiment.
[0051] S5. Select different subjects as the target domain, and repeat steps S2-S4 until all subjects and all batches have been cycled once, and then the cross-subject experiment ends.
[0052] S6. In the cross-batch experiment, select one of the test batches as the target domain and the remaining data as the source domain data, and then proceed with steps S3-S4; the cross-batch experiment ends after all subjects and all test batches have been cycled once.
[0053] Example 2:
[0054] The feasibility of the above method was verified using the SEED dataset, established and released by the BCMI laboratory of a university in Shanghai. The SEED dataset includes positive, negative, and neutral emotions. Fifteen participants (7 males, 8 females; mean age 23.27 years) watched 15 film clips (5 for each of the three emotions), and data was collected each time. The average film length was 226 seconds, and each clip ranged from 185 to 165 seconds, including a 5-second prompt, a 45-second self-assessment, and a 15-second rest within each clip. Each participant underwent three experiments, approximately one week apart, for a total of 45 experiments. EEG signals were recorded using 62 electrodes at a sampling rate of 1000 Hz, with downsampling down to 200 Hz, and filtered from 0-75 Hz.
[0055] To better compare the superiority of the method proposed in this invention, we selected three domain adaptation methods as comparative experiments:
[0056] (1)DDC: A single-layer domain adaptive architecture that uses MMD for measurement.
[0057] (2) JDA: A single-layer domain adaptive architecture, which uses JP-MMD for measurement.
[0058] (3) DAN: A multi-layer domain adaptive architecture, which is metric using MK-MMD.
[0059] Table 1 Comparative experimental data
[0060]
[0061]
[0062] The experimental results above show that JDA has a higher accuracy than DDC, indicating that JP-MMD is more effective for EEG signals than traditional MMD and has a positive effect on the classification accuracy of emotion recognition transfer learning. Compared to the single-layer network structure of DDC, the multi-layer network of DAN is obviously more conducive to emotion recognition transfer learning, as the multi-layer network structure has better domain adaptability. The MCD-DDA method proposed in this invention combines JP-MMD and a multi-layer network structure, providing a better deep domain adaptive structure for emotion recognition transfer learning. MCD-DDA's classification accuracy is superior to JDA and DAN, and it has better domain adaptability. The network architecture of this invention is easy to understand, has strong adaptability, and can significantly improve the classification accuracy of emotion recognition transfer learning, making it suitable for widespread application.
[0063] Terminology Explanation:
[0064] MMD: Maximum mean difference.
[0065] RKHS: Regenerating Core Hilbut Space.
[0066] MCD-DDA: Edge Condition Distribution Depth Domain Adaptive.
[0067] JP-MMD: Maximum mean difference in joint probabilities.
[0068] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions conceived without creative effort should be included within the scope of protection of the present invention.
Claims
1. A deep transfer learning domain adaptation method for emotion recognition, characterized in that, Includes the following steps: S1: Sample and preprocess the raw EEG signal, and extract differential entropy features; S2: In a cross-subject experiment, select one subject's data as the target domain data and the remaining subject's data as the source domain data, and perform feature dimensionality reduction on the data; S3: Input source domain data and target domain data into the deep domain adaptive model, perform edge distribution adaptation, and perform conditional distribution adaptation; perform deep domain adaptation on the dimensionality-reduced data; the deep domain adaptation is a three-layer network topology model. S4: Predict the labels for the target domain data and compare them with the actual results to obtain the accuracy of this cross-subject experiment; S5: Select different subjects as the target domain, repeat S2-S4, until all subjects and all batches have been cycled once, and then the cross-subject experiment ends. S6: In a cross-batch experiment, select one test batch as the target domain data and the remaining data as the source domain data. Execute S3-S4 until all subjects and all test batches have been cycled once, and then the cross-batch experiment ends.
2. The deep transfer learning domain adaptation method for emotion recognition according to claim 1, characterized in that, The three-layer network topology model in S3 calculates the JP-MMD value of source domain data and target domain data in the first layer of the model. The JP-MMD value considers both edge distribution and conditional distribution. The last two layers of the model calculate the MMD value.
3. The deep transfer learning domain adaptation method for emotion recognition according to claim 1, characterized in that, S3 specifically includes the following steps: S31: Perform feature transformation on the source domain data and the target domain data; S32: Calculate the distance between the source domain data and the target domain data samples in the k-dimensional embedding; S33: After obtaining the pseudo-labels of the unlabeled target domain data, match the class conditional distribution and measure the distance between the class conditional distributions; S34: After matching the marginal distribution and the conditional distribution, the source domain data and the target domain data reach the last two layers of the network model for deep domain adaptation. The last two layers of the network use the maximum mean difference calculation to compare different distributions.
4. The deep transfer learning domain adaptation method for emotion recognition according to claim 3, characterized in that, The feature transformation formula in S31 is as follows: in, Represents the trace of a matrix. Represents the input data matrix. , The central matrix is a 1. matrix, n is the source domain data With target domain data Total data volume To calculate the covariance matrix.
5. The deep transfer learning domain adaptation method for emotion recognition according to claim 3, characterized in that, The formula for calculating the distance between the source domain data and the target domain data samples in the k-dimensional embedding in S32 is as follows: in, The maximum mean difference matrix is calculated as follows: in, They are source domain data and target domain data, respectively. and These represent the data volume of the source domain and the data volume of the target domain, respectively. and They represent and Data from the source domain and data from the target domain.
6. The deep transfer learning domain adaptation method for emotion recognition according to claim 3, characterized in that, The formula for the distance of the measurement class condition distribution in S33 is as follows: in, This represents the set of samples belonging to class c in the source domain data. for Authentic labels, and , This represents the set of samples belonging to class c in the target domain. for The pseudo-tags, and MMD matrix containing class tags The calculation method is as follows: 。 7. A deep transfer learning domain adaptation method for emotion recognition according to claim 3, 4, 5, or 6, characterized in that, In S34, the last two layers of the network use the maximum mean difference calculation to compare different distributions. The calculation formula is the same as the formula in S32 for calculating the distance between source domain data and target domain data samples in k-dimensional embedding.
8. The deep transfer learning domain adaptation method for emotion recognition according to claim 1, characterized in that, The three-layer network topology model employs a multi-core computation method, and its kernel function... The calculation method is as follows: Where k represents the obtained multi-core, and h represents the number of defined power spectral density kernel convex combinations. is a coefficient.