An EEG emotional state classification method based on multi-source domain adaptation of knowledge distillation

Abstract

The application discloses a multi-source domain adaptation EEG emotion state classification method based on knowledge distillation. First, data is acquired for band pass filtering, and independent component analysis technology is used to remove artifacts. Second, the electroencephalogram feature is extracted through the differential entropy method, and the three-dimensional electroencephalogram time sequence is converted into a two-dimensional sample matrix. Then, the training set and the test set are respectively demarcated under two task scenarios, and it is ensured that they do not coincide. The application adopts the pseudo-label triplet loss based on marginal sampling combined with the maximum mean difference. The application learns knowledge from different source domains to maximize the use of multiple single-source models and realize a more powerful model with less time consumption. Finally, the classification accuracy is used to evaluate the performance of the model under two task scenarios. The application combines the triplet loss and the maximum mean difference, which can not only realize unbiased alignment between each pair of source domains and the target domain at the domain level, but also consider the correlation at the data pair level.

Description

Technical Field

[0001] This invention belongs to the field of EEG emotion state recognition in the field of biometrics, specifically involving an EEG emotion state classification method based on multi-source domain adaptation of knowledge distillation. Background Technology

[0002] Emotion recognition plays a crucial role in human-computer interaction. In recent years, with the improvement of computing power, deep learning-based emotion recognition methods have received increasing attention. These methods make decisions that reflect human emotions by deeply mining the user's potential objective emotional characteristics.

[0003] Emotion-based brain-computer interfaces (aBCIs) represent an important application of emotion recognition. By measuring signals from the peripheral and central nervous systems, they extract features related to the user's emotional state and utilize these features to tailor human-computer interaction (HCI). They show potential in rehabilitation and communication.

[0004] Generally, emotion recognition can be divided into two categories: methods based on non-physiological signals, such as facial expressions, body gestures, and speech signals; and methods based on physiological signals, such as electroencephalography (EEG), electromyography (EMG), and electrocardiography (ECG). However, compared to non-physiological signals, physiological signals can directly access an individual's internal emotional state, making them less susceptible to conscious or unconscious manipulation. Among various physiological signal-based emotion recognition methods, EEG is one of the most commonly used methods because it collects data directly from the cerebral cortex and is valuable for reflecting a person's psychological state. With the rapid development of EEG acquisition and processing methods, EEG-based emotion recognition has received increasing attention in recent years.

[0005] However, building efficient and robust EEG-based deep learning models for emotion recognition remains a significant challenge due to low signal-to-noise ratios (SNR) and substantial individual differences across time periods and subjects. Furthermore, it is crucial to utilize existing labeled data to analyze new unlabeled data in EEG-based BCI. To address this, domain adaptation has been widely used in research, training a model that performs well on relevant but distinct target data distributions by learning from the source data distribution. However, in practice, multiple source domains are often present, making multi-source domain adaptation a powerful extension of domain adaptation. Nevertheless, the technique used for domain alignment in multi-source domain adaptation is typically maximum mean difference (MMD), which only considers domain-level adaptation and lacks data-pair-level adaptation. This limitation can lead to a lack of discriminative power. Moreover, most multi-source domain adaptation frameworks simply use the average predictions of multiple single-source domain models as the final result, failing to fully utilize these single-source domain models. Summary of the Invention

[0006] To address the shortcomings of existing technologies and better leverage the advantages of multiple single-source models, this invention proposes a multi-source domain adaptation EEG sentiment state classification method (MS-KTF) based on knowledge distillation.

[0007] The technical solution adopted in this invention is:

[0008] This invention uses differential entropy (DE) features as the frequency domain features of the EEG signals used, slightly modifies the EEGNet model as the feature extractor, and uses a single-layer linear layer as the classifier to analyze the EEG signals, thereby achieving the task of emotion state recognition in both cross-subject and cross-time periods.

[0009] This invention divides the training process into three steps: (1) pre-training each teacher model based on each labeled source domain; (2) adapting each teacher model to the corresponding labeled source domain and unlabeled target domain using source domain classification loss (SCL), target domain classification loss (TCL), maximum mean difference (MMD), and pseudo-label triplet loss; and (3) transferring the knowledge of multiple single-source domain teachers to the student model. Furthermore, in step (2), to improve the effectiveness of the pseudo-label triplet loss, we employ a marginal sampling strategy to filter the original features, selecting only those features with marginal scores higher than a preset threshold as the embedding features for calculating the pseudo-label triplet loss.

[0010] The embodiments of the present invention include the following steps:

[0011] Step S1. Data Processing:

[0012] Taking the SEED emotion dataset as an example, the processing steps for the raw EEG data collected by the EEG acquisition device are as follows:

[0013] S1-1: Data Denoising

[0014] The dataset used to verify the model performance in this invention comes from SEED. First, the raw EEG signals collected in the dataset are downsampled to 200Hz, then bandpass filtering and ICA techniques are applied from 0-75Hz to remove electrooculogram artifacts from the signals, and finally, traditional moving average and linear dynamic system (LDS) methods are used to further smooth the features.

[0015] S1-2: DE Feature Extraction

[0016] Dependency extraction (DE) was performed on the artifact-removed EEG data. For each subject, a 1-second non-overlapping sliding window was used for data segmentation, resulting in 3394 data samples. For each data sample x... iThe number of EEG data acquisition channels was 62; and frequency domain features of five frequency bands, namely δ (1-3HZ), θ (4-7HZ), α (8-13HZ), β (14-30HZ) and γ (31-50HZ), were extracted.

[0017] Step S2. Data definition and dataset partitioning:

[0018] There are two testing scenarios for classifying emotional states: across subjects and across time periods. The model testing in these two scenarios has different data definitions and dataset divisions, which will be explained in detail below.

[0019] Assume there are N participants, each participating in D different (conversational) sessions. The entire sample set is represented as... Where i represents the subject's sequence number, j represents the session (time period) sequence number, and X i Let Y represent the sample set of subject i, and the corresponding label set be Y. i .

[0020] For the sentiment state classification task spanning multiple time periods, leave-one-out cross-validation of the dataset is also employed. Specifically, for each participant i, the data from 15 sentiment trials of all participants in the latest session are used as the test set; the remaining D-1 sessions are used as one source domain in the training set, resulting in D-1 source domains as the training set. A total of N experiments are conducted, and the average accuracy is calculated.

[0021] For the cross-subject emotional state classification task, leave-one-out cross-validation of the dataset is employed. Specifically, within a session (time period), data from all 15 emotional trials of a single subject are iteratively extracted, assuming their emotional state labels are unknown, and used as the test set. From the remaining N-1 subjects, R subjects are randomly and non-repeatingly grouped into a single group to form a source domain in the training set, ultimately obtaining... The training set consists of D × N source domains (rounded down). A total of D × N experiments are conducted, and the average accuracy is calculated.

[0022] Step S3. Construction and training of the MS-KTF model:

[0023] The main parameters in the MS-KTF neural network model include:

[0024] ① The dimension d of the feature embedding space f , which is the dimension of the feature space after it is extracted by the feature extractor and enters the embedding space, is closely related to the representational ability of the embedded features.

[0025] ② The threshold for edge sampling is the key parameter that determines whether a feature should be sampled. Features that score higher than the threshold are sampled.

[0026] ③ Temperature for distillation learning, i.e., the temperature coefficient in the softmax operation, is mainly used to adjust the smoothness of the predicted distribution.

[0027] S3-1: Initialization

[0028] MS-KFT consists of two parts: a teacher model based on a single source domain and a student model operating on the target domain. Both the teacher and student models are derived from a domain-specific feature extractor N. f And label classifier N y These two modules constitute the initialization process for multiple single-source-domain teacher models based on a multi-source-domain model and a target-domain student model.

[0029] S3-2: Pre-training multiple single-source-domain teacher models

[0030] Based on multiple source domain sample sets, a feature extractor N for a domain-specific teacher model is pre-trained using a single labeled source domain sample. f And label classifier N y This enables them to have a certain pattern recognition capability in their respective source domains.

[0031] S3-3: Domain adaptation of feature extractors for multiple single-source domain teacher models

[0032] Each labeled source domain sample and unlabeled target domain sample form a branch. In each branch, the feature extractor N of the corresponding domain-specific teacher model is used. f Feature extraction is performed on the source domain samples and target domain samples respectively, and the features are extracted from the original feature space into the embedding space.

[0033] Then, the embedded features are aligned at the domain level in the feature space based on the maximum mean difference; the embedded features are aligned at the data group level based on the pseudo-label triple loss based on edge sampling.

[0034] The feature extractor N is trained by minimizing the maximum mean difference and pseudo-label triplet loss for multiple single-source-domain teacher models. f This enables it to perform domain-invariant feature extraction in both the source and target domains.

[0035] S3-4: Training a label classifier N from multiple single-source-domain teacher models y

[0036] In each single-source-domain teacher model, the extracted source-domain feature information is processed by the label classifier N. y Obtaining predictive sentiment And calculate predicted sentiment Corresponding label Y in the actual sampleS The cross-entropy; similarly, calculating the predicted sentiment based on the feature information of the target domain. The pseudo tags generated by it Cross-entropy.

[0037] By minimizing the two obtained cross-entropies, a label classifier N for a multi-unit domain teacher model is trained. y This enables them to have good emotion classification capabilities in their respective source and target domains.

[0038] S3-5: Merge the knowledge from multiple single-source-domain teacher models;

[0039] To balance the performance of teacher models, two different merging strategies are used:

[0040] ① A teacher model for selection and merging based on voting.

[0041] This approach is more suitable for situations where teacher performance is poorly balanced. Using unlabeled target domain samples, the sentiment prediction results are first obtained through the feature extractor and label classifier of the teacher model. Generate the corresponding one-hot encoding result Based on the one-hot coding results generated by each teacher Conduct a vote and use the voting results as a decision variable. If the teacher model's emotion prediction results With decision variables If the results are the same, the teacher will be selected for knowledge merging.

[0042] Calculate the average of the sentiment predictions for all selected teacher models. As a merged knowledge of all teacher models.

[0043] ② Use the average form to merge teacher models

[0044] This approach is more suitable for situations where teacher performance is well-balanced. In this case, all teacher models have the same weight, and the average of the predictions from all selected teacher models is calculated. As a merged knowledge of all teacher models.

[0045] S3-6: Instruct the merged knowledge of the teacher model to the student model.

[0046] Using unlabeled target domain sample data, the prediction results of the student model are obtained through the feature extractor and label classifier of the student model.

[0047] Based on a preset distillation temperature, the merged knowledge of the teacher model is used. Predictions from the student model Smoothing was performed. The difference between the two predictions was evaluated using KL divergence.

[0048] By minimizing the KL divergence between the teacher and student models, the student model learns the knowledge of the teacher model, gaining more generalizable feature extraction and label classification capabilities in the target domain compared to the teacher.

[0049] Step S4. Model performance evaluation in two scenarios: across time periods and across participants:

[0050] This invention specifically verifies the model performance on the SEED dataset.

[0051] The target domain sample set is used to train a converged student model to predict emotional states. and the true state Y T Compare the results to obtain the accuracy score and evaluate the model performance. Accuracy is the percentage of correctly classified samples out of the total number of test samples. The formula for calculating model accuracy is as follows:

[0052]

[0053] Wherein, TP represents positive class samples predicted as positive by the model, TN represents negative class samples predicted as negative by the model, FP represents negative class samples predicted as positive by the model, and FN represents positive class samples predicted as negative by the model.

[0054] An EEG emotion state classification system based on knowledge distillation and multi-source domain adaptation includes a pre-training module, a teacher model domain adaptation module, and a student model training module. The pre-training module pre-trains each teacher model based on each labeled source domain. The teacher model domain adaptation module adapts each teacher model based on the corresponding labeled source domain and unlabeled target domain, using source domain classification loss (SCL), target domain classification loss (TCL), maximum mean difference (MMD), and pseudo-label triplet loss. The student model training module transfers knowledge from multiple single-source domain teachers to the student model.

[0055] Furthermore, in the teacher model domain adaptation module, in order to improve the effectiveness of pseudo-label triplet loss, we adopted a marginal sampling strategy to filter the original features, selecting only those features whose marginal scores are higher than a preset threshold as the embedded features for calculating pseudo-label triplet loss.

[0056] The beneficial effects of this invention are as follows:

[0057] This invention addresses the blind estimation problem of the Maximum Mean Difference (MMD) technique in multi-source domain adaptation by utilizing pseudo-label triplet loss. Furthermore, it employs a marginal sampling strategy based on uncertainty measurement to improve its effectiveness; simultaneously, it introduces a knowledge extraction technique, training a more robust student model by teaching knowledge from multiple teacher models to maximize the utilization of multi-source domain knowledge. Experimental validation on the public sentiment dataset SEED demonstrates a significant improvement over previous methods. Attached Figure Description

[0058] Figure 1 This is a schematic diagram of the process of the present invention;

[0059] Figure 2 This is a structural diagram of the feature extractor for the model in this discovery;

[0060] Figure 3 MS-KTF model architecture diagram;

[0061] Figure 4 Data partitioning and construction diagram for the MS-KTF model;

[0062] Figure 5 A diagram of the teacher model structure;

[0063] Figure 6 A diagram of the student model structure; Detailed Implementation

[0064] The preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, so that the advantages and features of the present invention can be more easily understood by those skilled in the art, thereby providing a clearer and more explicit definition of the scope of protection of the present invention.

[0065] Multi-source domain adaptation (MSDA) aims to transfer knowledge from multiple source domains to an unlabeled target domain, which is highly applicable for EEG emotion recognition across sessions and subjects. However, existing MSDA models only consider the domain level of each pair of feature relationships between the source and target domains, and rarely consider the correlation between data pairs at the data pair level between the two domains, resulting in poor robustness.

[0066] This invention discloses a multi-source domain knowledge transfer framework (MS-KTF) for EEG emotion recognition. First, the acquired data undergoes bandpass filtering, and artifacts are removed using Independent Component Analysis (ICA). Second, EEG features are extracted using the Differential Entropy (DE) method, converting the three-dimensional EEG time series into a two-dimensional sample matrix. Then, training and test sets are defined for two task scenarios, ensuring they do not overlap. For these samples, MS-KTF employs a marginal sampling-based pseudo-label triplet loss combined with Maximum Mean Difference (MMD). This method not only achieves unbiased alignment between each pair of source and target domains at the domain level but also considers the correlation at the data pair level. Specifically, the framework learns knowledge from different source domains to maximize the use of multiple single-source models and achieve a more powerful model with less time consumption. Finally, the model's performance is evaluated using classification accuracy in both task scenarios. This invention combines triplet loss and maximum mean difference, which to some extent solves the problem of insufficient alignment of EEG signal distribution differences. It trains a high-precision emotional state classifier that spans time periods and subjects, and has advantages such as low time complexity, high computational efficiency, and strong generalization ability. It is expected to have broad application prospects in actual brain-computer interaction.

[0067] For a detailed implementation of this invention, please refer to [link / reference]. Figure 1 , Figure 2 , Figure 3 , Figure 4 , Figure 5 , Figure 6 The embodiments of the present invention include the following steps:

[0068] Step S1. Data Processing:

[0069] Taking an emotion dataset as an example, the processing steps for the raw EEG data collected by the EEG acquisition device are as follows:

[0070] S1-1: Data Denoising

[0071] The dataset used to verify the model performance in this invention comes from SEED, and details can be found in the paper "Investigating Critical Frequency Bands and Channels for EEG-Based Emotion Recognition with Deep Neural Networks". First, the raw EEG signals collected from the dataset are downsampled to 200Hz, then bandpass filtered from 0.3 to 50Hz, and finally, ICA technology is used to remove electrooculography artifacts from the signals.

[0072] S1-2: DE Feature Extraction

[0073] Descriptive features (DE) were extracted from the artifact-removed EEG data. Each participant watched 15 videos that elicited significant emotional shifts. EEG data collected within the same video playback duration was considered a single emotional trial, and each participant had 15 emotional trials. Data was segmented using a 1-second non-overlapping sliding window for each participant, resulting in 3394 data samples. For each sample x... i The number of EEG data acquisition channels was 62; frequency domain features of five frequency bands, namely δ (1-3HZ), θ (4-7HZ), α (8-13HZ), β (14-30HZ) and γ (31-50HZ), were extracted.

[0074] Step S2. Data definition and dataset partitioning:

[0075] There are two testing scenarios for classifying emotional states: across subjects and across time periods. The model testing in these two scenarios has different data definitions and dataset divisions, which will be explained in detail below.

[0076] Assume there are N participants, each participating in D different (conversational) sessions. The entire sample set is represented as... Where i represents the subject's sequence number, j represents the session (time period) sequence number, and X i Let Y represent the sample set of subject i, and the corresponding label set be Y. i .

[0077] For the sentiment state classification task spanning multiple time periods, leave-one-out cross-validation of the dataset is also employed. Specifically, for each participant, the data from 15 sentiment trials of all participants in the most recent session are used as the test set; the remaining D-1 sessions are used as one source domain in the training set, resulting in D-1 source domains as the training set. A total of N experiments are conducted, and the average accuracy is calculated.

[0078] For the cross-subject emotional state classification task, leave-one-out cross-validation of the dataset is employed. Specifically, within a session (time period), data from all 15 emotional trials of a single subject are iteratively extracted, assuming their emotional state labels are unknown, and used as the test set. From the remaining N-1 subjects, R subjects are randomly and non-repeatingly grouped together as a source domain in the training set, ultimately obtaining... The training set consists of (rounded down) source domains. Finally, the model performance is validated on a test set with N subjects, for a total of D×N experiments, and the average accuracy is calculated.

[0079] Step S3: Construction and Training of the MS-KTF Model

[0080] The main parameters in the MS-KTF neural network model include:

[0081] 1) Dimension d of the feature embedding space f , which is the dimension of the feature space after it is extracted by the feature extractor and enters the embedding space, is closely related to the representational ability of the embedded features.

[0082] 2) Threshold for edge sampling, which is the key parameter that determines whether a feature should be sampled. Features that score higher than the threshold are sampled.

[0083] 3) Temperature during distillation learning, i.e., the temperature coefficient during softmax operation, is mainly used to adjust the smoothness of the predicted distribution.

[0084] S3-1: Specific data partitioning and input for the model

[0085] S3-1-1: Data Partitioning of the Model

[0086] Data partitioning and construction of the model, such as Figure 4 As shown, the specific division is described below:

[0087] For cross-subject scenarios: the target domain sample set of the model is U. T ={X i}, where X i This represents the feature data set of the i-th subject; x j X represents i The j-th sample in X, where n represents X i The total number of samples in the model; the multi-source domain sample set of the model is Where [N]_i represents the set of all subject indices after removing the data of the i-th subject, and P_i represents the set of all subject indices. j This represents the set of subject serial numbers contained in the j-th source domain (*all data across subject contexts originate from the same session).

[0088] For scenarios spanning multiple time periods: the target domain sample set of the model is Where X i Let represent the feature data set of the i-th subject; j represents the j-th session (time period). The multi-source domain sample set of the model is . Where [D] / j represents the set of all session numbers after removing the j-th session.

[0089] S3-1-2: Data Input for the Model

[0090] like Figure 3 The left half, that is Figure 4 As shown, each labeled source domain sample set U S and unlabeled target domain samples UT The set of samples forms a branch for training the subsequent single-source-domain teacher model. For the student model, only the unlabeled target-domain sample set U is used. T .

[0091] S3-2: Model Initialization

[0092] like Figure 3 As shown in the right half, MS-KFT consists of two parts: a teacher model based on a single source domain ( Figure 3 (top right corner) and student model that only applies to the target domain ( Figure 3 (Bottom right corner). Both the teacher and student models are derived from a domain-specific feature extractor N. f And label classifier N y These two modules consist of N, where the feature extractor is... f For detailed structure, please refer to Figure 2 Label classifier N y It consists of a single linear layer and a softmax function.

[0093] S3-3: Pre-training of a single-source-domain teacher model

[0094] like Figure 5 The diagram shown is a structural diagram of multiple single-source-domain teacher models, with N feature extractors for each specific domain. f and domain-specific label classifier N y To form a single-source domain teacher model.

[0095] Based on multiple source domain sample sets U S Each feature extractor N of a domain-specific teacher model is pre-trained using a single set of labeled source domain samples. f And label classifier N y This enables them to have a certain pattern recognition capability in their respective source domains (the optimization objective is the same as SCL in formula (5) below, which will not be elaborated here).

[0096] S3-4: Training of the Feature Extractor for the Single-Source-Domain Teacher Model

[0097] After passing through the feature extractor N of the domain-specific teacher model f Then, the corresponding source domain data U S and target domain data U T Their respective low-dimensional features F will be extracted. S F T To ensure the unbiased adaptability of the extracted features, this patent uses two methods: domain-level distribution alignment and data pair-level distribution alignment.

[0098] S3-4-1: Domain-level Distribution Alignment

[0099] correspond Figure 5 This patent achieves domain adaptation of the teacher model based on two techniques: pseudo-label triplet loss and maximum mean difference, with no offset in the distribution alignment.

[0100] Maximum Mean Difference (MMD) is a distance metric in a probability metric space, widely used in machine learning and nonparametric testing. This distance metric is based on the idea of ​​embedding probabilities into a Reproducing Kernel Hilbert Space (RKHS), aiming to reduce the distributional difference between the source and target domains while preserving their specific discriminative information. During training, minimizing the MMD loss reduces the distance between the source and target domains in the feature space, thereby achieving domain-level alignment. The specific formula is shown below:

[0101]

[0102] in and N represents the low-dimensional features extracted from the i-th sample in the source domain and the j-th sample in the target domain, respectively; S and N T This indicates the number of samples in the source domain and the number of samples in the target domain.

[0103] S3-4-2: Alignment of data distribution at levels

[0104] Because MMD blindly estimates parameters to account for statistical information and their relationships, feature discriminability may decrease, and the relationship between intra-class and inter-class distances may be affected, as one distance value decreases while the other increases. Triple loss, which can reduce intra-class distances and increase inter-class distances, is one way to address this issue; however, in domain adaptation, the target domain is often unlabeled. Therefore, this patent uses edge-sampling-based triple loss for distribution alignment at the data pair level.

[0105] This patent uses the marginal score of each sample's prediction result as the basis for determining whether the sample should be sampled. This method can be expressed by the following formula:

[0106]

[0107] X selected ={x j |margin(x j )≥Threshold,x j ∈X} (3)

[0108] Where x is the input sample, g θ It is an abstract function of the label classifier, i *It is the category with the highest predicted probability in the prediction results, where k is the number of all categories, [k]\i * Indicates excluding i * The threshold is a pre-defined threshold for edge sampling, which is the set of all categories outside of the standard.

[0109] Using triplet loss requires triplet loss. Sampling is performed in the form of , where (Anchor point sample) and (Positive examples) are different samples of the same class from the i-th triplet. It is the sample with anchor point in the i-th triplet. Any samples from different categories. The purpose of triplet loss is to ensure that positive samples are correct. The distance between embedded features plus a fixed margin is less than that of negative sample pairs. The distance between embedded features. Formally, for a mini-batch sample set, the triplet loss is defined as:

[0110]

[0111]

[0112]

[0113] Where N is X selected The number of samples included, α is a pre-set marginal value used to guide separability, d(·) is a function to calculate the Euclidean distance between regularized embedded feature pairs, and f θ (·) is an abstract function for feature extraction.

[0114] S3-5: Training of the Single-Source-Domain Teacher Model Label Classifier

[0115] This patent uses cross-entropy (CE) loss as an evaluation metric for the classification results of the label classifier in the source and target domains. Specifically, SCL is used as the classification loss in the source domain, and TCL is used as the classification loss in the target domain.

[0116] In the source domain, there are true labels, so SCL uses the true labels and the classification results of the label classifier as comparison objects, as shown in the following formula:

[0117]

[0118]

[0119] Where x i It is the i-th source domain input sample. It is the true label of the i-th source domain input sample. f is the prediction result of the label classifier for the i-th source domain input sample. θ (·) is an abstract function for feature extraction, g θ (·) is an abstract function of the label classifier.

[0120] In the target domain, the samples lack true labels, so the corresponding TCL uses the generated pseudo-labels and the classification results of the label classifier as the comparison objects. The specific formula is as follows:

[0121]

[0122]

[0123]

[0124] Where x i It is the i-th target domain input sample. It is the pseudo-label generated from the i-th target domain input sample. f is the prediction result of the label classifier for the i-th target domain input sample. θ (·) is an abstract function for feature extraction, g θ (·) is an abstract function of the label classifier.

[0125] S3-6: Optimization Objectives and Training of the Single-Source-Domain Teacher Model

[0126] Summarizing S3-4 and 3-5, the final optimization objective in the domain adaptation phase of the teacher model is shown in the following formula:

[0127]

[0128] Where β, γ, and σ are weighting factors used to balance the loss function.

[0129] Using the stochastic gradient (SGD) optimizer combined with mini-batch training, the MMD loss and triplet loss in equation (7) are minimized. Domain-invariant features can be obtained at both the domain and data pair levels for each source and target domain pair. The classification loss is minimized in both the source and target domains. This will result in a superior classifier that can accurately predict source domain samples without sacrificing the ability to distinguish target domain samples.

[0130] S3-7: Training the Student Model

[0131] The specific structure of the student model can be seen here. Figure 6After domain adaptation of the teacher model, multiple single-source domain models were obtained. These models can effectively extract deep EEG pattern representations that are discriminative for classification tasks but transferable across different domains. To maximize the use of these single-source models, knowledge distillation was used to transfer knowledge learned from multiple domains and train a more powerful student model.

[0132] To better integrate the knowledge from the teacher models, this patent uses a voting-based method to select the teacher model knowledge to be integrated, which can be expressed as the following formula:

[0133]

[0134]

[0135]

[0136] Where x i It is the i-th input sample, N t This represents the number of teacher models, mode(·) is a function used to find the mode / multiple modes, and * is the dot product function. It is the prediction result of the j-th teacher model for the i-th input sample. It is the set of decision labels used to generate the teacher model mask for the i-th input sample.

[0137] After obtaining the combined knowledge from multiple single-source domain teacher models, the Kullback-Leibler (KL) divergence is used to evaluate the difference between the teacher model predictions and the student model predictions, as shown in the following formula:

[0138]

[0139] Where X is the input sample set, merge is the merged set of teacher knowledge, T is a pre-set temperature coefficient used to control the smoothness of the softmax function, and KLD[p, q] is an evaluation function used to measure the KL divergence between distributions p and q.

[0140] By using the Adam optimizer and combining it with mini-batch training, the KL loss in formula (9) is minimized, enabling the student model to fully learn the merged knowledge of the teacher model and achieve superior performance in the target domain.

[0141] S4: Model performance evaluation in two scenarios: across time periods and across participants.

[0142] This invention specifically verifies the model performance on the SEED and SEED-IV datasets.

[0143] The prediction result y obtained by the converged student model in the target domain pred and the true label y in the target domain T The confusion matrix is ​​used for comparison, and the comparison results are obtained to evaluate the model performance. Accuracy is the ratio of the number of correctly classified samples to the total number of test samples. The formula for calculating model accuracy is as follows:

[0144]

[0145] Where TP represents positive samples predicted as positive by the model, TN represents negative samples predicted as negative by the model, FP represents negative samples predicted as positive by the model, and FN represents positive samples predicted as negative by the model. The SEED dataset includes 15 participants, each participating in three trials, for a total of 45 trials. The average accuracy of the first two trials for the 15 participants is shown below:

[0146]

[0147] The formula for the mean square error of the results is shown below:

[0148]

[0149] The specific dataset partitioning for the two scenarios—cross-time period and cross-subject—is detailed in S3-1-1. For the cross-subject scenario, the model proposed in this invention was tested on EEG data from 15 subjects in one trial; for the cross-time period scenario, the model proposed in this invention was tested on EEG data from 15 subjects in one test. The final test results are compared with existing technologies (SVM, DGCNN, and RGNN) in the table below:

[0150] Table 1 shows the classifier performance comparison on the SEED dataset.

[0151] Classifier DDC DAN MS-MDA This invention Cross-time accuracy 81.53 / 6.83 79.93 / 7.06 88.56 / 7.80 97.58 / 1.46 Cross-subject accuracy 68.99 / 3.23 65.84 / 2.25 89.63 / 6.79 91.73 / 10.48

[0152] Table 2 shows the classifier performance comparison on the SEED-IV dataset.

[0153] Classifier DDC DAN MS-MDA This invention Cross-time accuracy 57.63 / 11.28 55.14 / 12.79 61.43 / 15.71 77.70 / 13.49 Cross-subject accuracy 37.71 / 6.36 32.44 / 9.02 59.34 / 5.48 74.19 / 15.84

[0154] As can be seen from the results in the table above, the method proposed in this invention achieves higher accuracy than DDC, DAN, and MS-MDA in both cross-time and cross-subject scenarios. This invention is not only applicable to research on emotion state recognition but also to any EEG-based cross-time and cross-subject classification and prediction task, thus addressing the issue of individual variability in EEG data to some extent.

Claims

1. A multi-source domain adaptation EEG emotion state classification method based on knowledge distillation, characterized in that, This method uses differential entropy features as the frequency domain features of the EEG signals, an improved EEGNet model as the feature extractor, and a single-layer linear layer as the classifier to analyze the EEG signals, achieving emotion state recognition tasks in both cross-subject and cross-time period scenarios. The specific process includes: I. Pre-train each teacher model based on each labeled source domain; II. Based on the corresponding labeled source domain and unlabeled target domain, domain adaptation is performed on each teacher model using source domain classification loss, target domain classification loss, maximum mean difference, and pseudo-label triplet loss. To improve the effectiveness of the pseudo-label triplet loss, a marginal sampling strategy is used to filter the original features, selecting only features with marginal scores higher than a preset threshold as the embedding features for calculating the pseudo-label triplet loss. The domain adaptation for each teacher model includes domain adaptation of the feature extractor for each teacher model, as detailed below: After passing through the feature extractor N of the domain-specific teacher model f Then, the corresponding source domain data U S and target domain data U T Their respective low-dimensional features F will be extracted. S F T To ensure the unbiased adaptability of the extracted features, two methods were used: domain-level distribution alignment and data pair-level distribution alignment. S3-4-1: Domain-level Distribution Alignment Domain adaptation of the teacher model is achieved using two techniques: pseudo-label triplet loss and maximum mean difference (MMD). During training, the distance between the source and target domains in the feature space is reduced by minimizing the MMD loss, thereby achieving domain-level alignment. The specific formula is shown below: in and N represents the low-dimensional features extracted from the i-th sample in the source domain and the j-th sample in the target domain, respectively; S and N T This indicates the number of samples in the source domain and the number of samples in the target domain. S3-4-2: Alignment of data distribution at levels Data pair level distribution alignment is performed using a triplet loss based on edge sampling. The edge score of each sample's prediction result is used as the criterion for deciding whether to sample that sample. This process is expressed by the following formula: X selected ={x j |margin(x j )≥Threshold,x j ∈X} (3) Where x is the input sample, g θ It is an abstract function of the label classifier, i * It is the category with the highest predicted probability in the prediction results, j is the number of all categories, and [k]\i * Indicates excluding i * The threshold for edge sampling is a pre-defined threshold for all categories outside the set of categories. Using triplet loss requires triplet loss. Sampling is performed in the form of anchor point samples. and positive samples These are different samples of the same class from the i-th triplet. It is the sample with anchor point in the i-th triplet. Any samples from different categories; the purpose of triplet loss is to ensure that positive samples are paired. The distance between embedded features plus a fixed margin is less than that of negative sample pairs. The distance between embedded features; for a mini-batch sample set, the triplet loss is defined as: Where N is X selected The number of samples included, α is a pre-set marginal value used to guide separability, d(·) is a function to calculate the Euclidean distance between regularized embedded feature pairs, and f θ (·) is an abstract function for feature extraction; III. Transfer knowledge from multiple single-source domain teachers to the student model.

2. The EEG emotion state classification method based on knowledge distillation and multi-source domain adaptation according to claim 1, characterized in that, This method involves two testing scenarios for emotion state classification: across subjects and across time periods. The model testing in these two scenarios has different data definitions and dataset partitioning, as detailed below: Assume there are N subjects, each of whom participated in D different conversations; the entire sample set is represented as... Where i represents the subject number, j represents the conversation number, and X i Let Y represent the sample set of subject i, and the corresponding label set be Y. i ; For the cross-time-period sentiment state classification task: leave-one-out method is used to cross-validate the dataset; specifically, in each subject i, the data of 15 sentiment trials of all subjects in the latest session are used as the test set; the remaining D-1 sessions are used as a unit, and each session is used as a source domain in the training set, finally obtaining D-1 source domains as the training set; a total of N experiments are conducted and the average accuracy is calculated. For the cross-subject emotional state classification task: leave-one-out cross-validation of the dataset is used; specifically, in a session, data from all 15 emotional trials of a subject are iteratively extracted and their emotional state labels are assumed to be unknown, serving as the test set; from the remaining N-1 subjects, R subjects are randomly and non-repeatingly grouped into a set as a source domain in the training set, ultimately obtaining... Using one source domain as the training set, a total of D×N experiments were conducted and the average accuracy was calculated.

3. The EEG emotion state classification method based on knowledge distillation and multi-source domain adaptation according to claim 1, characterized in that, The construction and training of the MS-KTF model in the implementation of this method are as follows: S3-1: Initialization; The MS-KFT model consists of two parts: a teacher model based on a single source domain and a student model operating on the target domain. Both the teacher and student models are derived from a domain-specific feature extractor N. f And label classifier N y These two modules consist of initializing the parameters of multiple single-source-domain teacher models based on a multi-source-domain model and a target-domain student model. S3-2: Pre-train multiple single-source-domain teacher models; Based on multiple source domain sample sets, a feature extractor N for a domain-specific teacher model is pre-trained using a single labeled source domain sample. f And label classifier N y This enables them to have a certain pattern recognition capability in their respective source domains; S3-3: Domain adaptation of feature extractors for multiple single-source domain teacher models; Each labeled source domain sample and unlabeled target domain sample form a branch. In each branch, the feature extractor N of the corresponding domain-specific teacher model is used. f Feature extraction is performed on the source domain samples and target domain samples respectively, and the features are extracted from the original feature space into the embedding space; Then, the embedded features are aligned at the domain level in the feature space based on the maximum mean difference; the embedded features are aligned at the data group level based on the pseudo-label triple loss based on edge sampling. The feature extractor N is trained by minimizing the maximum mean difference and pseudo-label triplet loss for multiple single-source-domain teacher models. f This enables domain-invariant feature extraction in both the source and target domains; S3-4: Training a label classifier N from multiple single-source-domain teacher models y In each single-source-domain teacher model, the extracted source-domain feature information is processed by the label classifier N. y Obtaining predictive sentiment And calculate predicted sentiment Corresponding label Y in the actual sample S The cross-entropy; similarly, calculating the predicted sentiment based on the feature information of the target domain. The pseudo tags generated by it Cross-entropy; By minimizing the two obtained cross-entropies, a label classifier N for a multi-unit domain teacher model is trained. y This enables them to have good emotion classification capabilities in their respective source and target domains; S3-5: Merge the knowledge from multiple single-source-domain teacher models; S3-6: Teach the merged knowledge of the teacher model to the student model.

4. The EEG emotion state classification method based on knowledge distillation and multi-source domain adaptation according to claim 3, characterized in that... To balance the performance of teacher models, two different merging strategies are used: ① The teacher model to be merged is selected based on a voting process; This approach is more suitable for situations where teacher performance is poorly balanced; it uses unlabeled target domain samples and first obtains sentiment prediction results through the feature extractor and label classifier of the teacher model. Generate the corresponding one-hot encoding result Based on the one-hot coding results generated by each teacher Conduct a vote and use the voting results as a decision variable. If the teacher model's emotion prediction results With decision variables If the results are the same, the teacher will be selected for knowledge merging. Calculate the average of the sentiment predictions for all selected teacher models. As a merged knowledge of all teacher models; ② Use an average approach to merge the teacher models; This approach is more suitable for situations where teacher performance is well-balanced; in this case, all teacher models have the same weights, and the average of the predictions from all selected teacher models is calculated. As a merged knowledge of all teacher models.

5. The EEG emotion state classification method based on knowledge distillation and multi-source domain adaptation according to claim 3, characterized in that... The process of teaching the merged knowledge from the teacher model to the student model is as follows: Using unlabeled target domain sample data, the prediction results of the student model are obtained through the feature extractor and label classifier of the student model. Based on a preset distillation temperature, the merged knowledge of the teacher model is used. Predictions from the student model Smoothing is performed; KL divergence is used to evaluate the difference between the two predictions; By minimizing the KL divergence between the teacher and student models, the student model learns the knowledge of the teacher model, gaining more generalizable feature extraction and label classification capabilities in the target domain compared to the teacher.

6. The EEG emotion state classification method based on knowledge distillation and multi-source domain adaptation according to claim 3, 4, or 5, characterized in that... The initialization process is implemented as follows: S3-1-1: Data partitioning of the model, the specific partitioning is described as follows: For cross-subject scenarios: the target domain sample set of the model is U. T ={X i }, where X i This represents the feature data set of the i-th subject; x j X represents i The j-th sample in X, where n represents X i The total number of samples in the model; the multi-source domain sample set of the model is Where [N]_i represents the set of all subject indices after removing the data of the i-th subject, and P_i represents the set of all subject indices. j This represents the set of subject serial numbers contained in the j-th source domain; For scenarios spanning multiple time periods: the target domain sample set of the model is Where X i Let represent the feature data set of the i-th subject; j represents the j-th session; the multi-source domain sample set of the model is . Where [D] / j represents the set of all session numbers after removing the j-th session; S3-1-2: Data Input for the Model Each labeled source domain sample set U S and unlabeled target domain samples U T The set forms a branch, which is used for the subsequent training of the single-source-domain teacher model; For the student model, only the unlabeled target domain sample set U is used. T .

7. A multi-source domain adaptation EEG emotion state classification method based on knowledge distillation according to claim 3, 4, or 5, characterized in that... Cross-entropy loss is used as an evaluation metric for the classification results of the label classifier in the source and target domains. Specifically, SCL is used as the classification loss in the source domain, and TCL is used as the classification loss in the target domain. In the source domain, there are true labels, so SCL uses the true labels and the classification results of the label classifier as comparison objects, as shown in the following formula: Where x i It is the i-th source domain input sample. It is the true label of the i-th source domain input sample. f is the prediction result of the label classifier for the i-th source domain input sample. θ (·) is an abstract function for feature extraction, g θ (·) is an abstract function of the label classifier; In the target domain, the samples lack true labels, so the corresponding TCL uses the generated pseudo-labels and the classification results of the label classifier as the comparison objects. The specific formula is as follows: Where x i It is the i-th target domain input sample. It is the pseudo-label generated from the i-th target domain input sample. f is the prediction result of the label classifier for the i-th target domain input sample. θ (·) is an abstract function for feature extraction, g θ (·) is an abstract function of the label classifier.

8. A multi-source domain adaptation EEG emotion state classification method based on knowledge distillation according to claim 3, 4, or 5, characterized in that... The optimization objective and training of the single-source-domain teacher model are as follows: In the domain adaptation phase of the teacher model, the final optimization objective is shown in the following formula: Where β, γ, and σ are weighting factors used to balance the loss function; Using a stochastic gradient optimizer combined with mini-batch training, the MMD loss and triplet loss in equation (7) are minimized. Obtain domain-invariant features at both the domain level and the data pair level for each source and target domain pair; minimize the classification loss in both the source and target domains. This will result in a superior classifier that can accurately predict source domain samples without sacrificing the ability to distinguish target domain samples.

9. A multi-source domain adaptation EEG emotion state classification method based on knowledge distillation according to claim 3, 4, or 5, characterized in that... The training of the student model is as follows: After domain adaptation of the teacher model, multiple single-source domain models were obtained. To better merge the knowledge from the teacher models, a voting-based approach was used to select the teacher model knowledge to be merged, expressed as the following formula: Where x i It is the i-th input sample, N t This represents the number of teacher models, mode(·) is a function used to find the mode / multiple modes, and * is the dot product function. It is the prediction result of the j-th teacher model for the i-th input sample. It is the set of decision labels used to generate the teacher model mask for the i-th input sample; After obtaining the combined knowledge from multiple single-source domain teacher models, KL divergence is used to evaluate the difference between the teacher model predictions and the student model predictions, as shown in the following formula: Where X is the input sample set, merge is the merged set of teacher knowledge, T is a pre-set temperature coefficient used to control the smoothness of the softmax function, and KLD[p,q] is an evaluation function used to measure the KL divergence between distributions p and q. By using the Adam optimizer and combining it with mini-batch training, the KL loss in formula (9) is minimized, enabling the student model to fully learn the merged knowledge of the teacher model and achieve superior performance in the target domain.

Images