An emotion recognition method based on self-training to maximize classifier differences

Abstract

The present invention belongs to the field of emotion recognition, and specifically relates to an emotion recognition method based on self-training to maximize classifier differences; the method comprises: acquiring electroencephalogram (EEG) data of a subject in real time, preprocessing the EEG data, inputting the preprocessed data into an emotion recognition model based on self-training to maximize classifier differences, and obtaining the emotion recognition result of the subject; the present invention proposes a new domain adaptive adversarial training method, which uses soft labels for cross-disciplinary EEG emotion classification. The method not only considers the decision boundary of a specific category, but also uses soft labels to further extract source domain information that is helpful for classifying a target domain, so that the emotion recognition result is more accurate and has good economic benefits.

Description

Technical Field

[0001] The invention belongs to the field of emotion recognition, and in particular relates to an emotion recognition method based on self-training to maximize classifier differences. Background Art

[0002] In recent years, emotion recognition has become an important issue in scientific research and engineering, and has received widespread attention from scholars at home and abroad. In psychological research, emotion recognition provides a reference indicator for studying emotion-related behaviors. In engineering research, it promotes more friendly human-computer interaction, enabling machines to recognize, process and interpret human emotions through systems or intelligent devices. In the medical field, emotion recognition helps to assist in the diagnosis and treatment of various mental illnesses such as autism or depression, allowing medical staff to accurately know the emotional state of patients with expression disorders and provide patients with high-quality services and care measures. In the field of entertainment and consumption, video websites and advertising companies obtain audience emotional information to judge audience preferences and recommend personalized videos and music.

[0003] At present, there are many methods to identify the psychological emotions of users, such as facial expressions, human posture and voice intonation. However, these emotional information based on non-physiological signals can be changed by human control and have certain deception. Some emotion recognition methods based on physiological signals include electroencephalogram (EEG), electromyogram (EMG), electrocardiogram (ECG), functional near-infrared spectroscopy (FNIRS), etc.; since people cannot control the changes in their own physiological signals, emotion recognition based on physiological signals appears to be more reliable. Among them, EEG has the highest correlation with the mental state of the human body, so brain signals are considered to be more suitable for determining the physiological signals of the subject's emotional fluctuations. At present, many scholars have conducted research on emotion recognition technology based on EEG in terms of feature extraction, channel selection and classification algorithms, and have achieved remarkable results. In terms of feature extraction, researchers extract features from EEG emotion data in the time domain, frequency domain and time-frequency domain. For example, in the prior art, there is a method that compares frequency domain features such as energy spectrum (ES) and differential entropy (DE), and uses a linear dynamic system to smooth the features. This method has achieved remarkable results and has been widely used; a method that uses a long short-term memory network (LSTM) to automatically extract time features. In terms of channel selection, there is a method that uses different neighbor weights for emotion recognition, taking into account the asymmetry of neuronal activity between the left and right hemispheres of the brain. In terms of classification algorithms, traditional machine learning and deep learning have been widely used in EEG-based emotion recognition research and have achieved remarkable results. A method that uses differential entropy in different channels as features and uses a hierarchical convolutional neural network (HCNN) to classify emotions into three categories has achieved a good recognition rate and proved that beta and gamma rhythms contain the richest emotional information.

[0004] Due to the strong individual differences and instability of EEG, it is difficult for the trained model to be applied well on new subjects, and the model effect of the previously trained model on the same subject will become worse and worse over time. A direct solution is to use a large amount of data to train a specific model for each subject, but this requires a lot of manpower, material resources and time to collect labeled data sets. Another solution is to use domain adaptation methods. Domain adaptation methods can transfer knowledge from a domain with rich labels (i.e., source domain) to a domain with scarce labels (i.e., target domain). For example, a method that applies style transfer mapping (STM) to multi-source domain cross-user emotion recognition based on EEG. In this method, a small amount of labeled data of the target domain subjects is used for data calibration, which is a semi-supervised transfer method. Experimental results show that compared with the method without domain adaptation, the classification accuracy of this method is improved by 12.72%; a method using domain adversarial neural network (DANN) can eliminate the differences between subjects and achieve significant improvement in classification effect; a method based on adversarial neural network using local attention mechanism and global attention mechanism can adaptively learn transferable brain areas and samples, thereby learning the discriminative information of emotions.

[0005] Although the above methods have achieved remarkable results, the domain adaptation method directly uses the features of the source domain and the target domain for complete matching without considering the category of the sample, resulting in poor domain adaptation results. The main reason is that it is very difficult to completely align the distribution of the source domain and the target domain, especially when faced with EEG source domain data integrated from multiple subjects. In addition, the method of directly using artificially extracted features ignores the relationship between frequency bands and the spatial information between channels, and cannot deeply explore the relationship between brain regions and frequency bands. In emotion recognition based on EEG signals, the signal strength of different frequency bands under different emotions is different and the areas of brain activity are also different. Most studies directly input artificially extracted features into the classifier, ignoring the relationship between frequency bands and the spatial information between channels, which reduces the classification effect. Summary of the invention

[0006] In view of the shortcomings of the prior art, the present invention proposes an emotion recognition method based on self-training to maximize classifier differences, the method comprising: acquiring EEG data of a subject in real time, preprocessing the EEG data, inputting the preprocessed data into an emotion recognition model based on self-training to maximize classifier differences, and obtaining an emotion recognition result of the subject;

[0007] The process of training the emotion recognition model based on self-training to maximize the classifier difference includes:

[0008] S1: Divide the preprocessed EEG data into 1s and 4s segments respectively; wherein the EEG data includes known subject data and new subject data;

[0009] S2: Filter all the segments in 5 different frequency bands to obtain filtered data;

[0010] S3: Use differential entropy to extract features from the filtered data to obtain differential entropy features;

[0011] S4: Project the differential entropy feature into a 2D plane feature to obtain five 2D plane features; stack the five 2D planes together to form a 3D cube feature;

[0012] S5: The cube features X of the known subjects S and its emotion label Y S The source domain (X S ,Y S ), the cube feature X of the new subject T and its emotion label Y T The target domain (X T ,Y T );

[0013] S6: The first training of the emotion recognition model based on self-training to maximize the classifier difference according to the source domain;

[0014] S7: The emotion recognition model trained for the first time based on self-training to maximize the classifier difference is trained for the second time according to the target domain;

[0015] S8: The emotion recognition model trained for the second time based on the self-training maximization classifier difference is trained for the third time according to the source domain and the target domain;

[0016] Among them, the emotion recognition model based on self-training to maximize the classifier difference includes a feature extractor F, a classifier C1 and a classifier C2.

[0017] Preferably, the first training of the emotion recognition model based on self-training maximizing classifier differences according to the source domain includes: inputting the source domain into the emotion recognition model based on self-training maximizing classifier differences to obtain predicted emotion labels of known subjects; continuously adjusting the parameters of the model according to the predicted emotion labels of the known subjects and the emotion labels of the known subjects; when the first training loss function is minimized, the first training is completed.

[0018] Furthermore, the loss function formula for the first training is:

[0019]

[0020]

[0021] Among them, L(X S ,Y S ) represents the first training loss function, n s represents the number of known subjects, l CE represents the cross entropy loss function, F(x i ) represents the features generated by the feature generator, C1(F(x i )) represents the probability output of classifier C1, C2(F(x i )) represents the probability output of classifier C2.

[0022] Preferably, the second training of the emotion recognition model based on self-training to maximize the classifier difference includes: constructing a soft label set according to the target domain, and obtaining soft labels according to the soft label set; replacing the emotion labels in the target domain with soft labels, and inputting the target domain after replacing the labels into the emotion recognition model based on self-training to maximize the classifier difference to obtain new soft labels; adjusting the model parameters according to the soft labels and the new soft labels, and the second training is completed when the loss function is minimized.

[0023] Furthermore, the new soft label is expressed as:

[0024]

[0025]

[0026] in, represents the soft label probability distribution output by classifier C1, represents the soft label probability distribution of the output of classifier C2, C1(F(x i )) represents the probability output of classifier C1, C2(F(x i )) represents the probability output of classifier C2.

[0027] Preferably, the third training of the emotion recognition model based on self-training maximizing classifier differences according to the source domain and the target domain includes: inputting the source domain and the target domain into the emotion recognition model based on self-training maximizing classifier differences, fixing the feature extractor, training only two classifiers, minimizing the KL divergence between soft labels and new soft labels, and completing the training of the fixed feature extractor when the loss function of the fixed feature extractor is minimized; fixing the two classifiers, training only the feature extractor, minimizing the KL divergence between soft labels and new soft labels, and completing the training of the fixed two classifiers when the loss function of the fixed two classifiers is minimized; and correcting the emotion recognition model based on self-training maximizing classifier differences according to the target domain, and completing the model training when the corrected loss function is minimized.

[0028] Furthermore, the loss function formula of the fixed feature extractor is:

[0029]

[0030]

[0031]

[0032] Among them, L dis (X T ) represents the classification difference between two classifiers for the same feature, L soft (X T ) represents the probability distribution difference between the new soft label and the soft label, n l represents the size of the corrected dataset in the target domain, n u represents the size of the validation dataset in the target domain, K represents the number of categories, and F(x i ) means that C1(F(x i )) represents the probability output of classifier C1, C2(F(x i )) represents the probability output of classifier C2, C1(F(x i )) k represents the probability output of the kth class of classifier C1, C2(F(x i )) k represents the probability output of the kth class of classifier C2; l KL represents the KL divergence, represents the soft label probability distribution output by classifier C1, Represents the soft label probability distribution output by classifier C2.

[0033] Furthermore, the loss function formula for fixing the two classifiers is:

[0034]

[0035]

[0036]

[0037] Among them, L dis (X T ) represents the classification difference between two classifiers for the same feature, L soft (X T ) represents the probability distribution difference between the new soft label and the soft label, n l represents the size of the corrected dataset in the target domain, n u represents the size of the validation dataset in the target domain, K represents the number of categories, and F(x i ) represents the features generated by the feature generator, C1(F(xi )) represents the probability output of classifier C1, C2(F(x i )) represents the probability output of classifier C2, C1(F(x i )) k represents the probability output of the kth class of classifier C1, C2(F(x i )) k represents the probability output of the kth class of classifier C2; l KL represents the KL divergence, represents the soft label probability distribution output by classifier C1, Represents the soft label probability distribution output by classifier C2.

[0038] The beneficial effects of the present invention are as follows: the present invention uses a convolutional neural network to deeply mine the relationship between brain regions and frequency bands, and combines frequency band information with channel space information to perform end-to-end automatic deep feature extraction, which is more conducive to the recognition of EEG emotions; a new domain adaptive adversarial training method is proposed, and soft labels are used for cross-disciplinary EEG emotion classification; this method not only considers the decision boundary of a specific category, but also maximizes the difference between two classifiers to achieve the effect of being able to detect target domain samples that are not supported in the source domain, and uses soft labels to further extract source domain information that is helpful for classifying the target domain, so that the emotion recognition result is more accurate and has good economic benefits. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 This is a diagram of the 3D cube feature extraction process in the present invention;

[0040] Figure 2 It is the structural diagram of the SMCD model in the present invention;

[0041] Figure 3 It is a test result diagram of the SMCD model and other models in the present invention. DETAILED DESCRIPTION

[0042] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0043] The present invention proposes an emotion recognition method based on self-training to maximize classifier differences, the method comprising: acquiring EEG data of a subject in real time, preprocessing the EEG data, inputting the preprocessed data into an emotion recognition model based on self-training to maximize classifier differences, and obtaining an emotion recognition result of the subject;

[0044] The process of training the emotion recognition model based on self-training to maximize the classifier difference includes:

[0045] S1: Divide the preprocessed EEG data into 1s and 4s segments respectively; wherein the EEG data includes known subject data and new subject data;

[0046] S2: Filter all the segments in 5 different frequency bands to obtain filtered data;

[0047] S3: Use differential entropy to extract features from the filtered data to obtain differential entropy features;

[0048] S4: Project the differential entropy feature into a 2D plane feature to obtain five 2D plane features; stack the five 2D planes together to form a 3D cube feature;

[0049] S5: The cube features X of the known subjects S and its emotion label Y S The source domain (X S ,Y S ), the cube feature X of the new subject T and its emotion label Y T The target domain (X T ,Y T );

[0050] S6: The first training of the emotion recognition model based on self-training to maximize the classifier difference according to the source domain;

[0051] S7: The emotion recognition model trained for the first time based on self-training to maximize the classifier difference is trained for the second time according to the target domain;

[0052] S8: The emotion recognition model trained for the second time based on the self-training maximization classifier difference is trained for the third time according to the source domain and the target domain;

[0053] Among them, the emotion recognition model based on self-training to maximize the classifier difference includes a feature extractor F, a classifier C1 and a classifier C2.

[0054] like Figure 1As shown, the present invention designs an emotion recognition model SMCD based on self-training maximization of classifier differences, SMCD includes a feature extractor F, a classifier C1 and a classifier C2; the feature extractor F is a convolutional neural network (CNN) feature extractor, which constructs the artificially extracted EEG features into a 3D cube (similar to a multi-channel color image), that is, a 3D cube feature; a feature generator is used to extract new features from the cube features, preferably, the feature generator includes 4 convolutional layers and a fully connected layer; in order to obtain more accurate emotion recognition results, the present invention uses two classifiers, because the neural network initialization parameters may be different, after training, the two classifiers predict different probability distributions for the same input feature, and the samples in the middle of each category in the source domain may output the same result, and the samples at the edge of each category may output different results, thereby detecting samples at the decision boundary of the source domain, so that the emotion recognition result is more accurate; the structural parameters of the feature generator (F) in the SMCD model are shown in Table 1, and the structural parameters of the fully connected layers of the two classifiers (C1 and C2) are shown in Table 2, where K is the number of categories.

[0055] Table 1. Feature generator (F) structural parameters

[0056]

[0057] Table 2 Fully connected layer structure parameters

[0058]

[0059] A specific implementation of training an emotion recognition model based on self-training to maximize classifier differences is as follows:

[0060] The process from the original EEG data set to the final construction of the 3D cube feature is as follows Figure 2 As shown, the specific process is:

[0061] The original EEG data was preprocessed: the original EEG data on the SEED and SEED-IV datasets were downsampled from 1000Hz to 200Hz; a 0.3-75Hz bandpass filter was used to further filter out noise and eliminate artifacts. The non-overlapping EEG signals after filtering each channel on the preprocessed SEED and SEED-IV datasets were divided into 1s and 4s segments respectively; these segments have their corresponding emotion labels.

[0062] The obtained fragments are filtered in five frequency bands; the five frequency bands are: delta (1-3Hz), theta (4-7Hz), alpha (8-13Hz), beta (14-30Hz) and gamma (31-50Hz); differential entropy (DE) features are extracted from the five frequency bands to obtain differential entropy features, and the feature dimension of each fragment is 62x5=310.

[0063] The differential entropy formula is:

[0064]

[0065] Where h(x) represents the obtained differential entropy, x represents the input brain signal, μ represents the mean of the input brain signal, and σ 2 Represents the variance of the input brain signal.

[0066] The international 10-20 system is used to measure EEG. The obtained 1DDE features are projected into a 9x9 2D plane feature according to the electrode coordinate distribution of the international 10-20 system. Preferably, the number of electrodes is 62; electrodes without data are filled with 0; the 5 2D planes constructed by the 5 frequency bands are stacked together to form a 3D cube feature. The cube feature is divided into known subject features and new subject features. The cube feature X of the known subject is S and its emotion label Y S The source domain (X S ,Y S ), the cube feature X of the new subject T and its emotion label Y T The target domain (X T ,Y T ), where the target domain (X T ,Y T ) can be expressed as:

[0067]

[0068] in is the new subject corrected data, is the new subject validation data.

[0069] The emotion recognition model based on self-training maximizing classifier differences is trained for the first time according to the source domain to obtain the emotion recognition model based on self-training maximizing classifier differences that has been trained for the first time; the training process is: inputting the source domain into the emotion recognition model based on self-training maximizing classifier differences to obtain the predicted emotion labels of the known subjects; continuously adjusting the parameters of the model according to the predicted emotion labels of the known subjects and the emotion labels of the known subjects; when the loss function of the first training is minimized, the first training is completed; the formula of the loss function of the first training is:

[0070]

[0071]

[0072] Among them, L(X S ,Y S ) represents the first training loss function, n s represents the number of known subjects, l CE represents the cross entropy loss function, F(x i ) represents the features generated by the feature generator, C1(F(x i )) represents the probability output of classifier C1, C2(F(x i )) represents the probability output of classifier C2.

[0073] The emotion recognition model based on self-training maximizing classifier differences that has been trained for the first time is trained for the second time according to the target domain to obtain the emotion recognition model based on self-training maximizing classifier differences that has been trained for the second time. The second training includes:

[0074] A soft label set is constructed according to the target domain, and soft labels are obtained according to the soft label set; the emotion labels in the target domain are replaced with soft labels, and the target domain after replacing the labels is input into the emotion recognition model based on self-training to maximize the classifier difference to obtain new soft labels; the model parameters are adjusted according to the soft labels and the new soft labels, and the second training is completed when the loss function is minimized.

[0075] Construct a set of soft labels, which is represented by:

[0076]

[0077] Among them, Y T* Indicates a soft tag. represents the target domain features, The predicted soft label is the new soft label, n l represents the size of the corrected dataset in the target domain, n u Represents the size of the validation dataset in the target domain.

[0078] The new soft label is represented as:

[0079]

[0080]

[0081] in, represents the soft label probability distribution output by classifier C1, represents the soft label probability distribution of the output of classifier C2, C1(F(xi )) represents the probability output of classifier C1, C2(F(x i )) represents the probability output of classifier C2.

[0082] The emotion recognition model based on self-training maximizing classifier difference trained for the second time is trained for the third time according to the source domain and the target domain to obtain a trained emotion recognition model based on self-training maximizing classifier difference; the third training includes:

[0083] The present invention designs a classifier training method, which uses two classifiers to align the feature distributions of existing subjects and new subjects in an adversarial manner, while ensuring the emotion classification effect of the existing subjects; inputs a source domain and a target domain into an emotion recognition model based on self-training maximization of classifier differences, and in order to detect target domain samples that are not supported in the source domain, that is, to obtain the decision boundary of each category, a feature extractor is fixed, only two classifiers are trained, and the KL divergence between soft labels and new soft labels is minimized, so that the classification difference between the two classifiers on the target domain is maximized, and when the loss function of the fixed feature extractor is minimized, the training of the fixed feature extractor is completed; in order to make the features extracted by the feature extractor as similar as possible to the real features, the two classifiers are fixed, the feature extractors are repeatedly trained, the KL divergence between the soft labels and the new soft labels is minimized, so that the difference between the two classifiers on the target domain is minimized, and when the loss function of the fixed two classifiers is minimized, the training of the fixed two classifiers is completed; the emotion recognition model based on self-training maximization of classifier differences is corrected according to the target domain, and when the corrected loss function is minimized, the model training is completed.

[0084] The loss function formula for a fixed feature extractor is:

[0085]

[0086]

[0087]

[0088] Among them, L dis (X T ) represents the classification difference between two classifiers for the same feature, L soft (X T ) represents the probability distribution difference between the new soft label and the soft label, that is, the probability distribution difference between the prediction result and the soft label, F(x i ) represents the features generated by the feature generator, C1(F(x i )) k represents the probability output of the kth class of classifier C1, C2(F(x i )) k represents the probability output of the kth class of classifier C2; lKL represents the KL divergence.

[0089] The loss function formula of the two classifiers is fixed as follows:

[0090]

[0091]

[0092]

[0093] The KL divergence formula between the soft label and the new soft label is:

[0094]

[0095] Among them, R represents the probability distribution predicted by the classifier, Q represents the probability distribution of the soft label, R(i) represents the probability of the i-th category predicted by the classifier, Q(i) represents the probability of the i-th category of the soft label, and K represents the number of categories.

[0096] The corrected loss function is:

[0097]

[0098]

[0099] in, represents the characteristics of the corrected dataset, Represents the sentiment labels of the rectified dataset.

[0100] By inputting the validation set data in the target domain into the trained SMCD model for performance testing, the predicted emotion labels of new subjects, i.e., emotion recognition results, can be obtained. The recognition results of the trained model are highly accurate, and users can then use the trained SMCD model for real-time recognition.

[0101] The present invention is evaluated on two public data sets, which confirms the effectiveness of the SMCD model in the present invention. The evaluation process and results are as follows:

[0102] The SEED and SEED-IV datasets are widely used datasets in emotion recognition based on EEG signals. The SEED dataset contains 15 subjects, each of whom needs to watch 15 movie clips containing emotional values ​​to elicit three emotions of the subject: positive, neutral, and negative; the SEED-IV dataset contains 15 subjects, each of whom needs to watch 24 movie clips containing emotional values ​​to elicit four emotions of the subject: happy, sad, afraid, and neutral. The EEG signals of both datasets were collected at a frequency of 1000Hz using the ESI NeuroScan System device and recorded using 62 channels according to the 10-20 system. This evaluation uses leave-one-out cross-validation to evaluate the SMCD model. Unlike the previous application of leave-one-out validation in cross-subject experiments, this evaluation uses partially labeled test set data. For the SEED dataset, the data of 14 subjects were used as the training set, and the emotions induced by 3 movie clips (one positive, one neutral, and one negative) of the remaining subject to be predicted were used as the correction data set for training, and the emotional data induced by the remaining 12 movie clips were used as the test set to be predicted. This process was repeated so that each subject's EEG signal was used as the test data once, and the average accuracy was used as the evaluation criterion. For the SEED-IV dataset, a similar leave-one-out cross-validation was used to evaluate our model as the SEED dataset. The only difference is that the SEED-IV dataset used emotions induced by 4 movie clips of a subject (one happy, one sad, one scared, and one neutral) as the correction data set for training, and the emotional data induced by the remaining 20 movie clips as the test set to be predicted, and the data of the remaining 14 subjects as the training set; the evaluation results are shown in the figure. Figure 3 As shown, by comparing the present invention with other models, the emotion recognition accuracy of the present invention is the highest. Other models include: support vector machine (SVM), transfer component analysis (TCA), subspace alignment (SA), geodesic flow kernel method (GFK), domain adaptive neural network (DANN), and transfer attention neural network (TANN).

[0103] The present invention uses a convolutional neural network to deeply mine the relationship between brain regions and frequency bands, and combines frequency band information with channel space information to perform end-to-end automatic deep feature extraction, which is more conducive to the recognition of EEG emotions; a new domain adaptive adversarial training method is proposed, and soft labels are used for cross-disciplinary EEG emotion classification; this method not only considers the decision boundary of a specific category, but also maximizes the difference between two classifiers to achieve the effect of being able to detect target domain samples that are not supported in the source domain, and uses soft labels to further extract source domain information that is helpful for classifying the target domain. The emotion recognition result is more accurate and has good economic benefits.

[0104] The above embodiments further illustrate the purpose, technical solutions and advantages of the present invention in detail. It should be understood that the above embodiments are only preferred implementation modes of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made to the present invention within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. An emotion recognition method based on self-training to maximize classifier differences, comprising: Acquire the EEG data of the subject in real time, preprocess the EEG data, input the preprocessed data into the emotion recognition model based on self-training to maximize the classifier difference, and obtain the emotion recognition result of the subject; The process of training the emotion recognition model based on self-training maximizing classifier differences includes: S1: Divide the preprocessed EEG data into 1s and 4s segments respectively; wherein the EEG data includes known subject data and new subject data; S2: Filter all the segments in 5 different frequency bands to obtain filtered data; S3: Use differential entropy to extract features from the filtered data to obtain differential entropy features; S4: Project the differential entropy feature into a 2D plane feature to obtain five 2D plane features; stack the five 2D planes together to form a 3D cube feature; S5: The cube features X of the known subjects S and its emotion label Y S The source domain (X S ,Y S ), the cube feature X of the new subject T and its emotion label Y T The target domain (X T ,Y T ); S6: The first training of the emotion recognition model based on self-training to maximize the classifier difference according to the source domain; S7: The emotion recognition model trained for the first time based on self-training to maximize the classifier difference is trained for the second time according to the target domain; S8: The emotion recognition model trained for the second time based on the self-training maximization classifier difference is trained for the third time according to the source domain and the target domain; Among them, the emotion recognition model based on self-training to maximize the classifier difference includes a feature extractor F, a classifier C1 and a classifier C2.

2. The emotion recognition method based on self-training to maximize classifier differences according to claim 1, characterized in that: The first training of the emotion recognition model based on self-training maximizing classifier differences according to the source domain includes: inputting the source domain into the emotion recognition model based on self-training maximizing classifier differences to obtain the predicted emotion labels of the known subjects; continuously adjusting the parameters of the model according to the predicted emotion labels of the known subjects and the emotion labels of the known subjects; when the first training loss function is minimized, the first training is completed.

3. The emotion recognition method based on self-training to maximize classifier differences according to claim 2, characterized in that: The loss function formula for the first training is: Among them, L(X S ,Y S ) represents the first training loss function, n s represents the number of known subjects, l CE represents the cross entropy loss function, F(x i ) represents the features generated by the feature generator, C1(F(x i )) represents the probability output of classifier C1, C2(F(x i )) represents the probability output of classifier C2.

4. The emotion recognition method based on self-training to maximize classifier differences according to claim 1, characterized in that: The second training of the emotion recognition model based on self-training to maximize the classifier difference includes: constructing a soft label set according to the target domain, and obtaining soft labels according to the soft label set; replacing the emotion labels in the target domain with soft labels, and inputting the target domain after replacing the labels into the emotion recognition model based on self-training to maximize the classifier difference to obtain new soft labels; adjusting the model parameters according to the soft labels and the new soft labels, and the second training is completed when the loss function is minimized.

5. The emotion recognition method based on self-training to maximize classifier differences according to claim 4, characterized in that: The new soft label is represented as: in, represents the soft label probability distribution output by classifier C1, represents the soft label probability distribution of the output of classifier C2, C1(F(x i )) represents the probability output of classifier C1, C2(F(x i )) represents the probability output of classifier C2.

6. The emotion recognition method based on self-training to maximize classifier differences according to claim 1, characterized in that: The third training of the emotion recognition model based on self-training maximizing classifier differences according to the source domain and the target domain includes: inputting the source domain and the target domain into the emotion recognition model based on self-training maximizing classifier differences, fixing the feature extractor, training only the two classifiers, minimizing the KL divergence between the soft label and the new soft label, and completing the training of the fixed feature extractor when the loss function of the fixed feature extractor is minimized; fixing the two classifiers, training only the feature extractor, minimizing the KL divergence between the soft label and the new soft label, and completing the training of the fixed two classifiers when the loss function of the fixed two classifiers is minimized; and correcting the emotion recognition model based on self-training maximizing classifier differences according to the target domain, and completing the model training when the corrected loss function is minimized.

7. The emotion recognition method based on self-training to maximize classifier differences according to claim 6, characterized in that: The loss function formula for a fixed feature extractor is: Among them, L dis (X T ) represents the classification difference between two classifiers for the same feature, L soft (X T ) represents the probability distribution difference between the new soft label and the soft label, n l represents the size of the corrected dataset in the target domain, n u represents the size of the validation dataset in the target domain, K represents the number of categories, and F(x i ) represents the features generated by the feature generator, C1(F(x i )) represents the probability output of classifier C1, C2(F(x i )) represents the probability output of classifier C2, C1(F(x i )) k represents the probability output of the kth class of classifier C1, C2(F(x i )) k represents the probability output of the kth class of classifier C2; l KL represents the KL divergence, represents the soft label probability distribution output by classifier C1, Represents the soft label probability distribution output by classifier C2.

8. The emotion recognition method based on self-training to maximize classifier differences according to claim 6, characterized in that: The loss function formula for fixing two classifiers is: Among them, L dis (X T ) represents the classification difference between two classifiers for the same feature, L soft (X T ) represents the probability distribution difference between the new soft label and the soft label, n l represents the size of the corrected dataset in the target domain, n u represents the size of the validation dataset in the target domain, K represents the number of categories, and F(x i ) represents the features generated by the feature generator, C1(F(x i )) represents the probability output of classifier C1, C2(F(x i )) represents the probability output of classifier C2, C1(F(x i )) k represents the probability output of the kth class of classifier C1, C2(F(x i )) k represents the probability output of the kth class of classifier C2; l KL represents the KL divergence, represents the soft label probability distribution output by classifier C1, Represents the soft label probability distribution output by classifier C2.

Images