Complex emotion representation and perception method based on emotion vector

Abstract

The invention provides a complex emotion representing and sensing method based on an emotion vector, and the method employs the emotion vector to represent a complex emotion containing two or more emotions, the emotion vector is composed of a plurality of element emotion elements corresponding to human emotions, the value of each element represents the intensity of the element emotion, and the value of each element represents the intensity of the element emotion for the sensing of the emotion vector. According to the method, the approximation degree of the perceived emotion and the actual emotion is improved through a meta-emotion detection algorithm (ISL-MED) based on iterative self-learning. According to the invention, the confidence score of the element emotion examination is utilized to approximately represent the intensity of the element emotion, a plurality of element emotion detectors are designed to detect the complex emotion in the voice, and the detection results are combined into an emotion vector to represent the perceived emotion result. Evaluation of the emotion perception result is based on the distance or loss measurement between the labeled emotion vector and the perception emotion vector, and the actual voice emotion can be described more accurately.

Description

Technical Field

[0001] The present invention relates to the technical field of emotion recognition, and in particular to a complex emotion representation and perception method based on emotion vectors. Background Art

[0002] In real scenarios, the emotions in speech are naturally complex, and the perception of them is also somewhat ambiguous. However, the existing speech emotion recognition (SER) method only gives a single emotion classification result for a piece of speech, and cannot accurately represent the complex emotions of speech. To this end, the present invention proposes a complex emotion representation and perception method based on emotion vectors, and based on this, conducts research and experiments from three aspects: emotion representation, data processing, and emotion perception.

[0003] Among the existing emotion representation methods, emotion distribution labels are most similar to emotion vectors, both of which represent emotion categories and components in the form of vectors. However, the meanings of vector components are different. The vector components of emotion distribution labels represent the relative differences between emotions, while the vector components in emotion vectors represent the intensity or degree of meta-emotions. In emotion distribution labels, ∑y must be equal to 1, which ignores the measurement of the true emotion components. For example, speech A contains both weak anger and weak sadness, and speech B contains strong anger and strong sadness, but their emotion distribution labels are both represented as [0.5, 0.5, 0, 0]. However, emotion vectors can use values ​​such as [0.3, 0.3, 0, 0] and [0.8, 0.8, 0, 0] to represent the intensity of different emotions.

[0004] In artificial intelligence, iterative learning is a learning method that gradually improves the performance of a model or algorithm by repeatedly using feedback during the learning process. Specifically, in the task of emotion recognition, due to the subjective differences in human emotion perception, it is difficult for annotators to accurately label all emotion categories and the number of their components in speech / face. These factors result in rough emotion labels in existing emotion datasets, such as hard labels. Iterative learning provides a simple and effective solution to "learning rough emotion labels".

[0005] The existing iterative self-learning paradigm has certain defects in SER tasks:

[0006] Defect 1: Ignoring the impact of model misjudgment on converged parameters. Current iterative models generally use multi-label classifiers as the backbone network. Although this design can capture the relationship between labels, it also brings a challenge: misjudged emotion categories may serve as supervisory signals for the next iteration, causing potential interference to the parameters of the already converged categories.

[0007] Defect 2: Ignoring the impact of unreliable data on iterative training. In each iteration, samples that are misjudged by the model or incorrectly labeled by humans will be repeatedly used for training, hindering model convergence.

[0008] Defect 3: The iterative stop mechanism has defects. Due to the different recognition difficulties of different emotion categories, the iterative model using a multi-label classifier as the backbone network cannot ensure that the model parameters of all emotion categories reach the optimal convergence state at the same time.

[0009] The existing patent application (CN104036776A) discloses a method for extracting speech emotion information, which is characterized in that speech data is collected or transmitted by a mobile phone, computer, voice recorder or other device through a data collection or communication process, and the speaker's emotions are identified by two methods: speaker-independent and speaker-dependent. The speech database part is used as a benchmark for training an emotion recognizer, and includes at least one emotional speech database; the speech emotion modeling part is used to establish a speech emotion model as an emotion recognizer, and adjust the internal parameters by statistical methods to identify the emotions in the speech signal. This method can identify other complex emotions with a set of special parameters that describe basic emotions. Summary of the invention

[0010] In order to solve the above problems, the present invention provides a complex emotion representation and perception method based on emotion vectors, and conducts research and experimental demonstration from three aspects: emotion representation, data processing, and emotion perception. For emotion representation, in view of the unreliability of existing emotion representation methods, an accurate representation method for complex emotions, emotion vectors, is proposed. Emotion vectors represent any single or complex emotions through multiple meta-emotion elements (basic emotions). For emotion perception, this paper proposes a meta-emotion detection algorithm based on iterative self-learning to improve the degree of approximation between emotion perception and actual emotions. For data processing, in order to guide the model to simulate human emotion perception ability, a data selection method based on human-computer consensus and curriculum learning (HMCC) is proposed.

[0011] The present invention provides a complex emotion representation and perception method based on emotion vector, comprising the following steps:

[0012] S1. Record a complex emotion speech database and annotate the speech with an emotion vector containing all emotion categories and their intensity information, which is used as a benchmark for complex emotion perception.

[0013] S2. Designing a meta-emotion detection model based on an iterative self-learning algorithm for each meta-emotion through the complex emotion speech database;

[0014] S3, obtaining a speech sample to be tested;

[0015] S4, sequentially importing the speech samples to be tested into the meta-emotion detection model based on iterative self-learning algorithm;

[0016] S5. Determine the intensity of each meta-emotion and combine them into an emotion vector representation.

[0017] The present invention discloses a complex emotion representation and perception method based on emotion vectors. The method proposes a meta-emotion detection algorithm based on iterative self-learning (ISL-MED). ISL-MED includes a training phase and a verification / test phase. In the training phase, ISL-MED consists of four independent meta-emotion detectors (EDs): anger, sadness, excitement and happiness. All meta-emotion detectors use the same training process and backbone network, and the only difference is their training data. Each meta-emotion detector consists of an initial training module and multiple iterative self-learning modules. In the initial training module, the training set is divided into two parts: a positive class set containing specific emotion samples and a negative class set not containing these emotion samples. The initial training module trains the meta-emotion detector using manual hard labels. Then, the previous meta-emotion detector is updated using multiple iterative self-learning modules. The iterative self-learning module includes data selection, label update and model training. In the verification and testing phases, the detection results are combined into an emotion vector to represent the results of complex emotion perception. On the test set, the emotion perception results are evaluated based on the distance or loss metric between the labeled emotion vector and the perceived emotion vector, such as cosine distance and KL divergence. On the validation set, the iteration stops when the overall change rate of the evaluation index between two consecutive perceived emotion vectors is lower than the stopping threshold α; otherwise, the training process continues.

[0018] Take the anger detector as an example to illustrate the iterative self-learning process. In the initial training module, the training set S 1 (The superscript indicates the training data used in the first iteration) The samples containing "angry" are classified as positive, and the samples not containing "angry" are classified as negative. Manually annotated hard labels are used as supervision information for training Since this is a regression task, we use binary cross entropy as the loss function:

[0019]

[0020] Among them, y i is the true label of the i-th sample, y i ' is the confidence of the model predicting that the i-th sample is a positive class. n is the number of training samples in the current round.

[0021] Starting from the Tth round (T≥2), the iterative training module is used For the original training set S 1 Make a prediction to get the intermediate state S of the training data T′ :

[0022]

[0023] S T′ ={X 1 ,Y T-1},Y T-1 ∈[0,1]

[0024] Execute the data selection operation, S T′ By data selector F T Processing, it selects appropriate training data:

[0025] S T″ =F T (S T′ )

[0026] S T″ ={X T ,Y T-1},X T ∈X 1

[0027] Execute the label update operation and directly use the prediction result generated by the previous iteration, or add the artificial hard label to the prediction result generated by the previous iteration and weight it as the prediction generated by the previous iteration:

[0028] S T ={X T ,Y T}

[0029] Execute the model training operation, using the S generated above T As the training data for the Tth round of iteration,

[0030] The present invention discloses a complex emotion representation and perception method based on emotion vectors. As a preferred method, data selection includes a data selection method based on human-computer consensus and a data selection method based on curriculum learning. The data selection method based on human-computer consensus selects samples with consistent human-computer emotion judgments for iterative self-learning. The data selection method based on curriculum learning first learns simple samples with significant emotions, and then gradually learns difficult samples with ambiguous emotions. Simple samples are samples that can be predicted correctly by the model with high confidence, and difficult samples are samples that can be predicted correctly but with low confidence. The sample difficulty is controlled by the sample selection threshold β.

[0031] Data selection method based on human-machine consensus: In the real world, it is inevitable that the emotional judgments of humans and machines are inconsistent. The emotion recognition model trained on such inconsistent data may not be able to effectively simulate human emotional perception. Therefore, this work selects samples with consistent human-machine judgments for iterative self-learning.

[0032] Except for the initial training module, the entire training dataset S is used. 1 In addition, the training data S of the subsequent iterative module T All are selected by the data selector F based on human-machine consensus T From S 1 The specific steps are as follows:

[0033] S1, the training set S 1 pass Get the corresponding prediction confidence Y T-1 ∈[0,1];

[0034] S2, the confidence Y T-1 Input data filter F T-1 , we will use Y T-1 and Y 1 The consistency between them determines which samples will be used for the Tth iteration. T-1 Samples that do not meet the above conditions will be excluded from the next round of iterative training. Specifically, we set a sample selection threshold β. 1 For the positive samples in the T-1 >β samples are used as the positive set for the T-th round of iterative training For S 1 For the negative samples in the T-1 Samples <1-β are used as negative sets for the Tth iteration. Iterate the training data S T YesS 1 After F T The data obtained after the T-1 After F T Then get it.

[0035] Data selection method based on curriculum learning: In subjective detection tasks such as emotion recognition, for significant emotion samples that are easy for humans to judge, the present invention requires the model to output accuracy predictions with lower entropy values. For ambiguous emotion samples that are difficult for humans to judge, the model should be allowed to output uncertainty predictions with higher entropy values. Inspired by the curriculum learning paradigm, the present invention introduces the data selection method based on human-machine consensus into the training strategy based on curriculum learning.

[0036] The course learning guidance model first learns simple samples with significant emotions, and then gradually learns difficult samples with ambiguous emotions. The present invention defines samples that can be predicted correctly by the model with high confidence as simple samples; samples that can be predicted correctly but with low confidence are defined as difficult samples. Specifically, we use the sample selection threshold β to control the sample difficulty.

[0037] For S 1 The positive samples in the , when β is larger, it means The lower the entropy value of , the easier it is for the sample to be correctly predicted. When β is low, it means that the sample is more difficult to be accurately predicted. 1 The negative samples in , when 1-β is low, indicate The prediction entropy value of is low, and the sample is easier to be correctly predicted. When 1-β is high, sample i is more difficult to be accurately predicted. In the training stage, the sample selection threshold β is set from large to small, indicating that the iterative training data S T Gradually include difficult samples, and the learning difficulty increases from easy to difficult. At the same time, gradually reduce the learning rate to prevent fuzzy samples from destroying the emotional decision boundary.

[0038] In summary, the data selection method based on human-computer consensus and curriculum learning takes into account the consistency of human-computer emotion judgment on the basis of the existing curriculum learning strategy. This method effectively simulates the process of humans from perceiving salient emotions to recognizing ambiguous emotions.

[0039] The complex emotion representation and perception method based on emotion vector described in the present invention is, as a preferred embodiment, that label updating includes basic dynamic soft labels and weighted dynamic soft labels.

[0040] Basic dynamic soft label: This paper explores the impact of relying only on the model's prediction results as supervisory labels on iterative self-learning. Therefore, directly using model D T-1 The prediction result Y T- 1 as the supervision label for the next round of iterative training:

[0041] Y T ←Y T-1 =D T-1 (X 1 )

[0042] Weighted dynamic soft labels: This paper will explore the impact of supervised labels that combine model prediction with human emotion perception on iterative self-learning. 1 and the model predicts Y T-1 Update the label Y by using a weighted combination of T :

[0043] Y T =(1-γ)Y 1 +γY T-1

[0044] Wherein γ∈[0,1] is the label balance coefficient, and in the present invention, γ=0.5.

[0045] Compared with the existing emotion recognition model, the present invention independently models each emotion, avoiding the interference of the model parameters of the converged emotion category by the results of model prediction errors. The present invention can increase or delete the number of meta-emotion detectors according to needs, and is applicable to a variety of emotion detection scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 This is a flow chart of ISL-MED model training of a complex emotion representation and perception method based on emotion vectors of the present invention;

[0047] Figure 2 A flow chart of complex emotion detection in speech based on a complex emotion representation and perception method of an emotion vector according to the present invention;

[0048] Figure 3 A schematic diagram of a data selection method based on human-computer consensus and curriculum learning for a complex emotion representation and perception method based on emotion vectors of the present invention;

[0049] Figure 4 The comparative experimental results of ISL-MED and TISL-ED when the backbone network of the complex emotion representation and perception method based on emotion vector of the present invention is Wav2vec2.0;

[0050] Figure 5 The comparative experimental results of ISL-MED and TISL-ED when the backbone network of the complex emotion representation and perception method based on emotion vector of the present invention is HuBERT;

[0051] Figure 6 This is an experimental confusion matrix of a complex emotion representation and perception method based on emotion vectors of the present invention. DETAILED DESCRIPTION

[0052] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0053] Example 1

[0054] like Figures 1-2 As shown, a speech emotion recognition method comprises the following steps:

[0055] S1. Record a complex emotion speech database and annotate the speech with an emotion vector containing all emotion categories and their intensity information, which is used as a benchmark for complex emotion perception.

[0056] S2. Designing a meta-emotion detection model based on an iterative self-learning algorithm for each meta-emotion through the complex emotion speech database;

[0057] S3, obtaining a speech sample to be tested;

[0058] S4, sequentially importing the speech samples to be tested into the meta-emotion detection model based on iterative self-learning algorithm;

[0059] S5. Determine the intensity of each meta-emotion and combine them into an emotion vector representation.

[0060] The meta-emotion detection model (ISL-MED) based on iterative self-learning used in this embodiment includes training and verification / testing stages. In the training stage, ISL-MED contains four independent meta-emotion detectors (ED): anger, sadness, excitement and happiness. All meta-emotion detectors have the same training process and backbone network, the only difference is their training data. Each meta-emotion detector includes an initial training module and multiple iterative self-learning modules. In the initial training module, the training set is divided into two parts: a positive set containing specific emotion samples and a negative set without the emotion. The first round of models trains the meta-emotion detector using the hard labels provided by the data set. Subsequently, multiple iterative self-learning modules are executed to update the previous detector. The iterative self-learning module includes three steps: data selection, label update and model training. In the verification / testing stage, each sample passes through the four meta-emotion detectors obtained from each iteration in turn. Finally, the results from all detectors are combined into an emotion vector. We evaluate the model performance by comparing the distance between the emotion vector and the labeled emotion vector. When the overall metric change rate between the sentiment confidence distributions predicted by adjacent iterative models is lower than the stopping threshold α, the iterative process terminates; otherwise, the iteration continues.

[0061] Take the anger detector as an example to illustrate the iterative self-learning process. In the initial training module, the training set S 1 (The superscript indicates the training data used in the first iteration) The samples containing "angry" are classified as positive, and the samples not containing "angry" are classified as negative. Manually annotated hard labels are used as supervision information for training Since this is a regression task, we use binary cross entropy as the loss function:

[0062]

[0063] Among them, y i is the true label of the i-th sample, y i ' is the confidence of the model predicting that the i-th sample is a positive class. n is the number of training samples in the current round.

[0064] Starting from the Tth round (T≥2), the iterative training module is used For the original training set S 1 Make a prediction to get the intermediate state S of the training data T′ :

[0065]

[0066] S T′ ={X 1 ,Y T-1},Y T-1 ∈[0,1]

[0067] S4. Execute the data selection operation, S T′ By data selector F T Processing, it selects appropriate training data:

[0068] S T″ =F T (ST ′ )

[0069] S T″ ={X T ,Y T-1},X T ∈X 1

[0070] S5. Execute the label update operation, directly use the prediction result generated by the previous round of iteration, or add the artificial hard label to the prediction result generated by the previous round of iteration and weight it as the prediction generated by the previous round of iteration:

[0071] S T ={X T ,Y T}

[0072] S6, execute the model training operation, using the S generated above T As the training data for the Tth round of iteration, like Figure 3 , data selection includes data selection methods based on human-machine consensus and data selection methods based on curriculum learning; the data selection method based on human-machine consensus selects samples with consistent human-machine judgments for iterative self-learning; the data selection method based on curriculum learning first learns simple samples with significant emotions, and then gradually learns difficult samples with ambiguous emotions. Simple samples are samples that can be predicted correctly by the model with high confidence, and difficult samples are samples that can be predicted correctly but with low confidence. The sample difficulty is controlled by the sample selection threshold β.

[0073] Data selection method based on human-machine consensus: In the real world, it is inevitable that humans and machines have inconsistent emotional judgments. Emotion recognition models trained on such inconsistent data may not be able to effectively simulate human emotional perception. Therefore, this work selects samples with consistent human-machine judgments for iterative self-learning.

[0074] Except for the initial training module, the entire training dataset S is used. 1 In addition, the training data S of the subsequent iterative module T All are selected by the data selector F based on human-machine consensus T From S 1 The specific steps are as follows:

[0075] S1, the training set S 1 pass Get the corresponding prediction confidence Y T-1 ∈[0,1];

[0076] S2, the confidence Y T-1 Input data filter F T-1 , we will use Y T-1 and Y 1 The consistency between them determines which samples will be used for the Tth iteration. T-1 Samples that do not meet the above conditions will be excluded from the next round of iterative training. Specifically, we set a sample selection threshold β. 1 For the positive samples in the T-1 >β samples are used as the positive set for the T-th round of iterative training For S 1 For the negative samples in the T-1 Samples <1-β are used as negative sets for the Tth iteration. Iterate the training data S T YesS 1 After F T The data obtained after the T-1 After F T Then get it.

[0077] Data selection method based on curriculum learning: In subjective detection tasks such as emotion recognition, for significant emotion samples that are easy for humans to judge, the present invention requires the model to output accurate predictions with lower entropy values. For ambiguous emotion samples that are difficult for humans to judge, the model should be allowed to output uncertainty predictions with higher entropy values. Inspired by the curriculum learning paradigm, the present invention will introduce a data selection strategy based on curriculum learning on the basis of the data selection method of human-computer consensus.

[0078] The course learning guidance model first learns simple samples with significant emotions, and then gradually learns difficult samples with ambiguous emotions. The present invention defines samples that can be predicted correctly by the model with high confidence as simple samples; samples that can be predicted correctly but with low confidence are defined as difficult samples. Specifically, we use the sample selection threshold β to control the sample difficulty.

[0079] For S 1 The positive samples in the , when β is larger, it means The higher the entropy value, the easier it is for the sample to be correctly predicted. When β is low, it means More difficult to predict accurately. 1 The negative samples in , when 1-β is low, indicate The prediction entropy value of is lower, and the sample is more likely to be correctly predicted. When 1-β is higher, it means It is more difficult to be accurately predicted. In the training stage, the sample selection threshold β is set from large to small, indicating that the iterative training data S T Gradually include difficult samples, and the learning difficulty increases from easy to difficult. At the same time, gradually reduce the learning rate to prevent fuzzy samples from destroying the emotional decision boundary.

[0080] In summary, the data selection method based on human-computer consensus and curriculum learning takes into account the consistency of human-computer emotion judgment on the basis of the existing curriculum learning strategy. This method effectively simulates the process of humans from perceiving salient emotions to recognizing ambiguous emotions.

[0081] Label updates include basic dynamic soft labels and weighted dynamic soft labels.

[0082] Basic dynamic soft label: This paper explores the impact of relying only on the model's prediction results as supervisory labels on iterative self-learning. Therefore, directly using model D T-1 The prediction result Y T-1 As the supervision label for the next round of iterative training:

[0083] Y T ←Y T-1 =D T-1 (X 1 )

[0084] Weighted dynamic soft labels: This paper will explore the impact of supervised labels that combine model prediction with human emotion perception on iterative self-learning. 1 and the model predicts Y T-1 Update the label Y by using a weighted combination of T :

[0085] Y T =(1-γ)Y 1 +γY T-1

[0086] Where γ∈[0,1] is the label balance coefficient and is set to 0.5.

[0087] like Figures 4-5 This embodiment verifies the effectiveness of the present invention for complex emotion perception on CNSCED (Chinese Natural Speech Complex Emotion Dataset). The scalability and generalization of the present invention in traditional single emotion classification tasks are verified on IEMOCAP (Interactive Emotion Binary Motion Capture Dataset).

[0088] Based on the existing work, this embodiment designs a traditional iterative self-learning emotion detection model (TISL-ED) as a baseline model. Unlike ISL-MED, in TISL-ED, all emotion categories share a backbone network, and K binary prediction units are added at the end of the model. The present invention uses the Sigmoid activation function and the mean square error (MSE) as the target loss function. In addition, TISL-ED does not apply data selection methods in the iteration process; instead, the entire training set is used to train the model.

[0089] This embodiment uses the average distance and loss between the sample predicted emotion vector and the emotion vector annotated in CNSCED as the metric, for example, cosine distance, Chebyshev distance, Jaccard distance, Clark distance, Hamming loss, KL divergence. The results are shown in Table 1:

[0090]

[0091]

[0092] Table 1 Comparative experimental results of ISL-MED and TISL-ED on CNSCED

[0093] Regardless of the backbone network architecture, ISL-MED achieved significant improvements in most indicators. When HuBERT was used as the backbone network, ISL-MED achieved the best performance. Except for the cosine distance, the remaining indicators increased by 10.47%, 16.58%, 0.67%, 1.85% and 22.22% respectively, with an overall improvement of 8.16%. This experiment demonstrates the combined advantages of the independent detector iterative self-learning framework combined with the HMCC data selection method.

[0094] Subsequently, the model generated by each round of iterative training is tested on the CNSCED test set, and the test results are visualized.

[0095] In general, the ISL-MED method performs better than TISL-ED in all six evaluation metrics. Obviously, except for the cosine distance metric and the Clark distance metric, ISL-MED usually reaches the best performance after the 3rd or 4th round of iterative training. This shows that ISL-MED can effectively reduce the interference of ambiguous emotion samples on the convergence of the model, thereby accelerating the convergence time. However, the TISL-ED metric fluctuates significantly, and no obvious regularity is found.

[0096] This embodiment also verifies generalizability and extensibility.

[0097] In order to verify the generalization and extensibility of ISL-MED on a single emotion classification task, the present invention conducts experiments on the IEMOCAP dataset. The present invention performs a post-processing step on the obtained complex emotion confidence distribution, that is, selecting the maximum confidence as the final prediction category.

[0098] For fair comparison, we additionally trained a neutral emotion detector. In this experiment, wav2vec2.0 was used as the backbone network, and weighted dynamic soft labeling was applied for label update. Finally, we selected the best detector combination from all iterative trainings to predict the test set. The evaluation indicators used weighted accuracy WA and unweighted accuracy UA.

[0099]

[0100] Table 2 Single emotion classification results of ISL-MED on IEMOCAP

[0101] As shown in Table 2, ISL-MED achieves satisfactory recognition accuracy compared with the state-of-the-art methods.

[0102] like Figure 6 As shown in the confusion matrix, the WA of the angry, happy, and sad categories are 77.15%, 68.27%, and 76.38%, respectively, while the WA of the neutral emotion detector is 63.70%. This is because the neutral category utterances often contain the emotional atmosphere of the current conversation.

[0103] The above description is only illustrative rather than restrictive of the present invention. Those skilled in the art will understand that any modification, change or equivalent that can be made without departing from the spirit and scope defined by the claims will fall within the protection scope of the present invention.

Claims

1. A complex emotion representation and perception method based on emotion vector, characterized by: The following steps are involved: S1, recording a complex emotion speech database, and annotating the speech with an emotion vector containing all emotion categories and intensity information, and using the emotion vector as a benchmark for complex emotion perception; S2. Designing a meta-emotion detection model based on an iterative self-learning algorithm for each meta-emotion through the complex emotion speech database; S3, obtaining a speech sample to be tested; S4, importing the speech samples to be tested into the meta-emotion detection model in sequence; S5. Determine the intensity of each meta-emotion and combine them into an emotion vector representation.

2. The complex emotion representation and perception method based on emotion vector according to claim 1, characterized in that: Emotion vector Y = [Y 1 ,Y 2 ,…,Y K ] is composed of K-dimensional meta-emotional elements; each meta-emotional element corresponds to a basic human emotion, such as anger, sadness, and happiness; the emotion vector can represent a complex emotion containing two or more meta-emotions; where Y j = [0,1] indicates the intensity or degree of meta-emotion, 3. The complex emotion representation and perception method based on emotion vector according to claim 1, characterized in that: The emotion vector represents any single or complex emotion by controlling the intensity values ​​of different meta-emotion elements; the meta-emotion detection model includes four independent meta-emotion detectors of anger, sadness, excitement and happiness; in the training stage, the meta-emotion detectors have the same training process and network architecture, and the meta-emotion detectors include an initial training module and multiple iterative self-learning modules; During the testing phase, the speech sample to be tested passes through the four meta-emotion detectors in turn, and the results from all the meta-emotion detectors are combined into a perceived emotion vector. The performance of the iterative self-learning multi-emotion detection model is evaluated by comparing the distance between the perceived vector and the labeled emotion vector. When the overall metric change rate between the emotion vectors perceived by adjacent iterative models is lower than the stopping threshold α, the iterative process is terminated; otherwise, the iteration continues.

4. The complex emotion representation and perception method based on emotion vector according to claim 3, characterized in that: Each of the meta-emotion detectors includes an initial training module and multiple iterative self-learning modules; in the initial training module, samples containing a certain emotion are defined as a positive training set, and samples not containing the emotion are defined as a negative training set, and manually annotated hard labels are used as supervisory information; the iterative self-learning module uses the output of the initial training module as supervisory information.

5. A complex emotion representation and perception method based on emotion vectors according to patent claim 4, characterized in that: The operation steps of the iterative self-learning module include three steps: data selection, label update and model training; in data selection, a data selection method based on human-machine consensus and curriculum learning is used, which selects samples with consistent emotional judgments between humans and machines to participate in the operation of the next round of iterative self-learning module; at the same time, the samples participate in the iterative self-learning training process in the order of sample detection difficulty from easy to difficult.

6. The complex emotion representation and perception method based on emotion vector according to claim 5, characterized in that: The training process of the anger detector includes the following steps: S1. In the initial training module, the training set S 1 , samples containing "angry" are classified as positive, and samples not containing "angry" are classified as negative, where the superscript represents the training round, X represents the speech data, and Y represents the label; S2. Use manually annotated hard labels as supervision information for training The objective loss function is: Among them, y i is the true label of the i-th sample, y' i is the confidence of the model predicting that the i-th sample is a positive class, and n is the number of training samples in the current round; S3, starting from the Tth round (T≥2) of iterative training modules, using For the original training set S 1 Make a prediction to get the intermediate state S of the training data T′ : S T′ ={X 1 ,AND T-1 },AND T-1 ∈[0,1] S4. Execute the data selection operation, S T′ By data selector F T Processing, it selects appropriate training data: S T″ =F T (S T′ ) S T″′ ={X T ,Y T-1 },X T ∈X 1 S5. Execute the label update operation, directly use the prediction result generated by the previous round of iteration, or add the artificial hard label to the prediction result generated by the previous round of iteration and weight it as the prediction generated by the previous round of iteration: S T ={X T ,Y T } S6, execute the model training operation, using the S generated above T As the training data for the Tth round of iteration, 7. The complex emotion representation and perception method based on emotion vector according to claim 5, characterized in that: The data selection includes a data selection method based on human-computer consensus and a data selection method based on curriculum learning; the data selection method based on human-computer consensus selects samples with consistent human-computer emotion judgments for iterative self-learning; the data selection method based on curriculum learning first learns simple samples with significant emotions, and then gradually learns difficult samples with ambiguous emotions. The simple samples are samples that can be predicted correctly by the model with high confidence, and the difficult samples are samples that can be predicted correctly but with low confidence. The sample difficulty is controlled by the sample selection threshold β.

8. The complex emotion representation and perception method based on emotion vector according to claim 5, characterized in that: The specific steps of the data selection method based on human-machine consensus are as follows: S1, the training set S 1 pass Get the corresponding prediction confidence Y T-1 ∈[0,1]; S2, the confidence level Y T-1 Input data filter F T-1 , according to Y T-1 and Y 1 The consistency between them determines which samples will be used for the Tth iteration; S3. For S 1 For the positive samples in the T-1 >β samples are used as the positive set for the Tth round of iterative training For S 1 For negative samples in the T-1 Samples <1-β are used as negative sets for the Tth iteration. Samples that do not meet the above conditions will not participate in the next round of iterative training.

9. The complex emotion representation and perception method based on emotion vector according to claim 5, characterized in that: The data selection method based on curriculum learning controls the difficulty of the sample selected by the sample by controlling the sample selection threshold β; for S 1 The positive samples in the , when β is larger, it means The lower the entropy value, the easier it is for the sample to be correctly predicted; when β is lower, it means More difficult to predict accurately. 1 The negative samples in , when 1-β is lower, it means The prediction entropy value of is lower, and the sample is more likely to be correctly predicted. When 1-β is higher, it means More difficult to predict accurately; In the training phase, the sample selection threshold β is set from large to small in [1.0, 0.5], with an interval of 0.1, indicating that the iterative training data S T Gradually include difficult samples, and the learning difficulty increases from easy to difficult; at the same time, gradually reduce the learning rate to prevent fuzzy samples from destroying the emotional decision boundary.

Images