Fair voice emotion recognition method and system

Abstract

The invention provides a fair voice emotion recognition method, and the method generates an emotion prediction result corresponding to voice data based on the voice data and an unbiased voice emotion recognition model. The unbiased speech emotion recognition model is trained by training a fair constraint adversarial network according to a labeled training set of known biases and a loss function. The fair constraint adversarial network comprises a domain classifier used for carrying out prejudice classification and the voice emotion recognition model, and a loss function used for training the fair constraint adversarial network is positively correlated to Warisstein distance loss.

Description

Technical Field

[0001] The present disclosure relates to speech emotion recognition, and more particularly to a speech emotion recognition method and system for dealing with the problem of evaluator bias. Background Art

[0002] Speech Emotion Recognition (SER) plays a key role in many speech-based solutions. However, there are many unique fairness issues in the field of speech emotion recognition, which are becoming increasingly prominent in the development and application of related technologies. For example, when evaluators provide emotional perceptions based on their own inherent biases, they may produce different evaluation results, leading to the problem of evaluator bias.

[0003] Past methods have attempted to address these bias issues by improving the composition of the dataset itself, or adjusting the weight ratio of the data, etc. However, these methods are mostly limited to specific fields in dealing with bias issues, cannot be applied in a one-size-fits-all manner, and require more training data.

[0004] In addition, previous technologies have not provided users with enough transparency in achieving fairness, allowing them to clearly understand the different results given by different evaluators. This has, to some extent, led to users’ concerns about the “black box” problem that machine learning systems may have, and has limited the fairness of the system in practical applications. Summary of the invention

[0005] In view of this, the present disclosure proposes a speech emotion recognition method and system, which can effectively learn and deal with bias to improve the fairness of speech emotion recognition.

[0006] The first aspect of the present disclosure proposes a computer-implemented method for training a speech emotion recognition model, comprising: providing a fairness constraint adversarial network; and training the fairness constraint adversarial network based on a labeled training set with known bias and a first loss function to obtain an unbiased speech emotion recognition model. The fairness constraint adversarial network includes a domain classifier and the speech emotion recognition model, wherein the domain classifier is used to perform bias classification on input data, and the speech emotion recognition model is used to determine the emotion corresponding to the input data. The first loss function used to train the fairness constraint adversarial network is positively correlated to the Wasserstein distance loss.

[0007] The second aspect of the present disclosure proposes a computer-implemented method for speech emotion recognition, comprising: receiving speech data; and generating an emotion prediction result corresponding to the speech data based on the speech data and an unbiased speech emotion recognition model. The speech emotion recognition model is trained by training a fair constraint adversarial network based on a labeled training set with known bias and a first loss function, wherein the fair constraint adversarial network includes a domain classifier for bias classification and the speech emotion recognition model. The first loss function used to train the fair constraint adversarial network is positively correlated to the Wasserstein distance loss.

[0008] The third aspect of the present disclosure proposes a computer-readable storage medium storing a program, wherein the program includes instructions, which, when executed by one or more processors of an electronic device, cause the electronic device to execute the computer-implemented method of speech emotion recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The accompanying drawings exemplarily illustrate the embodiments and constitute a part of the specification, and together with the text description of the specification, are used to explain the exemplary implementation of the embodiments. Obviously, the drawings described below are only some embodiments of the present disclosure, and for those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work. In all drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0010] Figure 1 is a schematic diagram showing a training speech emotion recognition model according to an embodiment of the present disclosure;

[0011] Figure 2 is a flowchart showing a method for training an unbiased speech emotion recognition model according to an embodiment of the present disclosure;

[0012] Figure 3 is a flowchart showing a training multi-view speech emotion recognition model according to an embodiment of the present disclosure;

[0013] Figure 4 is a schematic diagram showing speech emotion recognition according to an embodiment of the present disclosure;

[0014] Figure 5 is a flow chart showing speech emotion recognition according to an embodiment of the present disclosure;

[0015] Figure 6 is a block diagram illustrating a computing system according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0016] In order to make the purpose, technical solutions and advantages of the present disclosure clearer, the present disclosure will be further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present disclosure.

[0017] In this disclosure, the term "plurality" means two or more, unless otherwise specified. In this disclosure, the term "and / or" describes the association relationship of associated objects, covering any one of the listed objects and all possible combinations. The character " / " generally indicates that the associated objects before and after are in an "or" relationship.

[0018] In the present disclosure, unless otherwise specified, the terms "first", "second", etc. are used to distinguish similar objects without intending to limit their positional relationship, temporal relationship, or importance relationship. It should be understood that the terms used in this way can be interchangeable where appropriate, so that the embodiments of the present disclosure described herein can be implemented in ways other than those illustrated or described herein.

[0019] In addition, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus that includes a series of steps or elements is not necessarily limited to those steps or elements explicitly listed, but may include other steps or elements not explicitly listed or inherent to such process, method, system, product, or apparatus.

[0020] The embodiments of the present disclosure provide a speech emotion recognition method and system that can effectively learn and process bias.

[0021] In some embodiments, the above-mentioned bias includes assessor bias. Specifically, assessor bias refers to the subjective bias that the assessor may have when evaluating a project or situation. This bias may be caused by the assessor's personal experience, beliefs, values, or identity characteristics such as race, gender, and age. In the field of machine learning, if the training data of the model is labeled by the assessor, then the assessor bias may affect the results of the data labeling and ultimately affect the training results and performance of the model. For example, in the training data related to speech emotion recognition, the assessor needs to label the emotions of the speech sample. However, assessors of different types (for example, gender, race, age group, etc.) may have different understandings and interpretations of the emotions of the same speech sample, which may cause assessor bias, and the trained model may be biased towards the views of certain assessors.

[0022] In some embodiments, the above-mentioned bias includes binary bias, such as evaluator gender bias. In the following text, the evaluator gender bias will be used as an example to illustrate the speech emotion recognition method and system proposed in the embodiment of the present disclosure. However, it should be understood by those skilled in the art that other types of bias can also be applied under the technical concepts introduced in the embodiment of the present disclosure.

[0023] Unbiased Speech Emotion Recognition Model

[0024] Figure 1 is a schematic diagram showing a training speech emotion recognition model according to an embodiment of the present disclosure; Figure 2 is a flowchart showing a method for training an unbiased speech emotion recognition model according to an embodiment of the present disclosure.

[0025] Please refer to Figure 1 and Figure 2 In order to train an unbiased speech emotion recognition model, firstly, in step S210 , a fairness-constrained adversarial network (FCAN) 100 is provided, which includes a domain classifier 110 and a speech emotion recognition model 120 .

[0026] Specifically, the domain classifier 110 serves as a discriminator of the fairness constraint adversarial network 100 to perform bias classification on input data.

[0027] In some embodiments, the input data is, for example, speech data including an emotion tag or a feature including an emotion tag, and the domain classifier 110 is used to classify the input data as "first type bias" or "second type bias." For example, the domain classifier 110 is used to classify speech data or features including emotion tags as the first type bias: "biased towards male perspective" or the second type bias: "biased towards female perspective."

[0028] It must be noted that the present disclosure does not limit the format or content of the emotion tag, and those skilled in the art can design or adjust it according to their needs. In some embodiments, the voice data may correspond to a single binary emotion (e.g., whether happy); in some embodiments, the voice data may correspond to multiple binary emotions (e.g., whether happy, angry, frustrated, etc.); in some embodiments, the voice data may correspond to a single quantified emotion (e.g., a happiness score); in some embodiments, the voice data may correspond to multiple quantified emotions (e.g., a happiness score, an anger score, a frustration score, etc.).

[0029] In some embodiments, the domain classifier 110 is, for example, a deep neural network composed of fully-connected layers, but the present disclosure is not limited thereto. In other embodiments, as long as the input data can be classified as described above, the present disclosure does not limit the specific architecture of the domain classifier 110.

[0030] On the other hand, the speech emotion recognition model 120 serves as a generator of the fairness constraint adversarial network 100 to generate fair embeddings corresponding to input data and to judge or predict emotions corresponding to the input data.

[0031] In some embodiments, the speech emotion recognition model 120 is, for example, a deep neural network composed of fully connected layers, but the present disclosure is not limited thereto. In other embodiments, as long as a fair embedding corresponding to the input data can be generated and the emotion corresponding to the input data can be judged or predicted, the present disclosure does not limit the specific architecture of the speech emotion recognition model 120.

[0032] In some embodiments, in the architecture of the speech emotion recognition model 120, at least one fully connected layer is included before the output layer. For example, the layer before the output layer of the speech emotion recognition model 120 is a fully connected layer. The fair embedding corresponding to the input data can be obtained, for example, by extracting this fully connected layer, and the emotion corresponding to the input data is obtained through the output layer.

[0033] Please go back Figure 1 and Figure 2 In step S220, the fairness constrained adversarial network 100 is trained based on the labeled training set 200 with known bias and the first loss function to obtain an unbiased speech emotion recognition model 120.

[0034] In some embodiments, the labeled training set 200 includes multiple training data, each of which corresponds to an overall emotion label. In some embodiments, in addition to the overall emotion label, each of the training data further corresponds to multiple perspective emotion labels of multiple perspectives.

[0035] Specifically, the overall sentiment label corresponding to each piece of training data may be determined by a joint vote of multiple evaluators, so the voting result may be subject to evaluator bias. More specifically, the evaluators participating in the voting may include different types of evaluators, and each type of evaluator will evaluate from their own perspective, and obtain multiple perspective sentiment labels from multiple perspectives. However, there may be no consensus between each type of evaluator, resulting in the final voting result (e.g., the overall sentiment label) being biased towards a specific type of evaluator or towards a specific perspective, thus generating evaluator bias.

[0036] For example, when the overall sentiment label corresponding to each piece of training data is determined by voting of multiple evaluators of different genders, there may be no consensus between male evaluators and female evaluators, that is, the evaluation results of male evaluators alone are different from those of female evaluators alone (for example, the sentiment label from the male perspective is different from that from the female perspective). In this case of no consensus, the final voting result may be gender biased. For example, the voting result is the same as the evaluation result of the male evaluator and different from the evaluation result of the female evaluator (for example, biased towards the male perspective), or the voting result is the same as the evaluation result of the female evaluator and different from the evaluation result of the male evaluator (for example, biased towards the female perspective). On the other hand, when there is a consensus between male evaluators and female evaluators, the final voting result will not be gender biased. For example, the voting result is the same as the evaluation result of the male evaluator and the same as the evaluation result of the female evaluator.

[0037] In some embodiments, the labeled training set 200 with known bias can be divided into an unbiased first labeled training set S1 and a biased second labeled training set S2, wherein each training data in the second labeled training set S2 corresponds to a bias type (e.g., a first type of bias or a second type of bias).

[0038] For example, when the overall emotion label corresponding to a training data is the same as the emotion label from the male perspective and the same as the emotion label from the female perspective, this training data belongs to the first labeled training set S1; when the overall emotion label corresponding to a training data is the same as the emotion label from the male perspective but different from the emotion label from the female perspective, this training data is "biased towards the male perspective" and belongs to the second labeled training set S2; when the overall emotion label corresponding to a training data is the same as the emotion label from the female perspective but different from the emotion label from the male perspective, this training data is "biased towards the female perspective" and belongs to the second labeled training set S2.

[0039] Since the domain classifier 110 in the fairness constraint adversarial network 100 is used to perform bias classification, such as classifying input data as "first type bias" or "second type bias", the biased second labeled training set S2 can be used to train the domain classifier 110 in a supervised manner, including calculating the first loss associated with the domain classifier 110. On the other hand, the speech emotion recognition model 120 in the fairness constraint adversarial network 100 is used to perform emotion judgment, such as judging the emotion label corresponding to the input data, so the unbiased first labeled training set S1 and the biased second labeled training set S2 can both be used to train the speech emotion recognition model 120 in a supervised manner, including calculating the second loss associated with the speech emotion recognition model 120. The first loss and the second loss will be described in detail later.

[0040] It must be noted that, in addition to the characteristics of the training data, this document does not provide a detailed description and limitation on the format of the training data. In some embodiments, the training data may be the original voice signal. In some embodiments, the training data may be features after acoustic feature extraction (e.g., eGeMAPS feature extraction) of the voice signal. In some embodiments, the training data may be pre-processed voice data (e.g., based on pre-trained wav2vec, vq-wav2vec, HuBert or Whisper models), and the present disclosure is not limited thereto.

[0041] In some embodiments, the fairness constraint adversarial network 100 is designed to encourage the speech emotion recognition model 120 to generate unbiased embeddings based on the input data (e.g., in the fully connected layer before the output layer), and at the same time suppress the ability of the domain classifier 110 to classify the bias of the generated embeddings. In this way, when the generated embeddings of the speech emotion recognition model 120 are difficult for the domain classifier 110 to distinguish their biases, it means that the speech emotion recognition model 120 is trained to generate fair or unbiased embeddings and can also generate fair or unbiased emotion prediction results.

[0042] The above purpose can be achieved by designing the first loss function L used in training the fairness constraint adversarial network 100. REP To achieve.

[0043] In some embodiments, the first loss function L REP Negatively related to the first loss L associated with the domain classifier 110 1 (e.g., adversarial loss), and is positively correlated to the second loss L2 associated with the speech emotion recognition model 120. For example, the first loss function LREP can be expressed as: L REP=-L1+L2.

[0044] In some embodiments, the first loss L1 and the second loss L2 are, for example, the cross-entropy loss of the domain classifier 110 and the speech emotion recognition model 120. However, the present disclosure does not limit the types of the first loss L1 and the second loss L2, and those skilled in the art can select or design them according to their needs.

[0045] In some embodiments, the first loss function L REP Correction of the Wasserstein distance (WD) loss L associated with the domain classifier 110 D For example, the first loss function L REP It can be expressed as: L REP =-L1+αL D +L2,

[0046] Here, α is, for example, a hyperparameter.

[0047] In some embodiments, the Wasserstein distance loss L D For example, the distance can be obtained by calculating the Wasserstein distance between the features of the “first bias type” (eg, biased towards males) and the features of the “second bias type” (eg, biased towards females) classified by the domain classifier 110 .

[0048] In this way, using the labeled training set 200 with known bias, the parameters of the fairness constraint adversarial network 100 can be minimized by minimizing the first loss function L REP To complete the training, and at the same time train an unbiased speech emotion recognition model 120. Those skilled in the art can learn the training method of the fair constraint adversarial network from relevant literature, so the specific details are not repeated here.

[0049] Table 1 below shows the fairness of the unbiased speech emotion recognition model 120 trained according to the embodiment of the present disclosure in terms of statistical parity scores. The dataset used is a biased sub-dataset in the emotional speech database IEMOCAP. Table 1

[0050] In Table 1, DNN is a deep neural network; LFR is the model in “Learning fair representations” published by Rich Zemel, Yu Wu, Kevin Swersky, Toni Pitassi, and Cynthia Dwork in 2013; NRL is the model in “Learning fair representations via an adversarial framework” published by Rui Feng, Yang Yang, Yuehan Lyu, Chenhao Tan, Yizhou Sun, and Chunping Wang in 2019; FairRep G is a speech emotion recognition model 120 trained according to the embodiment of the present disclosure without considering the first loss L1 (for example, adversarial loss); FairRep F According to the embodiment of the present disclosure, without considering the Wasserstein distance loss L D FairRep is the unbiased speech emotion recognition model 120 trained according to the embodiment of the present disclosure.

[0051] It can be seen from Table 1 that, compared with previous models, the unbiased speech emotion recognition model 120 trained according to the embodiment of the present disclosure has better performance in almost every emotion.

[0052] In addition, it can be seen from Table 1 that when training the speech emotion recognition model 120, the Wasserstein distance loss L D Taking into account, compared with not considering the Wasserstein distance loss L D A speech emotion recognition model 120 with better performance can be trained.

[0053] <<Multi-view speech emotion recognition model>>

[0054] Figure 3 is a flowchart showing a method for training a multi-view speech emotion recognition model according to an embodiment of the present disclosure.

[0055] Please refer to Figure 1 and Figure 3 In step S310 , a fair embedding set 200 ′ corresponding to the labeled training set 200 with known bias is generated based on the unbiased speech emotion recognition model 120 . Specifically, the fair embedding set 200 ′ includes a fair embedding corresponding to each training data in the labeled training set 200 .

[0056] In some embodiments, in the architecture of the speech emotion recognition model 120, at least one fully connected layer is included before the output layer. For example, the previous layer of the output layer of the speech emotion recognition model 120 is a fully connected layer. Therefore, by inputting each training data in the labeled training set 200 into the unbiased speech emotion recognition model 120, the corresponding fair embedding can be obtained by extracting the above-mentioned fully connected layer. Accordingly, a fair embedding set 200' corresponding to the entire labeled training set 200 can be obtained.

[0057] In some embodiments, each piece of training data corresponds to an overall emotion label and multiple perspective emotion labels of multiple perspectives. Therefore, each fair embedding in the fair embedding set 200' also corresponds to an overall emotion label and multiple perspective emotion labels of multiple perspectives.

[0058] In step S320 , the multi-view speech emotion recognition model 130 is trained according to the fair embedding set 200 ′ and a second loss function, wherein the second loss function is positively correlated to the metric learning loss.

[0059] In some embodiments, the multi-perspective speech emotion recognition model 130 is used to generate emotion prediction results from multiple perspectives. For example, each piece of training data in the labeled training set 200 corresponds to an overall emotion label, a male perspective emotion label, and a female perspective emotion label, and according to an embodiment of the present disclosure, the multi-perspective speech emotion recognition model 130 trained based on the labeled training set 200 is used to generate emotion prediction results from the male perspective and emotion prediction results from the female perspective.

[0060] In some embodiments, the multi-view speech emotion recognition model 130 is, for example, a deep neural network composed of fully connected layers, but the present invention is not limited thereto. In other embodiments, as long as the emotion prediction results of the corresponding multiple views can be generated with fair embedding, the present invention does not limit the specific architecture of the multi-view speech emotion recognition model 130.

[0061] In order to train the multi-view speech emotion recognition model 130 , in addition to obtaining the overall emotion label corresponding to each fair embedding in the fair embedding set 200 ′, it is also necessary to obtain multiple view emotion labels of multiple views corresponding to each fair embedding.

[0062] by Figure 1For example, multiple fair embeddings in the fair embedding set 200' correspond to multiple perspective emotion labels of the first perspective 210 (e.g., male perspective), for example, represented by a first array A1 as [0, 1, 1, 0, 1, ..., 1, 0, 0], and multiple perspective emotion labels corresponding to the second perspective 220 (e.g., female perspective), for example, represented by a second array A2 as [1, 1, 0, 1, 0, ..., 0, 0, 1]. From the fact that the perspective emotion labels of the first perspective 210 and the second perspective 220 corresponding to the first fair embedding (for example, the first element of the first array A1 and the second array A2) are 0 and 1 respectively, it can be seen that the evaluator of the first perspective 210 (for example, a male evaluator) and the evaluator of the second perspective 220 (for example, a female evaluator) do not have a consensus on the evaluation results of the training data corresponding to the first fair embedding (for example, biased training data); from the fact that the perspective emotion labels of the first perspective 210 and the second perspective 220 corresponding to the second fair embedding (for example, the second element of the first array A1 and the second array A2) are both 1, it can be seen that the evaluator of the first perspective 210 and the evaluator of the second perspective 220 have a consensus on the evaluation results of the training data corresponding to the second fair embedding (for example, unbiased training data).

[0063] Based on the fair embedding set 200 ′, the multi-view speech emotion recognition model 130 can be trained by minimizing a second loss function that is positively correlated to the metric learning loss.

[0064] In some embodiments, the metric learning loss includes a triplet loss L T (Triplet Loss), so the second loss function L PER Positively correlated to the triplet loss L T .

[0065] In some embodiments, the triplet loss L T The anchor a(anchor), positive sample p(positive) and negative sample n(negative) are set based on the corresponding bias, and the corresponding bias is associated with the overall emotion label and multiple perspective emotion labels of multiple perspectives. For example, when the multiple perspective emotion labels are inconsistent and the overall emotion label is the same as one of the perspective emotion labels, it indicates that bias exists; when the multiple perspective emotion labels are consistent, it indicates that bias does not exist.

[0066] For example, when the training data corresponding to the first fair embedding in the fair embedding set 200' belongs to the first bias type (e.g., biased towards males), that is, the overall emotion label (e.g., represented by "1") corresponding to it is the same as the perspective emotion label of the first perspective 210 (e.g., the first element "1" of the first array A1) but different from the perspective emotion label of the second perspective 220 (e.g., the first element "0" of the second array A2), then the anchor point a corresponding to the first fair embedding can be set to the first fair embedding, the positive sample p can be set to other fair embeddings whose corresponding bias type is the same as the first fair embedding, that is, other fair embeddings corresponding to the first bias type, and the negative sample n can be set to a fair embedding whose corresponding bias type is different from the first fair embedding, that is, a fair embedding corresponding to the second bias type. Based on this, the triplet loss L corresponding to the first fair embedding T can be calculated.

[0067] For example, when the training data corresponding to the third fair embedding in the fair embedding set 200' belongs to the second bias type (e.g., biased towards females), that is, the overall emotion label (e.g., represented by "0") corresponding to it is the same as the perspective emotion label of the second perspective 220 (e.g., the third element "0" of the second array A2) but different from the perspective emotion label of the first perspective 210 (e.g., the third element "1" of the first array A1), then the anchor point a corresponding to the third fair embedding can be set to the third fair embedding, the positive sample p can be set to other fair embeddings that are the same as the third fair embedding and of the second bias type, and the negative sample n can be set to a fair embedding that is different from the third fair embedding and corresponds to the first bias type. Based on this, the triplet loss L corresponding to the third fair embedding T can be calculated.

[0068] In some embodiments, the second loss function L PER It is also positively correlated with the third loss L3 associated with the multi-view speech emotion recognition model 130. For example, the second loss function L PER It can be expressed as: L PER =L3+λL T ,

[0069] Where λ is, for example, the third loss L3 and the triplet loss L T The weights between , which can be considered as hyperparameters, for example.

[0070] In some embodiments, the third loss L3 is associated with the perspective emotion labels of the perspectives. For example, the third loss L3 may be the cross entropy loss of the multi-perspective speech emotion recognition model 130. However, the present disclosure does not limit the type of the third loss L3, and those skilled in the art may select or design it according to their needs.

[0071] In some embodiments, the metric learning loss may include one or more of the following losses: triplet loss, quadruplet loss, N-pair loss, contrastive loss, and center loss.

[0072] In some embodiments, in addition to the metric learning loss and the third loss L3, the second loss function L PER Other types of losses may also be involved, and the present disclosure is not limited thereto.

[0073] In this way, using the fair embedding set 200 ′, the parameters of the multi-view speech emotion recognition model 130 can be minimized by minimizing the second loss function L PER To complete the training, it can output emotion prediction results from multiple perspectives.

[0074] Table 2 below shows the performance of the multi-view speech emotion recognition model 130 trained according to the embodiment of the present disclosure in terms of consistency scores. The dataset used is a biased sub-dataset of the emotional speech database IEMOCAP. Table 2

[0075] In Table 2, FairPer is the multi-view speech emotion recognition model 130 trained according to the embodiment of the present disclosure.

[0076] It can be seen from Table 2 that compared with previous models, the multi-perspective speech emotion recognition model 130 trained according to the embodiment of the present disclosure can have a higher consistency score for data with more serious bias after passing through the multi-perspective speech emotion recognition model 130, which also means that the predictions made by the multi-perspective speech emotion recognition model 130 for multiple perspectives are more consistent and perform better.

[0077] <<Application of Speech Emotion Recognition Model>>

[0078] Figure 4 is a schematic diagram showing speech emotion recognition according to an embodiment of the present disclosure; Figure 5 is a flow chart showing speech emotion recognition according to an embodiment of the present disclosure.

[0079] Please refer to Figure 4 and Figure 5 The speech emotion recognition system 400 includes an unbiased speech emotion recognition model 120 and a multi-view speech emotion recognition model 130 .

[0080] In step S510 , voice data 410 is received.

[0081] In some embodiments, the speech data 410 may be speech data 410 received from an input element such as a microphone of the speech emotion recognition system 400. However, the present disclosure does not limit the source of the speech data 410.

[0082] In step S520 , firstly, the unbiased speech emotion recognition model 120 is used to generate a fair embedding 410 ′ corresponding to the speech data 410 .

[0083] In some embodiments, in the architecture of the speech emotion recognition model 120, at least one fully connected layer is included before the output layer. For example, the previous layer of the output layer of the speech emotion recognition model 120 is a fully connected layer. Therefore, when the speech data 410 is input into the unbiased speech emotion recognition model 120, the corresponding fair embedding 410' can be obtained by extracting the above-mentioned fully connected layer.

[0084] In some embodiments, before the speech data 410 is input into the speech emotion recognition model 120, the speech data 410 may be pre-processed by, for example, extracting acoustic features, etc., but the present disclosure is not limited thereto. The method of extracting acoustic features or pre-processing the speech data has been illustrated in the previous paragraphs, so it will not be repeated here.

[0085] In step S530 , based on the fair embedding 410 ′ corresponding to the speech data 410 , an emotion prediction result is generated using the unbiased speech emotion recognition model 120 or the multi-view speech emotion recognition model 130 .

[0086] In some embodiments, the emotion prediction result may include an emotion corresponding to a single binary (e.g., whether happy); in some embodiments, the emotion prediction result may include emotions corresponding to multiple binary (e.g., whether happy, angry, frustrated, etc.); in some embodiments, the emotion prediction result may include an emotion corresponding to a single quantified emotion (e.g., a happiness score); in some embodiments, the emotion prediction result may include emotions corresponding to multiple quantified emotions (e.g., a happiness score, an anger score, a frustration score, etc.). In other words, the present disclosure does not limit the specific format of the emotion prediction results generated by the speech emotion recognition models 120 and 130, and those skilled in the art may design them according to their needs.

[0087] In some embodiments, after the speech data 410 is converted into a fair embedding 410 ′ using the unbiased speech recognition model 120 in step S520 , an unbiased emotion prediction result is obtained directly from the output layer of the unbiased speech recognition model 120 in step S530 .

[0088] In some embodiments, the fair embedding 410 ′ corresponding to the speech data 410 is input into the multi-view speech emotion recognition model 130 to generate emotion prediction results for multiple views.

[0089] In some embodiments, the speech emotion recognition system 400 may, for example, provide a user switching function to allow the user to select at least one of the unbiased and multi-perspective speech emotion recognition services. If the user selects the unbiased speech emotion recognition service, the speech emotion recognition system 400 will (for example, through the output element of the speech emotion recognition system 400) output the unbiased speech emotion recognition result; if the user selects the speech emotion recognition service of the first perspective in the multi-perspective, the speech emotion recognition system 400 will output the emotion prediction result of the first perspective; if the user selects the speech emotion recognition service of the second perspective in the multi-perspective, the speech emotion recognition system 400 will output the emotion prediction result of the second perspective, and so on.

[0090] For example, the speech emotion recognition system 400 may provide a switching function, so that the user can select at least one of the speech emotion recognition services of unbiased, male perspective, and female perspective. If the unbiased speech emotion service is selected, the speech emotion recognition system 400 will output the unbiased speech emotion recognition result; if the speech emotion service of male perspective and / or female perspective is selected, the fair embedding 410' will be input into the multi-perspective speech emotion recognition model 130 to generate and output the emotion prediction results of male perspective and / or female perspective.

[0091] In addition to providing unbiased voice emotion recognition services, the disclosed embodiments further provide voice emotion recognition services from specific perspectives that can be selected by users. In this way, the overall system is fairer and more valuable for reference for users.

[0092] Figure 6 is a block diagram illustrating a computing system according to an embodiment of the present disclosure.

[0093] Please refer to Figure 6The computer-implemented methods such as the speech emotion recognition model training method and the speech emotion recognition method introduced in this article can be implemented on a computing system 600 having multiple hardware components. In some embodiments, the computing system 600 can be implemented in the form of an electronic device, which includes, but is not limited to, one or more of the following components: a processor (Central Processing Unit, CPU) 610, a graphics processing unit (Graphics Processing Unit, GPU) 620, an input / output component 630, a network component 640, and a memory 650. The above components can communicate and transmit data via the system bus 660. However, the present disclosure does not limit the specific model, quantity, and configuration of each component. Those skilled in the art can adjust, select, or add or reduce components according to the specific needs and operating environment when implementing the present disclosure.

[0094] In some embodiments, the main computing core within the computing system 600 is one or more processors 610. The processor 610 may be responsible for running the main computing process and related control logic of algorithms such as deep learning. In some embodiments, the processor 610 is used to execute processing instructions (i.e., machine executable instructions) stored in a non-volatile computer-readable medium (e.g., storage device 670).

[0095] In some embodiments, in order to improve the computing efficiency of deep learning, the computing system 600 may also include one or more graphics processors 620 designed for large-scale parallel computing. The graphics processor 620 can effectively improve the computing power of the system when performing deep learning training and reasoning.

[0096] In some embodiments, the computing system 600 may include a variety of input / output components 630 for receiving user input and displaying system output. For example, the input / output components 630 may include a keyboard, a mouse, a touch pad, a display screen, a speaker, and other types of sensing devices.

[0097] In some embodiments, the computing system 600 may also include a network element 640 for network communication. For example, the network element 640 may include a network interface card for wired or wireless network connection, or a communication module for 3G, 4G, 5G or other wireless communication technologies.

[0098] In some embodiments, the computing system 600 includes one or more memory elements 650, such as volatile memory elements such as random access memory (RAM). The memory 650 stores the parameters of the deep learning model and other data and programs used to run algorithms such as deep learning.

[0099] In addition, the computing system 600 may further include one or more of the following components: a storage device 670 , a power management component 680 , and various other hardware components 690 .

[0100] In some embodiments, the computing system 600 may include one or more storage devices 670, such as non-volatile memory devices such as hard disk drives (HDD) or solid state drives (SSD). These storage devices 670 can be used to store information such as program code, training data, model parameters, etc. of deep learning software. In addition, the storage device 670 can also be used to store intermediate results and final outputs of deep learning algorithms.

[0101] In some embodiments, the computing system 600 may include one or more power management components 680 for providing power to various hardware components of the computing system 600 and managing their power consumption. The power management components 680 may include batteries, power converters, and other power management devices.

[0102] In some embodiments, the computing system 600 may further include various other hardware components 690 , such as cooling fans, heat sinks, and various other control and monitoring devices, but the present disclosure is not limited thereto.

[0103] In addition, the embodiments of the present invention may also be implemented as one or more computer program products, including a computer program having one or more instructions. Specifically, a computer program (also referred to as a program, software, script, or code) may be presented in any form of programming language, and the computer program may be deployed in any form. During the operation of the computing system 600 (e.g., an electronic device), the instructions or a portion thereof may also reside completely or at least partially in the processor 610 so that the processor 610 performs the methods described herein.

[0104] In summary, the speech emotion recognition method and system proposed in the embodiments of the present disclosure further improves the performance of the unbiased speech emotion recognition model trained by adding Wasserstein distance loss to the loss function of training the fair constrained adversarial network. In addition, the embodiments of the present disclosure further propose a method of first learning fair embeddings and then further using metric learning losses such as triplet loss to train a multi-perspective speech emotion recognition model. In this way, in addition to providing users with unbiased speech emotion recognition services, it can also further provide users with speech recognition services from specific perspectives that can be selected.

[0105] Although this paper includes many details, these details should not be interpreted as limitations to the scope of the claimed invention, but are interpreted as descriptions of features peculiar to specific embodiments. Some features described in the context of different embodiments herein may also be combined in a single embodiment. On the contrary, the various features described in the context of a single embodiment may also be separated or implemented in multiple embodiments with any suitable sub-combination. In addition, although features may be described as acting with some combinations and even initially claimed above, one or more features from the claimed combination may be removed from the combination in some cases, and the claimed combination may be directed to a variation of a sub-combination or a sub-combination. Similarly, although operations are depicted in a particular order in the drawings, this should not be understood as requiring to perform such operations in the particular order shown or in a sequential order, or to perform all described operations to achieve desired results.

[0106] Only a few implementations and examples are described, and other implementations, enhancements, and variations may be made based on what is described and illustrated in this disclosure.

Claims

1. A computer-implemented method for training a speech emotion recognition model, It is characterized in that include: A fairness constraint adversarial network is provided, wherein the fairness constraint adversarial network includes a domain classifier and a speech emotion recognition model, wherein the domain classifier is used to perform bias classification on input data, and the speech emotion recognition model is used to determine the emotion corresponding to the input data; as well as According to a labeled training set with known bias and a first loss function, the fairness constrained adversarial network is trained to obtain the unbiased speech emotion recognition model, The first loss function is positively correlated to a Wasserstein distance loss.

2. The computer-implemented method of claim 1, It is characterized in that The first loss function is negatively correlated to a first loss associated with the domain classifier, and positively correlated to a second loss associated with the speech emotion recognition model.

3. The computer-implemented method of claim 1, It is characterized in that The labeled training set with known bias includes a plurality of training data, and each of the plurality of training data corresponds to an overall emotion label and a plurality of perspective emotion labels of a plurality of perspectives.

4. The computer-implemented method of claim 3, It is characterized in that Also includes: Based on the unbiased speech emotion recognition model, generating a fair embedding set corresponding to the labeled training set; as well as A multi-view speech emotion recognition model is trained based on the fair embedding set and a second loss function, wherein the multi-view speech emotion recognition model is used to generate multiple specific view emotion prediction results corresponding to the multiple views, and the multiple second loss functions are positively correlated to a metric learning loss.

5. The computer-implemented method of claim 4, It is characterized in that The metric learning loss includes a triplet loss, and an anchor point, a positive sample, and a negative sample of the triplet loss are set based on the overall emotion label and the multiple perspective emotion labels.

6. The computer-implemented method of claim 4, It is characterized in that The metric learning loss includes at least one of a triplet loss, a quadruple loss, an N-pair loss, a contrast loss, and a center loss.

7. The computer-implemented method of claim 4, It is characterized in that The second loss function is positively correlated to a third loss associated with the multi-view speech emotion recognition model, and the third loss is associated with the multiple view emotion labels.

8. The computer-implemented method of claim 1, It is characterized in that The domain classifier is used to perform a binary biased classification.

9. The computer-implemented method of claim 8, It is characterized in that The binary bias includes an evaluator gender bias.

10. A computer-implemented method for speech emotion recognition, It is characterized in that include: receiving a voice data; as well as Based on the speech data and an unbiased speech emotion recognition model, an emotion prediction result corresponding to the speech data is generated, wherein: The unbiased speech emotion recognition model is trained by training a fairness constrained adversarial network based on a labeled training set with known bias and a first loss function. The first loss function is positively correlated to a Wasserstein distance loss, and The fairness constraint adversarial network includes a domain classifier for bias classification and the speech emotion recognition model.

11. The computer-implemented method of claim 10, It is characterized in that The first loss function is negatively correlated to a first loss associated with the domain classifier, and positively correlated to a second loss associated with the speech emotion recognition model.

12. The computer-implemented method of claim 10, It is characterized in that The labeled training set with known bias includes a plurality of training data, and each of the plurality of training data corresponds to an overall emotion label and a plurality of perspective emotion labels of a plurality of perspectives.

13. The computer-implemented method of claim 12, It is characterized in that Also includes: Based on the speech data and the unbiased speech emotion recognition model, generating a fair embedding corresponding to the speech data; as well as Based on the fair embedding and a multi-view speech emotion recognition model, at least one of a plurality of specific view emotion prediction results corresponding to the speech data is generated, wherein: The multi-view speech emotion recognition model is trained according to a fair embedding set corresponding to the labeled training set and a second loss function, The fair embedding set is generated based on the unbiased speech emotion recognition model, and The second loss function is positively correlated to a metric learning loss.

14. The computer-implemented method of claim 13, It is characterized in that The metric learning loss includes a triplet loss, and an anchor point, a positive sample, and a negative sample of the triplet loss are set based on the overall emotion label and the multiple perspective emotion labels.

15. The computer-implemented method of claim 13, It is characterized in that The metric learning loss includes at least one of a triplet loss, a quadruple loss, an N-pair loss, a contrast loss, and a center loss.

16. The computer-implemented method of claim 13, It is characterized in that The second loss function is positively correlated to a third loss associated with the multi-view speech emotion recognition model, and the third loss is associated with the multiple view emotion labels.

17. The computer-implemented method of claim 10, It is characterized in that The domain classifier is used to perform a binary biased classification.

18. The computer-implemented method of claim 17, It is characterized in that The binary bias includes an evaluator gender bias.

19. A computer-readable storage medium storing a program, the program comprising instructions, It is characterized in that When the instructions are executed by one or more processors of an electronic device, the instructions cause the electronic device to perform the computer-implemented method according to any one of claims 10 to 18.

Images