Emotion recognition learning method, emotion recognition method, emotion recognition learning device, emotion recognition device and program

Abstract

The present invention involves: learning data that, for a plurality of speakers, includes a plurality of ways of input utterances by the speakers, a plurality of ways of ordinary-state emotion utterances corresponding to the input utterances, and correct-answer labels for emotion corresponding to the input utterances; and a model that receives inputs of an input utterance and a corresponding ordinary-state emotion utterance to output an emotion recognition result. The invention causes a computer to execute a learning procedure for training the model on the basis of a first loss function for minimizing a difference (error) between a correct-answer label corresponding to the input utterance input to the model and the recognition result output by the model on the basis of the learning data, and a second loss function for achieving, for every speaker, a constant vector representing the quality of emotional expression calculated for the ordinary-state emotion utterance in the course of the model outputting the recognition result. In this way, the invention contributes to an increase in the accuracy of recognizing the emotion of a speaker from an utterance.

Description

[Technical Field]

[0001] The present invention relates to an emotion recognition learning method, an emotion recognition method, an emotion recognition learning device, an emotion recognition device, and a program. [Background technology]

[0002] Recognizing a speaker's emotions from their speech is an important technology. For example, recognizing a speaker's emotions during counseling can visualize a patient's feelings of anxiety or sadness, which is expected to deepen the counselor's understanding and improve the quality of their guidance. Furthermore, recognizing human emotions in human-machine dialogue can enable the construction of a more friendly dialogue system, such as sharing in a person's joy when they are happy or encouraging a person when they are sad. Hereafter, we refer to the technology that takes an utterance as input and estimates which emotion class (e.g., neutral, anger, joy, sadness, etc.) the speaker's emotion contained in that utterance belongs to as "emotion recognition."

[0003] Among emotion recognition techniques, a technology (hereinafter referred to as "conventional technology") that improves recognition accuracy by using utterances made when expressing "normal" emotions (emotions that are neither positive emotions such as joy nor negative emotions such as anger or sadness, but are in a normal state) (hereinafter referred to as "normal emotional utterances") has been proposed in Patent Document 1 and Non-Patent Document 1. The conventional technology is based on the hypothesis that "if a person's normal speaking style (=normal emotional utterances) is known, the accuracy of emotion recognition for that person will improve."

[0004] Figure 1 is a diagram illustrating the conventional technology. In the conventional technology, estimation requires not only the input utterance to be recognized but also normal emotional utterances from the same speaker as the input utterance, and these are input into an emotion recognition model to obtain emotion recognition results. Inside the emotion recognition model, an "emotion expression vector extraction block" is first used for each of the input utterance and the normal emotional utterance to extract an emotional expression vector that represents the nature of the emotional expression of the entire utterance. Then, based on the emotional expression vectors of the input utterance and the normal emotional utterance, an "emotion estimation block for input utterance using the emotional expression vector of normal emotional utterance" is used to estimate the emotion of the input utterance. The emotion recognition model uses a statistical model based on deep learning, and each block in the emotion recognition model is simultaneously trained prior to estimation using a set of labeled data consisting of pairs of the input utterance, the normal emotional utterance, and the correct emotion label of the input utterance (the correct value for emotion recognition of the input utterance). [Prior art documents] [Patent documents]

[0005] [Patent Document 1] International Publication No. 2021 / 171552 [Non-patent literature]

[0006] [Non-Patent Document 1] Andreas Triantafyllopoulos, Shuo Liu, Bjoern W. Schuller, "DEEP SPEAKER CONDITIONING FOR SPEECH EMOTION RECOGNITION", Proc. of ICME, pp.1-6, 2021 Summary of the Invention [Problem to be solved by the invention]

[0007] Conventional technologies have the problem that the recognition results change when different normal emotional utterances are used during recognition (for example, when input utterance X and normal emotional utterance A are used, the recognition result for input utterance X may be "joy," whereas when X and normal emotional utterance B are used, the recognition result for input utterance X may be "sadness.") This is because, when training an emotion recognition model, a set of input utterance, normal emotional utterance, and correct emotion label for the input utterance is used, so each block in the emotion recognition model is optimized to output recognition results that are specialized for combinations of input utterance and normal emotional utterance. Conventional technologies also use a method of including combinations with various normal emotional utterances in the training data to prevent the problem of excessive specialization in combinations, but this method does not explicitly reduce specialization in combinations and therefore does not adequately address the problem.

[0008] The present invention has been made in view of the above points, and aims to contribute to improving the accuracy of recognizing a speaker's emotion from an utterance. [Means for solving the problem]

[0009] Therefore, in order to solve the above problem, a computer executes a learning procedure to train a model based on training data including, for a plurality of speakers, a plurality of input utterances by speakers, a plurality of normal emotional utterances corresponding to each of the input utterances, and correct labels of emotions corresponding to each of the input utterances, using a first loss function to minimize an error in an emotion recognition result output by a model to which any of the input utterances and the normal emotional utterance corresponding to that input utterance are input, relative to the correct label corresponding to that input utterance, and a second loss function to keep constant for each speaker a vector representing the nature of emotional expression calculated for the normal emotional utterance in the process of the model outputting the recognition result. [Effects of the Invention]

[0010] This can contribute to improving the accuracy of recognizing a speaker's emotions from their speech. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 is a diagram for explaining a conventional technique. [Figure 2] 1 is a diagram illustrating an example of a hardware configuration of an emotion recognition device 10 according to an embodiment of the present invention. [Figure 3] FIG. 2 is a diagram illustrating an example of a functional configuration during learning of the emotion recognition device 10 according to the embodiment of the present invention. [Figure 4] FIG. 2 is a diagram illustrating an example of the configuration of an emotion recognition model m1 according to an embodiment of the present invention. [Figure 5] FIG. 1 is a diagram for explaining an outline of a learning method for an emotion recognition model m1. [Figure 6] FIG. 2 is a diagram illustrating an example of a functional configuration of the emotion recognition device 10 during emotion recognition in an embodiment of the present invention. [Figure 7] 10 is a flowchart illustrating an example of a processing procedure executed by the emotion recognition device 10 during learning. [Figure 8] FIG. 10 is a diagram for explaining calculation of an average value of emotional expression vectors of normal emotional utterances for each speaker. [Figure 9] FIG. 10 is a diagram for explaining calculation of the distance Sji,k between each emotional expression vector eji and each speaker average ck. DETAILED DESCRIPTION OF THE INVENTION

[0012] In this embodiment, when training an emotion recognition model using normal emotional utterances (utterances expressing emotions in a normal state that are neither positive emotions such as joy nor negative emotions such as anger or sadness), in addition to the loss function used in conventional methods for minimizing the error between the correct emotion label and the emotion recognition result (the error of the emotion recognition result for the correct emotion label), a new loss function (speaker identity loss function for normal emotional utterances) is introduced to ensure that the emotional expression vectors of normal emotional utterances show the same vector value if the speaker is the same (in other words, even if the normal emotional utterance changes, they show a constant vector value if the speaker is the same). This optimizes the emotional expression vector extraction block so that a constant emotional expression vector of normal emotional utterance is obtained for different normal emotional utterances, solving the problem of specialization for combinations of input utterances and normal emotional utterances.

[0013] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0014] Fig. 2 is a diagram showing an example of the hardware configuration of the emotion recognition device 10 according to an embodiment of the present invention. The emotion recognition device 10 in Fig. 2 includes a drive device 100, an auxiliary storage device 102, a memory device 103, a processor 104, and an interface device 105, which are all interconnected via a bus B.

[0015] A program that realizes processing in the emotion recognition device 10 is provided by a recording medium 101 such as a CD-ROM. When the recording medium 101 storing the program is set in the drive device 100, the program is installed from the recording medium 101 to the auxiliary storage device 102 via the drive device 100. However, the program does not necessarily have to be installed from the recording medium 101, but may be downloaded from another computer via a network. The auxiliary storage device 102 stores the installed program as well as necessary files, data, etc.

[0016] When an instruction to start the program is received, the memory device 103 reads out and stores the program from the auxiliary storage device 102. The processor 104 is a CPU or a GPU (Graphics Processing Unit), or a CPU and a GPU, and executes functions related to the emotion recognition device 10 in accordance with the program stored in the memory device 103. The interface device 105 is used as an interface for connecting to a network.

[0017] FIG. 3 is a diagram showing an example of the functional configuration of the emotion recognition device 10 during learning in an embodiment of the present invention. As shown in FIG. 3, the emotion recognition device 10 (emotion recognition learning device) during learning has two acoustic feature extraction units 11 (acoustic feature extraction unit 11-1, acoustic feature extraction unit 11-2) and one learning unit 12. These units are realized by processing in which one or more programs installed in the emotion recognition device 10 are executed by the processor 104. The emotion recognition device 10 also uses a learning data storage unit 121. The learning data storage unit 121 can be realized using, for example, the auxiliary storage device 102 or a storage device connectable to the emotion recognition device 10 via a network.

[0018] [Learning data storage unit 121] The training data storage unit 121 stores a large amount of training data in advance, which are sets of input utterances, correct emotion labels (correct emotion labels) for the input utterances, normal emotional utterances by the same speaker as the input utterance, and speaker labels that identify the speaker of the normal emotional utterances. The input utterances and normal emotional utterances in each training data set are assumed to be different utterances. The correct emotion labels refer to the correct value of emotion recognition for the input utterances. Emotion recognition refers to estimating, based on an utterance, which emotion of the speaker contained in the utterance belongs to an emotion class (e.g., normal, anger, joy, sadness, etc.). Normal emotional utterances refer to utterances spoken with "normal" emotions. Furthermore, "utterance" refers to speech (audio data) generated by actions that express linguistic expressions. The input utterances and normal emotional utterances in each training data set differ in emotion and content (text to be spoken). The training data storage unit 121 stores such training data for multiple speakers. The content of each speaker's utterance may be different or the same. It is also desirable to include multiple sets of training data with different utterance content for the same speaker. This is to avoid the possibility of constructing an emotion recognition model that returns a specific emotion recognition result based on the utterance content (phonological bias) if each speaker speaks the same content. In this case, it is desirable that the normal emotional utterances in each set of training data for the same speaker are also different. Therefore, it can be said that the training data storage unit 121 stores, as training data for multiple speakers, multiple input utterances by the speaker, multiple normal emotional utterances corresponding to each input utterance, and correct emotion labels corresponding to each input utterance.

[0019] [Acoustic feature extraction unit 11] The acoustic feature extraction unit 11 receives an utterance (audio data of the utterance) as input, extracts an acoustic feature sequence from the utterance, and outputs the acoustic feature sequence. During training, the acoustic feature extraction unit 11-1 extracts an acoustic feature sequence from the input utterance of each training data, and the acoustic feature extraction unit 11-2 extracts an acoustic feature sequence from a normal emotion utterance of each training data.

[0020] An acoustic feature sequence is data obtained by dividing an input utterance into short-time windows, calculating acoustic features for each short-time window, and arranging the acoustic feature vectors in chronological order. The acoustic features include one or more of the following: power spectrum, Mel filter bank output, MFCC, fundamental frequency, logarithmic power, HNR (Harmonics-to-Noise Ratio), speech probability, zero-crossing count, and their first or second derivatives. The speech probability is calculated, for example, by the likelihood ratio of a pre-trained speech / non-speech GMM model. The HNR is calculated, for example, by a cepstrum-based method (see Peter Murphy and Olatunji Akande, "Cepstrum-Based Harmonics-to-Noise Ratio Measurement in Voiced Speech," Lecture Notes in Artificial Intelligence, Nonlinear Speech Modeling and Applications, Vol. 3445, Springer-Verlag, 2005). Using more acoustic features allows for the expression of various characteristics contained in speech, which tends to improve emotion recognition accuracy.

[0021] [Study Section 12] The learning unit 12 inputs the acoustic feature sequences of the input utterance and normal emotional utterance into an emotion recognition model (hereinafter simply referred to as "emotion recognition model m1") based on the acoustic feature sequences output from the acoustic feature extraction unit 11 for each of the input utterance and normal emotional utterance of each piece of learning data, and the correct emotion label and speaker label of each piece of learning data, and learns the emotion recognition model m1 using the correct emotion label and speaker label of the input utterance as training data.

[0022] Fig. 4 is a diagram showing an example of the configuration of emotion recognition model m1 in an embodiment of the present invention. As shown in Fig. 4, emotion recognition model m1 includes two emotional expression vector extraction blocks m11 (emotion expression vector extraction block m11-1 and emotional expression vector extraction block m11-2) and one emotion probability estimation block m12.

[0023] In the process in which emotion recognition model m1 recognizes the speaker's emotion, emotional expression vector extraction block m11 converts the acoustic feature sequence into a fixed-length vector (hereinafter referred to as an "emotional expression vector") that represents the nature (or characteristics) of the emotional expression of the entire utterance. Emotional expression vector extraction block m11 uses a deep learning model (for example, a model composed of a Transformer and Self Attentive Pooling) that extracts fixed-length expressions from an input vector sequence of any length. In Figure 4, emotional expression vector extraction block m11-1 converts the acoustic feature sequence extracted from the input utterance into an emotional expression vector. Emotional expression vector extraction block m11-2 converts the acoustic feature sequence extracted from normal emotional utterance into an emotional expression vector.

[0024] The emotion probability estimation block m12 uses a deep learning model (for example, a model with one or more layers of fully connected layers and activation functions) that projects a vector representing the posterior probability of each emotion (a vector in which each dimension indicates the posterior probability of a different emotion) from a fixed-length vector (emotion expression vector).

[0025] As shown in FIG. 5, the training unit 12 uses a speaker identity loss function L p (Figure A3) and the loss function L based on the error (error) between the output of the emotion recognition model m1 and the correct emotion label. a (Loss function for minimizing the error between the correct emotion label of the input utterance and the emotion recognition result) and the loss function L (L = αL a +(1-α)L p ) and updates the model parameters of all models based on the model parameter m1. α is a manually determined weighting coefficient. The learning unit 12 updates the model parameters using the stochastic gradient descent method, as in conventional technology. After updating the model parameters a certain number of times, the learning unit 12 outputs the finally obtained emotion recognition model m1.

[0026] 6 is a diagram showing an example of the functional configuration of the emotion recognition device 10 in the embodiment of the present invention when recognizing emotions. In Fig. 6, the same parts as those in Fig. 3 are given the same reference numerals, and their description will be omitted.

[0027] As shown in Fig. 6, during emotion recognition, the emotion recognition device 10 has an emotion recognition unit 13 instead of the learning unit 12. The emotion recognition unit 13 is realized by processing in which one or more programs installed in the emotion recognition device 10 are executed by the processor 104. Furthermore, during emotion recognition, the emotion recognition device 10 does not use the learning data storage unit 121. Note that different computers may be used during learning and emotion recognition.

[0028] [Emotion recognition part 13] The emotion recognition unit 13 receives as input an acoustic feature sequence extracted by the acoustic feature extraction unit 11-1 from the input utterance of a person whose emotion is to be recognized, and an acoustic feature sequence extracted by the acoustic feature extraction unit 11-2 from the person's normal emotional utterance, and outputs an emotion recognition result (hereinafter simply referred to as the "emotion recognition result") by comparing the acoustic feature sequence with the normal emotional utterance by forward propagating the acoustic feature sequence through a trained emotion recognition model m1. The emotion recognition result output by the emotion recognition unit 13 includes a posterior probability vector for each emotion (the output of the forward propagation of the emotion recognition model m1) and the emotion class with the maximum posterior probability in the posterior probability vector. The emotion class with the maximum posterior probability is used as the final emotion recognition result.

[0029] The processing procedure executed by the emotion recognition device 10 will be described below.

[0030] [Study time] FIG. 7 is a flowchart illustrating an example of a processing procedure executed by the emotion recognition device 10 during learning.

[0031] In step S101, the learning unit 12 acquires one mini-batch (hereinafter referred to as a "target mini-batch") by randomly selecting a plurality of (N) pieces of learning data from the learning data group stored in the learning data storage unit 121. More precisely, the learning unit 12 randomly rearranges the learning data and then generates the mini-batch.

[0032] Next, the emotion recognition device 10 executes a loop process including steps S102 to S104 for each of the N pieces of training data included in the target mini-batch. The training data being processed in this loop process will be referred to as "target training data" hereinafter.

[0033] In step S102, the acoustic feature extraction unit 11-1 extracts an acoustic feature sequence from the input utterance of the target training data, and the acoustic feature extraction unit 11-2 extracts an acoustic feature sequence from the normal emotion utterance of the target training data.

[0034] Next, the learning unit 12 inputs the two extracted acoustic feature sequences into emotional expression model a, and acquires the recognition results (posterior probability vectors for each emotion) and emotional expression vectors of normal emotional utterances output by emotional expression model a (S103). That is, in the process in which emotional expression model a outputs the recognition results, emotional expression vector extraction block m11-2 calculates emotional expression vectors related to normal emotional utterances, and the learning unit acquires these emotional expression vectors. The learning unit 12 stores the acquired emotional expression vectors in association with the speaker labels of the target training data.

[0035] Next, the learning unit 12 calculates a loss function L based on the correct emotion labels of the target learning data and the recognition results (the posterior probability vectors of each emotion). a By calculating the cross entropy function between the correct emotion label of the input utterance and the posterior probability vector of each emotion, the loss (hereinafter referred to as loss L) based on the error of the recognition result relative to the correct emotion label is calculated. a ") is calculated (S104).

[0036] When steps S102 to S104 have been executed for all training data in the target mini-batch, the training unit 12 calculates the average value for each speaker of the emotional expression vectors (of normal emotional utterances) acquired for each training data in the target mini-batch in step S103 (S105).

[0037] FIG. 8 is a diagram for explaining the calculation of the average value of the emotional expression vector of normal emotional utterances for each speaker. In FIG. 8, e ji indicates the emotional expression vector of the i-th normal emotional utterance of speaker j in the target mini-batch. The speaker of a certain emotional expression vector can be identified based on the speaker label associated with the emotional expression vector in step S103. Fig. 8 shows an example in which emotional expression vectors of normal emotional utterances are obtained for each speaker whose speaker label is 1, 2, or 3.

[0038] The learning unit 12 calculates the average value of each set of emotion expression vectors that are associated with the same (common) speaker label. Hereinafter, the average value for each speaker will be referred to as the "speaker average c k " (k is the speaker label). In the example of FIG. 8, the speaker label is 1, 2, or 3, so speaker means c1, c2, and c3 are calculated.

[0039] Next, the learning unit 12 calculates the emotional expression vector e ji For each speaker, the average c k Distance S ji,k is calculated (S106).

[0040] Figure 9 shows the emotional expression vectors e ji and the average c for each speaker k Distance S ji,k FIG. 10 is a diagram for explaining the calculation of

[0041] Figure 9 shows the ji Each row corresponds to each c k The matrix to which each column corresponds is shown in Fig. Each element S of the matrix ji,k is e ji and c kThe learning unit 12 calculates the distance based on, for example, the following formula:

[0042]

number

[0043] Next, the learning unit 12 calculates the following loss function L p Using this, the loss based on the distance (hereinafter referred to as "loss L p ") is calculated (S107).

[0044]

number

[0045] In other words, the loss function L p is a loss function for making the emotional expression vector calculated for normal emotional utterances constant for each speaker in the process in which the emotion recognition model m1 outputs the recognition results (for minimizing the error in the emotional expression vector for each speaker).

[0046] Next, the learning unit 12 calculates the loss function L pLoss based on L p and the loss function L a Loss based on L a The weighted sum of these is calculated as the loss L of the entire emotion recognition model m1 (S108). Here, the calculation formula for the loss L is, for example, as follows: L = αΣ(L a / N)+(1-α)L p However, α is L a The role of "reducing emotion recognition errors" and L p This is a hyperparameter (weighting coefficient) for adjusting which of the roles "the normal emotion vector of the same speaker takes a similar value" is prioritized. a The larger the weight of L p The effect of L p The larger the weight, the more emotion recognition errors are tolerated.

[0047] Also, N is the number of training data in the target mini-batch. That is, Σ(L a / N), is the L a is the average value of

[0048] Next, the learning unit 12 uses the backpropagation algorithm based on L to learn each block in the emotion recognition model m1 and the loss function L P The parameters of the blocks are simultaneously updated (S109). By repeating steps S101 to S109, the parameters of the blocks are simultaneously optimized.

[0049] When steps S101 to S109 are repeated a predetermined number of times (Yes in S110), the learning unit 12 ends the processing procedure in FIG.

[0050] [When recognized] The process executed by the emotion recognition device 10 during recognition is as explained using Figure 6. That is, the emotion recognition device 10 receives as input an utterance from a speaker to be the target of emotion recognition and the speaker's normal emotional utterance. The acoustic feature extraction unit 11-1 extracts an acoustic feature sequence from the utterance, and the acoustic feature extraction unit 11-2 extracts an acoustic feature sequence from the normal emotional utterance. The emotion recognition unit 13 inputs these two acoustic feature sequences to a trained emotion recognition model m1 and recognizes the speaker's emotion based on the probability for each emotion label output by the emotion recognition model m1.

[0051] As described above, according to this embodiment, the emotional expression vector extraction block m11-2 is p Based on this, the system learns to output similar (relatively small distance) emotional expression vectors for multiple normal emotional utterances by the same speaker. p By including the weighted sum of in L, it is possible to alleviate the problem that, in conventional technologies, there is no guarantee that the same emotional expression vector will result from the same normal emotional utterance from the same speaker (= the recognition result often changes if the normal emotional utterance changes). In other words, it is possible to eliminate the phenomenon in emotion recognition using normal emotional utterances where an estimated result is output that is specialized for the combination of the input utterance and the normal emotional utterance. This increases the likelihood of obtaining the same recognition result even for different normal emotional utterances, contributing to improving the accuracy of recognizing a speaker's emotions from their utterances.

[0052] In this embodiment, the loss function L a is an example of the first loss function. p is an example of the second loss function.

[0053] Although the embodiments of the present invention have been described in detail above, the present invention is not limited to such specific embodiments, and various modifications and variations are possible within the scope of the gist of the present invention as described in the claims. [Explanation of symbols]

[0054] 10 Emotion recognition device 11-1 Acoustic feature extraction unit 11-2 Acoustic feature extraction unit 12 Learning Department 13 Emotion recognition section 100 Drive device 101 Recording media 102 Auxiliary storage device 103 Memory Device 104 processors 105 Interface Device 121 Learning data storage unit B Bus m1 emotion recognition model m11 Emotion expression vector extraction block m11-1 Emotion expression vector extraction block m11-2 Emotion expression vector extraction block m12 Emotion probability estimation block

Claims

1. a learning procedure for learning the model based on learning data including, for a plurality of speakers, a plurality of input utterances by speakers, a plurality of normal emotion utterances corresponding to each of the input utterances, and correct labels of emotions corresponding to each of the input utterances, using a first loss function for minimizing an error in an emotion recognition result output by a model to which any of the input utterances and the normal emotion utterance corresponding to the input utterance have been input, relative to the correct label corresponding to the input utterance, and a second loss function for keeping constant, for each speaker, a vector representing the nature of emotional expression calculated for the normal emotion utterance in the process of the model outputting the recognition result; The emotion recognition learning method is characterized in that the above is executed by a computer.

2. the second loss function is based on a distance between an average value of the vectors calculated by the model for each of the normal emotion utterances corresponding to the same speaker and each of the vectors; 2. The emotion recognition learning method according to claim 1.

3. the learning procedure includes learning the model based on a weighted sum of the first loss function and the second loss function.

3. The emotion recognition learning method according to claim 1 or 2.

4. an emotion recognition procedure for recognizing the emotion of a speaker in an input utterance and a normal emotional utterance using a model trained by the emotion recognition training method according to claim 1; The emotion recognition method is characterized in that the above is executed by a computer.

5. a learning unit configured to learn the model based on learning data including, for a plurality of speakers, a plurality of input utterances by speakers, a plurality of normal emotion utterances corresponding to each of the input utterances, and correct labels of emotions corresponding to each of the input utterances, using a first loss function for minimizing an error in an emotion recognition result output by a model to which any of the input utterances and the normal emotion utterance corresponding to the input utterance have been input, relative to the correct label corresponding to the input utterance, and a second loss function for making constant, for each speaker, a vector representing the nature of emotional expression calculated for the normal emotion utterance in the process of the model outputting the recognition result; An emotion recognition learning device comprising:

6. an emotion recognition unit configured to recognize the emotion of a speaker related to an input utterance and a normal emotional utterance using a model trained by the emotion recognition training method according to claim 1; An emotion recognition device comprising:

7. 2. A program for causing a computer to execute the emotion recognition learning method according to claim 1.

8. A program for causing a computer to execute the emotion recognition method according to claim 4.

Images