Training device, emotion recognition device, training method, emotion recognition method, and program

Abstract

A training device according to one aspect of the present disclosure comprises: an acoustic feature sequence extraction unit that uses a given voice and a first parameter to extract an acoustic feature sequence representing features of the voice; a speaker attention unit that uses information relating to a speaker extracted from the voice, the acoustic feature sequence, and a second parameter to extract information indicating a clue to an emotional expression for each speaker from the acoustic feature sequence on the basis of the information relating to the speaker; a posterior probability calculation unit that uses the information indicating the clue to the emotional expression for each speaker and a third parameter to calculate posterior probability for each predetermined emotion class; and a training unit that uses the posterior probability for each emotion class and the correct answer of the emotion class to which the voice belongs to update the first parameter, the second parameter, and the third parameter.

Description

Learning device, emotion recognition device, learning method, emotion recognition method, and program

[0001] The present disclosure relates to a learning device, an emotion recognition device, a learning method, an emotion recognition method, and a program.

[0002] Emotion recognition technology that estimates a speaker's emotion from their voice (utterance) using an emotion recognition model constructed using machine learning technology or the like has been known for some time. One of the challenges with emotion recognition technology is that the way emotions are expressed can vary from speaker to speaker. For example, some people express anger by raising their voice, while others express anger in a subdued manner. In response to this challenge, emotion recognition technology that can adapt to the emotional expressions that can vary from speaker to speaker has been proposed (e.g., Non-Patent Documents 1 and 2, etc.).

[0003] Y. Li, T. Zhao, and T. Kawahara, "Improved end-to-end speech emotion recognition using self attention mechanism and multitask learning," in Proc. of Interspeech, 2019, pp. 2803-2807.CL Moine, N. Obin, and A. Roebel, "Speaker attentive speech emotion recognition," in Proc. of Interspeech, 2021, pp. 2866-2870.

[0004] However, conventional emotion recognition technology, which can adapt to emotional expressions that can differ from speaker to speaker, estimates not only emotions but also speaker information, so that when training an emotion recognition model, not only emotion labels, which are correct values ​​for emotions, but also speaker information labels, which are correct values ​​for speaker information, are required. This means that in addition to the labeling work for emotion labels, the labeling work for speaker information labels also becomes necessary, resulting in significant labeling costs.

[0005] The present disclosure has been made in consideration of the above points, and provides a technology that can reduce the cost required for training an emotion recognition model that can adapt to emotional expressions that may differ from speaker to speaker.

[0006] A learning device according to one aspect of the present disclosure includes: an acoustic feature sequence extraction unit that uses given speech and a first parameter to extract a sequence of acoustic features that represent characteristics of the speech; a speaker attention unit that uses speaker information extracted from the speech, the sequence of acoustic features, and a second parameter to extract information that represents clues to emotional expression for each speaker from the sequence of acoustic features based on information about the speaker; a posterior probability calculation unit that uses the information that represents clues to emotional expression for each speaker and a third parameter to calculate a posterior probability for each predetermined emotion class; and a learning unit that updates the first parameter, the second parameter, and the third parameter using the posterior probability for each emotion class and a correct answer for the emotion class to which the speech belongs.

[0007] A technology is provided that can reduce the cost required to train an emotion recognition model that can adapt to emotional expressions that may differ from speaker to speaker.

[0008] FIG. 1 is a diagram showing an example of an emotion recognition model in the prior art. FIG. 2 is a diagram showing an example of an unsupervised speaker adaptation emotion recognition model. FIG. 3 is a diagram showing an example of a multi-head attention mechanism. FIG. 4 is a diagram showing an example of a speaker expression vector for each speaker. FIG. 5 is a diagram showing an example of a speaker expression vector for each emotion. FIG. 6 is a diagram showing an example of the hardware configuration of the emotion recognition device according to the present embodiment. FIG. 7 is a diagram showing an example of the functional configuration of the emotion recognition device during learning. A flowchart showing an example of the learning process according to the present embodiment. A diagram showing an example of the functional configuration of the emotion recognition device during inference. A flowchart showing an example of the emotion recognition process according to the present embodiment.

[0009] An embodiment of the present invention will be described below.

[0010] <Conventional emotion recognition technology adaptable to emotional expressions that may differ from speaker to speaker> Emotion recognition technology uses an emotion recognition model constructed using machine learning technology or the like to input speech (utterance) and estimates to which emotion class (e.g., neutral, anger, joy, sadness, etc.) the speaker's emotion of the utterance belongs. One of the challenges with emotion recognition technology is that the way emotions are expressed may differ from speaker to speaker. In response to this challenge, Non-Patent Documents 1 and 2 propose emotion recognition technology adaptable to emotional expressions that may differ from speaker to speaker, i.e., emotion recognition technology using speaker adaptation (hereinafter referred to as "conventional emotion recognition technology" or simply "conventional technology").

[0011] Conventional emotion recognition technology uses a statistical discrimination model configured with a neural network or the like as an emotion recognition model, and during inference, estimates not only emotion but also speaker information from an input utterance (hereinafter referred to as "input utterance"). Speaker information refers to information about the speaker of the input utterance, and may include, for example, the gender of the speaker or a speaker ID (e.g., a unique number identifying the speaker).

[0012] An example of an emotion recognition model used in conventional emotion recognition technology (hereinafter referred to as a "conventional emotion recognition model") is schematically shown in Fig. 1. As shown in Fig. 1, the conventional emotion recognition model inputs an input utterance to an acoustic feature extraction encoder to obtain an acoustic feature vector sequence H, then inputs this acoustic feature vector sequence H to an average pooling layer, and finally inputs the output of the average pooling layer to a decoder to estimate emotions and speaker information, and outputs the results (emotion recognition result and speaker information estimation result).

[0013] As shown in Figure 1, conventional emotion recognition models estimate speaker information in addition to emotions, so when training a conventional emotion recognition model, utterances to which emotion labels and speaker information labels are attached are provided as training data, and parameters are updated so that the training data can be accurately estimated. In this way, conventional emotion recognition technology achieves emotion recognition using speaker adaptation, which estimates the speaker from the input utterance and outputs emotion recognition results appropriate for that speaker.

[0014] Therefore, conventional emotion recognition technology requires labeling of speaker information labels in addition to labeling of emotion labels, which requires a large labeling cost.

[0015] Therefore, we propose an emotion recognition model that can adapt to emotional expressions that may differ from speaker to speaker without using speaker information labels. Hereinafter, this emotion recognition model will be referred to as the "unsupervised speaker-adaptive emotion recognition model." Note that "unsupervised" here means that speaker information supervision data (i.e., speaker information labels) is not required.

[0016] <Unsupervised Speaker Adaptive Emotion Recognition Model> Fig. 2 shows a schematic representation of an example of the unsupervised speaker adaptive emotion recognition model proposed in this embodiment. As shown in Fig. 2, the unsupervised speaker adaptive emotion recognition model 1000 includes an acoustic feature extraction encoder 1100, a speaker expression vector extraction encoder 1200, a speaker attention pooling layer 1300, and a decoder 1400. Here, the speaker expression vector extraction encoder 1200 has been trained in advance using a speaker classification dataset, and parameters of the speaker expression vector extraction encoder 1200 are not updated when training the unsupervised speaker adaptive emotion recognition model 1000. On the other hand, parameters of the acoustic feature extraction encoder 1100, the speaker attention pooling layer 1300, and the decoder 1400 are subject to update. Note that the speaker classification dataset is a data set composed of speaker classification data in which utterances are associated with information indicating the classification of the speaker of the utterance.

[0017] An input utterance input to the unsupervised speaker adaptation emotion recognition model 1000 is input to an acoustic feature extraction encoder 1100 and a speaker expression vector extraction encoder 1200. The acoustic feature extraction encoder 1100 extracts an acoustic feature vector representing the acoustic features of the input utterance for each predetermined short time interval (also called a "frame"), and outputs a sequence of these acoustic feature vectors as an acoustic feature vector sequence H. Meanwhile, the speaker expression vector extraction encoder 1200 extracts a fixed-dimensional speaker expression vector q from the input utterance. A speaker expression vector is a vector that represents speaker information contained in a certain speech. Speaker expression vectors are widely used for reproducing speaker information in speech synthesis (see, for example, Reference 1).

[0018] The acoustic feature extraction encoder 1100 can be realized by any model, function, process, means, technique, or mechanism (hereinafter collectively referred to as "mechanism, etc.") capable of extracting a sequence of acoustic features (acoustic feature sequence) from an input utterance (speech). For example, the acoustic feature extraction encoder 1100 can be realized by HuBERT (Reference 2), an acoustic feature extraction encoder based on self-supervised learning. In this case, HuBERT is assumed to have been pre-trained using a large amount of speech data. However, HuBERT is only one example, and the acoustic feature extraction encoder 1100 may be realized by another mechanism, etc., capable of extracting an acoustic feature sequence from an input utterance. For example, the acoustic feature extraction encoder 1100 may be realized by an acoustic feature extraction encoder based on a neural network other than HuBERT, or by processing that divides the input utterance into short-time windows and obtains heuristic acoustic features such as a power spectrum or a fundamental frequency for each short-time window.

[0019] The speaker expression vector extraction encoder 1200 can be realized by any mechanism capable of extracting a fixed-dimensional speaker expression vector from an input utterance. For example, the speaker expression vector extraction encoder 1200 can be realized by ECAPA-TDNN (Reference 3), which is a speaker expression vector extraction encoder based on a neural network. However, ECAPA-TDNN is just one example, and the speaker expression vector extraction encoder 1200 may be realized by other mechanisms capable of extracting a fixed-dimensional speaker expression vector from an input utterance. For example, it may be realized by a non-neural network-based method such as i-vector.

[0020] The speaker attention pooling layer 1300 receives the acoustic feature vector sequence H and the speaker expression vector q as input, and extracts information representing clues to emotional expression for each speaker from the acoustic feature vector sequence H based on the speaker expression vector q. More specifically, the speaker attention pooling layer 1300 estimates weights indicating which dimensions of which acoustic feature vectors constituting the acoustic feature vector sequence H should be focused on based on the speaker expression vector q, and calculates a weighted sum of each acoustic feature vector using these weights as information representing clues to emotional expression for each speaker. In other words, the speaker attention pooling layer 1300 estimates weights indicating which dimensions of which acoustic feature vectors of which frames should be focused on based on speaker information represented by the speaker expression vector q for the sequence of acoustic feature vectors for each frame, and calculates a weighted sum of each acoustic feature vector using these weights as information representing clues to emotional expression for each speaker.

[0021] The speaker attention pooling layer 1300 can be realized by any mechanism capable of extracting weights indicating which dimension of which acoustic feature vector in the acoustic feature vector sequence H should be focused on from speaker information and calculating the weighted sum of the acoustic feature vectors. For example, the speaker attention pooling layer 1300 can be realized by an attention mechanism or the like. As an example, the following describes a case where the speaker attention pooling layer 1300 is realized by a multi-head attention mechanism. For details of the multi-head attention mechanism, see, for example, Reference 4.

[0022] When the speaker attention pooling layer 1300 is realized by a multi-head attention mechanism, the speaker attention pooling layer 1300 uses a multi-head attention mechanism with the speaker vector q as the query and the acoustic feature vector sequence H as the key and value. In this case, the configuration of the multi-head attention mechanism is shown in FIG. 3. As shown in FIG. 3, in the multi-head attention mechanism, the key, value, and query are linearly transformed in a fully connected layer (FC), and then a scaled dot-product attention mechanism is calculated in parallel. Then, vectors representing these calculation results are concatenated (concat), and finally, the matrix obtained by this concatenation is linearly transformed by the fully connected layer (FC). The scaled dot-product attention mechanism takes the query, key, and value as input, and applies a softmax function to the dot product of the query and key, dividing the product by a certain dimension, and multiplying the result by the value. For details of the reduced dot product attention mechanism, see, for example, Reference 4.

[0023] The decoder 1400 receives as input the output of the speaker attention pooling layer 1300 (i.e., information representing clues to emotional expression for each speaker) and calculates a posterior probability vector whose elements are the posterior probabilities for each predetermined emotion class (i.e., the posterior probability that the emotion of the speaker of the utterance belongs to each emotion class). As a result, for example, the emotion represented by the emotion class corresponding to the element with the largest value among the elements of the posterior probability vector (or an emotion class selected according to the posterior probability) is estimated as the emotion of the speaker. Note that the decoder 1400 may calculate the posterior probability for each emotion class by, for example, applying a softmax function to the weighted sum of acoustic feature vectors, which are information representing clues to emotional expression for each speaker.

[0024] As described above, the unsupervised speaker adaptation emotion recognition model 1000 proposed in this embodiment includes the speaker expression vector extraction encoder 1200 and the speaker attention pooling layer 1300. With this configuration, the speaker expression vector extraction encoder 1200 makes it possible to accurately estimate speaker information without using speaker information labels, and the speaker attention pooling layer 1300 realizes adaptation to emotional expressions that may differ for each speaker by utilizing speaker information.

[0025] The speaker expression vector extraction encoder 1200 must be able to extract a similar speaker expression vector for any utterance by a certain speaker, regardless of differences in emotional expression. The inventors of the present application have confirmed this through experiments. In the experiments, a set of utterances consisting of utterances from a total of 10 speakers, five male and five female, was used to extract 192-dimensional speaker expression vectors using the speaker expression vector extraction encoder 1200, and these speaker expression vectors were then projected onto the t-SNE (dimensionality reduction). Figure 4 shows the speaker expression vectors plotted on a two-dimensional plane for each speaker, and Figure 5 shows the speaker expression vectors plotted on a two-dimensional plane for each emotion estimated by the unsupervised speaker adaptation emotion recognition model 1000. Note that Ses01_F to Ses05 in Figure 4 represent female speakers, Ses01_M to Ses05_M represent male speakers, and neutral, happy, sad, and angry in Figure 5 represent emotion classes, respectively.

[0026] As shown in Figure 4, it can be seen that the speaker expression vectors are distributed in clusters for each speaker. Furthermore, as shown in Figures 4 and 5, it can be seen that various emotions are estimated from the utterances of each speaker. For this reason, it can be said that similar speaker expression vectors are extracted from the same speaker regardless of the utterance. Therefore, the speaker expression vector extraction encoder 1200 can stably estimate speaker expression vectors from input utterances.

[0027] Below, we will explain an emotion recognition device 10 that is capable of training an unsupervised speaker adaptation emotion recognition model 1000 and performing inference (i.e., emotion recognition) using the trained unsupervised speaker adaptation emotion recognition model 1000. Note that although below we will explain a case where training and inference are performed by the same emotion recognition device 10, learning and inference may also be performed by different devices. In this case, the device that performs training may be called a "learning device" or the like.

[0028] <Example of Hardware Configuration of Emotion Recognition Device 10> An example of the hardware configuration of the emotion recognition device 10 according to this embodiment is shown in Fig. 6. As shown in Fig. 6, the emotion recognition device 10 according to this embodiment includes an input device 101, a display device 102, an external I / F 103, a communication I / F 104, a RAM (Random Access Memory) 105, a ROM (Read Only Memory) 106, an auxiliary storage device 107, and a processor 108. Each of these pieces of hardware is connected to each other via a bus 109 so as to be able to communicate with each other.

[0029] The input device 101 is, for example, a keyboard, a mouse, a touch panel, a microphone, a physical button, etc. The display device 102 is, for example, a display, a display panel, etc. Note that the emotion recognition device 10 does not necessarily have to include at least one of the input device 101 and the display device 102, for example.

[0030] The external I / F 103 is an interface with an external device such as a recording medium 103a. Examples of the recording medium 103a include a CD (Compact Disc), a DVD (Digital Versatile Disk), an SD memory card (Secure Digital memory card), and a USB (Universal Serial Bus) memory card.

[0031] The communication I / F 104 is an interface for connecting the emotion recognition device 10 to a communication network. The RAM 105 is a volatile semiconductor memory (storage device) that temporarily stores programs and data. The ROM 106 is a non-volatile semiconductor memory (storage device) that can store programs and data even when the power is turned off. The auxiliary storage device 107 is a non-volatile storage device such as an HDD (Hard Disk Drive), an SSD (Solid State Drive), or a flash memory. The processor 108 is a computing device such as a CPU (Central Processing Unit) or a GPU (Graphic Processing Unit).

[0032] 6 is an example, and the hardware configuration of the emotion recognition device 10 is not limited to this. For example, the emotion recognition device 10 may have multiple auxiliary storage devices 107 or multiple processors 108, may not have some of the hardware shown in the figure, or may have various types of hardware other than the hardware shown in the figure.

[0033] During Training Below, we will explain the training of the unsupervised speaker adaptation emotion recognition model 1000. Here, during training, the emotion recognition device 10 is provided with a set of training data (hereinafter referred to as a "training dataset") represented by pairs of input utterances and emotion labels for these input utterances (i.e., the correct values ​​of the emotion classes to which the input utterances belong). Note that the emotion labels are represented, for example, by vectors in which only the elements corresponding to the emotion class to which the input utterances belong are 1 and the other elements are 0.

[0034] <Example of functional configuration of the emotion recognition device 10 during learning> Fig. 7 shows an example of the functional configuration of the emotion recognition device 10 during learning. As shown in Fig. 7, the emotion recognition device 10 during learning has an input unit 201, an emotion recognition unit 202, and a model learning unit 203. These units are realized, for example, by processing in which one or more programs installed in the emotion recognition device 10 are executed by the processor 108 or the like. The emotion recognition device 10 during learning also has a model parameter storage unit 210. The model parameter storage unit 210 is realized, for example, by a storage area of ​​the auxiliary storage device 107 or the like. However, the model parameter storage unit 210 may also be realized, for example, by a storage area of ​​a storage device or the like connected to the emotion recognition device 10 via a communications network.

[0035] The input unit 201 inputs training data that constitutes a training data set provided to the emotion recognition device 10 .

[0036] The emotion recognition unit 202 is realized by the unsupervised speaker adaptation emotion recognition model 1000, and takes as input an input utterance included in the training data input by the input unit 201, and calculates a posterior probability vector using parameters (first parameter to fourth parameter) stored in the model parameter storage unit 210. Here, the emotion recognition unit 202 includes an acoustic feature sequence extraction unit 301, a speaker expression extraction unit 302, a speaker attention unit 303, and a decoding unit 304. The acoustic feature sequence extraction unit 301 is realized by the acoustic feature extraction encoder 1100, and extracts an acoustic feature vector H from the input utterance using a first parameter that is a parameter of the acoustic feature extraction encoder 1100. The speaker expression extraction unit 302 is realized by the speaker expression vector extraction encoder 1200, and extracts a speaker expression vector q from the input utterance using a fourth parameter that is a parameter of the speaker expression vector extraction encoder 1200. The speaker attention unit 303 is realized by the speaker attention pooling layer 1300, and uses a second parameter that is a parameter of the speaker attention pooling layer 1300 to calculate a weighted sum of acoustic feature vectors as information representing clues to emotional expression for each speaker from the acoustic feature vector H based on the speaker expression vector q. The decoding unit 304 is realized by the decoder 1400, and uses a third parameter that is a parameter of the decoder 1400 to calculate a posterior probability vector whose elements are posterior probabilities for each emotional class from the weighted sum of acoustic feature vectors that are information representing clues to emotional expression for each speaker.

[0037] The model learning unit 203 uses the posterior probability vector calculated by the emotion recognition unit 202 and the emotion labels included in the training data input by the input unit 201 to update parameters to be updated (hereinafter referred to as "update target parameters") among the parameters stored in the model parameter storage unit 210 through supervised learning. The model learning unit 203 uses cross-entropy loss as a loss function, for example, and updates the update target parameters using backpropagation so as to minimize the cross-entropy loss between the posterior probability vector calculated by the emotion recognition unit 202 and the emotion labels included in the training data input by the input unit 201. More specifically, the model learning unit 203 uses backpropagation to find the gradient of the loss function (cross-entropy loss) for the update target parameters, and uses this gradient to update the update target parameters so as to minimize the cross-entropy loss between the posterior probability vector and the emotion label. Here, the parameters to be updated are the first to third parameters, which are parameters of the acoustic feature extraction encoder 1100, the speaker attention pooling layer 1300, and the decoder 1400 included in the unsupervised speaker adaptation emotion recognition model 1000. Note that the fourth parameter, which is a parameter of the speaker expression vector extraction encoder 1200, is not to be updated.

[0038] The model parameter storage unit 210 stores the parameters (first parameter to fourth parameter) of the unsupervised speaker adaptation emotion recognition model 1000. Of these parameters, the first parameter to third parameter, which are the parameters to be updated, are assumed to have been initialized to appropriate initial values ​​using a known method or the like before the model training unit 203 updates the parameters. On the other hand, the parameter (fourth parameter) of the speaker expression vector extraction encoder 1200 has been trained in advance.

[0039] <Learning Process> The learning process according to this embodiment will be described below with reference to Fig. 8. Note that, as an example, the following describes a case where the unsupervised speaker adaptation emotion recognition model 1000 is trained by online learning based on stochastic gradient descent. However, this is just an example, and a method other than stochastic gradient descent may be used, or a method other than online learning (for example, batch learning, mini-batch learning, etc.) may be used.

[0040] The input unit 201 randomly inputs one piece of training data that has not been input for the current epoch number from among the training data included in the given training data set (step S101). Note that an epoch indicates how many times the training data set is repeatedly trained, and the epoch number is incremented with an initial value of 1 each time all training data included in the training data set is input.

[0041] The emotion recognition unit 202 uses the input utterance included in the training data input in step S101 above as input and calculates a posterior probability vector using the parameters (first parameter to fourth parameter) stored in the model parameter storage unit 210 (step S102). That is, in the emotion recognition unit 202, the acoustic feature sequence extraction unit 301 extracts an acoustic feature vector H from the input utterance, and the speaker expression extraction unit 302 extracts a speaker expression vector q from the input utterance. Thereafter, the speaker attention unit 303 calculates a weighted sum of the acoustic feature vectors, and then the decoding unit 304 calculates the posterior probability vector. As a result, the posterior probability vector is estimated by the unsupervised speaker adapted emotion recognition model 1000 using the parameters.

[0042] The model learning unit 203 updates the parameters to be updated in the unsupervised speaker-adapted emotion recognition model 1000 by a supervised learning method, using the posterior probability for each emotion class calculated in step S102 above and the emotion labels included in the training data input in step S101 above (step S103).

[0043] The model learning unit 203 determines whether to terminate the model learning process (step S104). For example, the model learning unit 203 may determine to terminate the model learning process if a predetermined condition is met, and may determine not to terminate the model learning process if a predetermined condition is not met. Here, the termination condition may be, for example, that the update target parameters have converged, that the number of epochs has reached a predetermined number, or that the number of repetitions of steps S101 to S103 (also called the number of iterations) has reached a predetermined number.

[0044] If it is determined in step S104 that the model learning process should be terminated, the model learning unit 203 terminates the model learning process. On the other hand, if it is determined in step S104 that the model learning process should not be terminated, the model learning unit 203 transitions the process to step S101. As a result, step S101 is executed by the input unit 201.

[0045] As described above, the parameters to be updated for the unsupervised speaker-adapted emotion recognition model 1000 are trained. At this time, unlike conventional techniques, the emotion recognition device 10 according to this embodiment does not need to use speaker information labels. Therefore, compared to conventional techniques, the cost required for training the unsupervised speaker-adapted emotion recognition model 1000 (the cost of labeling training data) can be reduced. In addition, for example, the difficulty of assigning speaker information labels to input utterances collected by multiple different means or input utterances collected from the Web, etc. can be avoided, and these input utterances and their emotion labels can be used as training data for the unsupervised speaker-adapted emotion recognition model 1000, which is expected to further improve emotion recognition accuracy. Note that hereinafter, the unsupervised speaker-adapted emotion recognition model 1000 that uses the trained parameters to be updated will be referred to as the "trained unsupervised speaker-adapted emotion recognition model 1000."

[0046] During Inference The following describes the process of inference using the trained unsupervised speaker adapted emotion recognition model 1000. Here, during inference, an input utterance to be subjected to emotion recognition is provided to the emotion recognition device 10.

[0047] <Example of functional configuration of the emotion recognition device 10 during inference> Fig. 9 shows an example of the functional configuration of the emotion recognition device 10 during inference. As shown in Fig. 9, the emotion recognition device 10 during inference has an input unit 201, an emotion recognition unit 202, and an output unit 204. These units are realized, for example, by processing in which one or more programs installed in the emotion recognition device 10 are executed by the processor 108 or the like. The emotion recognition device 10 during inference also has a model parameter storage unit 210. The model parameter storage unit 210 is realized, for example, by a storage area of ​​the auxiliary storage device 107 or the like. However, the model parameter storage unit 210 may also be realized, for example, by a storage area of ​​a storage device or the like connected to the emotion recognition device 10 via a communications network.

[0048] The input unit 201 inputs an input utterance given to the emotion recognition device 10 .

[0049] The emotion recognition unit 202 is realized by the trained unsupervised speaker adaptation emotion recognition model 1000, and takes an input utterance input by the input unit 201 as input, and estimates the emotion of the speaker of the input utterance (i.e., the emotion class to which the input utterance belongs) using trained parameters stored in the model parameter storage unit 210. Here, the emotion recognition unit 202 includes an acoustic feature sequence extraction unit 301, a speaker expression extraction unit 302, a speaker attention unit 303, and a decoding unit 304. The acoustic feature sequence extraction unit 301 is realized by the acoustic feature extraction encoder 1100, and extracts an acoustic feature vector H from the input utterance using first parameters that are trained parameters of the acoustic feature extraction encoder 1100. The speaker expression extraction unit 302 is realized by the speaker expression vector extraction encoder 1200, and extracts a speaker expression vector q from the input utterance using fourth parameters that are parameters of the speaker expression vector extraction encoder 1200. The speaker attention unit 303 is realized by the speaker attention pooling layer 1300, and uses a second parameter that is a trained parameter of the speaker attention pooling layer 1300 to calculate a weighted sum of acoustic feature vectors from the acoustic feature vector H as information representing clues to emotional expression for each speaker, based on the speaker expression vector q. The decoding unit 304 is realized by the decoder 1400, and uses a third parameter that is a trained parameter of the decoder 1400 to calculate a posterior probability vector, the elements of which are posterior probabilities for each emotion class, from the weighted sum of acoustic feature vectors that are information representing clues to emotional expression for each speaker, and estimates, as the speaker's emotion, the emotion represented by the emotion class corresponding to the element with the largest value.

[0050] The output unit 204 outputs the emotion estimated by the emotion recognition unit 202 as an emotion recognition result to a predetermined output destination. Note that the output destination may be any predetermined output destination, and examples thereof include the display device 102 such as a display, and a terminal, device, or apparatus connected to the emotion recognition device 10 via a communication network.

[0051] The model parameter storage unit 210 stores the parameters of the trained unsupervised speaker-adaptive emotion recognition model 1000. Note that the parameters (first to third parameters) of the acoustic feature extraction encoder 1100, speaker attention pooling layer 1300, and decoder 1400 are updated in the model training process, and the parameter (fourth parameter) of the speaker expression vector extraction encoder 1200 is trained in advance.

[0052] <Emotion Recognition Processing> The emotion recognition processing according to this embodiment will be described below with reference to FIG.

[0053] The input unit 201 receives a given input utterance (step S201).

[0054] The emotion recognition unit 202 uses the input utterance input in step S201 above as input and the parameters (first to fourth parameters) stored in the model parameter storage unit 210 to estimate the emotion class to which the input utterance belongs (step S202). That is, the emotion recognition unit 202 extracts an acoustic feature vector H from the input utterance using the acoustic feature sequence extraction unit 301 and a speaker expression vector q from the input utterance using the speaker expression extraction unit 302. Then, the speaker attention unit 303 calculates a weighted sum of the acoustic feature vectors, and then the decoding unit 304 calculates a posterior probability vector and estimates the emotion class corresponding to the element with the largest value among the elements of the posterior probability vector. As a result, the emotion represented by the emotion class is estimated as the speaker's emotion, thereby achieving emotion recognition.

[0055] The output unit 204 outputs the emotion class estimated in step S202 as an emotion recognition result to a predetermined output destination (step S203).

[0056] As described above, emotion recognition is realized by the trained unsupervised speaker-adapted emotion recognition model 1000.

[0057] <Summary> As described above, the emotion recognition device 10 according to this embodiment trains an unsupervised speaker adaptation emotion recognition model 1000 that includes a pre-trained speaker expression vector extraction encoder 1200 that extracts speaker expression vectors from input utterances, and a speaker attention pooling layer 1300 that extracts information representing clues to emotional expressions for each speaker from the acoustic feature sequence of the input utterance and the speaker expression vectors. Due to the configuration of this unsupervised speaker adaptation emotion recognition model 1000, the emotion recognition device 10 according to this embodiment can train the unsupervised speaker adaptation emotion recognition model 1000 without using speaker information labels, unlike conventional techniques. Therefore, the cost required for training the unsupervised speaker adaptation emotion recognition model 1000 (the cost of labeling training data) can be reduced compared to conventional techniques. In addition to this, for example, assigning speaker information labels to input utterances collected by a plurality of different means or from the Web or the like can avoid difficulties, and these input utterances and their emotion labels can be used as training data for the unsupervised speaker adaptation emotion recognition model 1000, which is expected to further improve emotion recognition accuracy.

[0058] The emotion recognition device 10 according to this embodiment can realize emotion recognition that can adapt to emotional expressions that may differ from speaker to speaker by using the trained unsupervised speaker adaptation emotion recognition model 1000. Therefore, by applying the emotion recognition device 10 according to this embodiment to various fields, various effects can be expected in the fields. For example, by recognizing the speaker's emotion during counseling, it becomes possible to visualize the patient's feelings of anxiety or sadness, which is expected to deepen the counselor's understanding and improve the quality of guidance. Furthermore, for example, by recognizing human emotions in human-machine dialogue, it becomes possible to build a more friendly dialogue system that can share in the human's joy when the human is happy, or encourage the human when the human is sad, for example.

[0059] The emotion recognition device 10 according to the above embodiment provides a specific improvement over the problem in the prior art that the cost of labeling speaker information labels is required to train an emotion recognition model that can adapt to emotional expressions that may differ from speaker to speaker, and represents a reduction in learning costs and an improvement in inference accuracy in the technical field related to emotion recognition.

[0060] The following supplementary note is further disclosed regarding the above embodiment: (Supplementary Note 1) A learning device including: a memory; and at least one processor connected to the memory, wherein the processor: extracts a sequence of acoustic features representing characteristics of the speech using given speech and a first parameter; extracts information representing clues to emotional expression for each speaker from the sequence of acoustic features based on information about the speaker using information about the speaker extracted from the speech, the sequence of acoustic features, and a second parameter; calculates a posterior probability for each predetermined emotion class using the information representing clues to emotional expression for each speaker and a third parameter; and updates the first parameter, the second parameter, and the third parameter using the posterior probability for each emotion class and a correct answer for the emotion class to which the speech belongs. (Supplementary Item 2) The learning device according to Supplementary Item 1, wherein the processor estimates weights indicating which acoustics in the series of acoustic features to focus on and which dimensions of the vectors representing the acoustic features to focus on based on information about the speakers, and extracts a weighted sum of the vectors representing the acoustic features using the weights as information indicating clues to emotional expression for each speaker. (Supplementary Item 3) The learning device according to Supplementary Item 1 or 2, wherein the processor extracts information about the speaker from the speech using the speech and a fourth parameter that has been trained in advance. (Supplementary Item 4) An emotion recognition device comprising: a memory; and at least one processor connected to the memory, wherein the processor: uses given speech and pre-trained first parameters to extract a sequence of acoustic features representing characteristics of the speech, uses speaker information extracted from the speech, the sequence of acoustic features, and pre-trained second parameters to extract information representing clues to emotional expression for each speaker from the sequence of acoustic features based on information about the speaker, and uses the information representing clues to emotional expression for each speaker and pre-trained third parameters to estimate the emotion of the speaker in accordance with a posterior probability for each predetermined emotion class. (Supplementary Item 5) A non-transitory storage medium storing a program that causes a computer to function as the learning device according to Supplementary Item 1 or the emotion recognition device according to Supplementary Item 4.

[0061] The present invention is not limited to the above-described specifically disclosed embodiments, and various modifications, changes, and combinations with known technologies are possible without departing from the scope of the claims.

[0062] 1:Ye Jia, Yu Zhang, Ron J. Weiss, Quan Wang, Jonathan Shen, Fei Ren, Zhifeng Chen, Patrick Nguyen, Ruoming Pang, Ignacio Lopez Moreno, Yonghui Wu, "Transfer Learning from Speaker." Verification to Multispeaker Text-To-Speech Synthesis," Advances in NIPS, Hsu, B. Bolte, Y.-HH Tsai, K. Lakhotia, R. Salakhut-dinov, and A. Mohamed, "HuBERT: Self-supervised speech representation learning by masked prediction of hidden units," IEEE / ACM Trans. on Audio, Speech, and Language Processing, vol. 29, pp. 101-1 3451-3460, 2021. Equipment 3:B. http: / / dx.doi.org / 10.1037 / 0033-295X.101.1.101 Desplanques, J. Thienpondt, and K. Demuynck, "ECAPA-TDNN: Emphasized Channel Attention, Propagation, and Aggregation in TDNN Based Speaker Verification," in Proc. of Interspeech, 2020, pp. 107-1 3830-3 References 4:Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin, "Attention is All you Need," Advances in NIPS, vol.

[0063] 10 Emotion recognition device 101 Input device 102 Display device 103 External I / F 103a Recording medium 104 Communication I / F 105 RAM 106 ROM 107 Auxiliary storage device 108 Processor 109 Bus 201 Input unit 202 Emotion recognition unit 203 Model learning unit 204 Output unit 210 Model parameter storage unit 301 Acoustic feature sequence extraction unit 302 Speaker expression extraction unit 303 Speaker attention unit 304 Decoding unit 1000 Unsupervised speaker adaptive emotion recognition model 1100 Acoustic feature extraction encoder 1200 Speaker expression vector extraction encoder 1300 Speaker attention pooling layer 1400 Decoder

Claims

1. A learning device comprising: an acoustic feature sequence extraction unit that uses given speech and a first parameter to extract a sequence of acoustic features that represent characteristics of the speech; a speaker attention unit that uses speaker information extracted from the speech, the sequence of acoustic features, and a second parameter to extract information that represents clues to emotional expression for each speaker from the sequence of acoustic features based on information about the speaker; a posterior probability calculation unit that uses the information that represents clues to emotional expression for each speaker and a third parameter to calculate a posterior probability for each predetermined emotion class; and a learning unit that updates the first parameter, the second parameter, and the third parameter using the posterior probability for each emotion class and a correct answer for the emotion class to which the speech belongs.

2. The learning device described in claim 1, wherein the speaker attention unit estimates weights indicating which acoustic in the series of acoustic features should be focused on and which dimension of the vector representing the acoustic features should be focused on based on information about the speaker, and extracts a weighted sum of the vector representing the acoustic features using the weights as information indicating clues to emotional expression for each speaker.

3. A learning device according to claim 1 or 2, further comprising a speaker expression extraction unit that uses the speech and a fourth parameter that has been learned in advance to extract information about the speaker from the speech.

4. An emotion recognition device comprising: an acoustic feature sequence extraction unit that uses given speech and pre-trained first parameters to extract a sequence of acoustic features representing characteristics of the speech; a speaker attention unit that uses speaker information extracted from the speech, the sequence of acoustic features, and pre-trained second parameters to extract information representing clues to emotional expression for each speaker from the sequence of acoustic features based on information about the speaker; and an emotion estimation unit that uses the information representing clues to emotional expression for each speaker and pre-trained third parameters to estimate the emotion of the speaker in accordance with a posterior probability for each predetermined emotion class.

5. A learning method in which a computer executes the following steps: an acoustic feature sequence extraction step of extracting a sequence of acoustic features representing characteristics of the speech using given speech and a first parameter; a speaker attention step of extracting information representing clues to emotional expression for each speaker from the sequence of acoustic features based on information about the speaker using speaker information extracted from the speech, the sequence of acoustic features, and a second parameter; a posterior probability calculation step of calculating a posterior probability for each predetermined emotion class using the information representing clues to emotional expression for each speaker and a third parameter; and a learning step of updating the first parameter, the second parameter, and the third parameter using the posterior probability for each emotion class and a correct answer for the emotion class to which the speech belongs.

6. An emotion recognition method executed by a computer, comprising: an acoustic feature sequence extraction step of extracting a sequence of acoustic features representing characteristics of a given speech using a given speech and a first parameter previously trained; a speaker attention step of extracting information representing clues to emotional expression for each speaker from the sequence of acoustic features based on information about the speaker using information about the speaker extracted from the speech, the sequence of acoustic features, and a second parameter previously trained; and an emotion estimation step of estimating the emotion of the speaker in accordance with a posterior probability for each predetermined emotion class using the information representing clues to emotional expression for each speaker and a third parameter previously trained.

7. A program for causing a computer to execute the following steps: an acoustic feature sequence extraction procedure for extracting a sequence of acoustic features representing characteristics of a given speech using a given speech and a first parameter; a speaker attention procedure for extracting information representing clues to emotional expression for each speaker from the sequence of acoustic features based on information about the speaker using speaker information extracted from the speech, the sequence of acoustic features, and a second parameter; a posterior probability calculation procedure for calculating a posterior probability for each predetermined emotion class using the information representing clues to emotional expression for each speaker and a third parameter; and a learning procedure for updating the first parameter, the second parameter, and the third parameter using the posterior probability for each emotion class and a correct answer for the emotion class to which the speech belongs.

8. A program for causing a computer to execute the following steps: an acoustic feature sequence extraction procedure for extracting a sequence of acoustic features representing characteristics of a given speech using a given speech and a first parameter that has been trained in advance; a speaker attention procedure for extracting information representing clues to emotional expression for each speaker from the sequence of acoustic features based on information about the speaker using information about the speaker extracted from the speech, the sequence of acoustic features, and a second parameter that has been trained in advance; and an emotion estimation procedure for estimating the emotion of the speaker in accordance with the posterior probability for each predetermined emotion class using the information representing clues to emotional expression for each speaker and a third parameter that has been trained in advance.