Apparatus and method for training an emotion classifier

Abstract

To provide a training device and a training method for an emotion classifier that can classify a speaker's emotions with high accuracy even from voices by various speakers.SOLUTION: A training device 50 includes: a data selection unit 64 for selecting training data; an encoder 70 for receiving the selected training data; a sampling unit 68 for sampling depending on a label of the training data from a plurality of Gaussian distributions; a classifier 76 for receiving an output of the encoder 70; a discriminator 74 that selectively receives as input the output of the encoder 70 and the samples sampled by the sampling unit 68; and a training control unit 62 for training the encoder 70, the discriminator 74 and the classifier 76 so that the discriminator 74 discriminates between the acquired samples and the training data, and the classifier 76 estimates the label attached to the training data, based on the output of the encoder 70 that receives the selected training data.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to emotion classification technology, and more particularly to an apparatus and method for training an emotion classifier that estimates a speaker's emotion from speech. [Background technology]

[0002] Speech emotion recognition is used to track the emotions of facility users and improve the user experience between humans and agents. Recently, speech emotion recognition has become very advanced, especially due to the development of feature selection and so-called deep learning models. Since speech samples have time series attributes, most previous studies have attempted to understand time series information using GRU (Gated Recurrent Unit) and LSTM (Long Short-Term Memory), or a combination of CNN (Convolutional Neural Network) and LSTM.

[0003] In addition to considering the sequence of features contained in an utterance, new models have recently been proposed that can analyze information about the context of the conversation, further improving the accuracy of emotion recognition. Other techniques have been proposed that use attention to identify the most useful frames to apply labels to, and the idea of ​​multiple instance learning to identify and focus on the most salient parts of an utterance. Since voices vary from person to person and from person to person, a multi-task learning method has been proposed to classify both gender and emotion to improve the accuracy of speech emotion recognition. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Yunqi Cai, Lantian Li, Dong Wang, and Andrew Abel. “Deep normalization for speaker vectors.” arXiv preprint arXiv:2004.04095 (2020). [Non-Patent Document 2] Jiaxing Liu, Zhilei Liu, Longbiao Wang, Lili Guo, and Jianwu Dang. “Speech Emotion Recognition with Local-Global Aware Deep Representation Learning.” In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 7174-7178. IEEE, 2020. Summary of the Invention [Problem to be solved by the invention]

[0005] Despite these advances, however, little attention has been paid to regularizing speech. Because different people express emotions differently in their speech, such biases can lead to samples that belong to the same category not clustering together or to forming clusters with irregular boundaries (following non-Gaussian distributions). This problem is especially pronounced in higher-dimensional spaces, making models prone to overfitting on datasets within the same domain, oversensitive to local irregularities in the pattern distribution, and lower scores in multi-domain implementations.

[0006] To solve these problems, the aforementioned non-patent document 1 proposes a method called deep normalization using a new model called a discriminative normalization flow model, although it is in the field of speaker identification rather than emotion recognition.

[0007] According to the method of Non-Patent Document 1, it is possible to regularize the distribution space, but it is difficult to apply it directly to the field of speaker identification and emotion classification. Even if the method of Non-Patent Document 1 is applied to the field of emotion classification, further improvement is required.

[0008] SUMMARY OF THE PRESENT EMBODIMENTS It is therefore a primary object of the present invention to provide an apparatus and method for training an emotion classifier that is capable of classifying the emotions of speakers with high accuracy even from speech produced by various speakers. [Means for solving the problem]

[0009] A training device according to a first aspect of the present invention is a training device for training an emotion classifier consisting of a neural network that classifies the emotion of a speaker represented by voice data, the training device being used in connection with a training data storage device that stores a plurality of training data items, each of which includes voice data and a label indicating the emotion of the speaker represented by the voice data, the label being selected from a plurality of types of labels, the training device comprising: training data selection means for selecting a training data item from the training data; a first neural network connected to receive as input the training data item selected by the training data selection means; sampling means for acquiring a sample from a distribution corresponding to the label of the training data item selected by the training data selection means, among a plurality of different distributions that are prepared in advance according to the plurality of types of labels; a second neural network that receives as an input the output of the network; a third neural network that is connected to selectively receive as input the output of the first neural network and the samples sampled by the sampling means; and training means for training the first, second and third neural networks such that the third neural network distinguishes between the samples obtained by the sampling means and the training data items selected by the training data selection means, and the second neural network estimates the labels attached to the training data items based on the output of the first neural network that has received the training data items selected by the training data selection means, and classifies the training data items, wherein when training by the training means is completed, the second neural network is an emotion classifier.

[0010] Preferably, the training means includes a first training means for training the third neural network so that the third neural network identifies the sample acquired by the sampling means as a first type of data; a second training means for training the first and third neural networks so that the third neural network identifies an output of the first neural network, which has received as input a training data item selected by the training data selection means, as a second type of data that is not the first type; a third training means for training the first and second neural networks so that the second neural network, which has received the output of the first neural network for the training data item selected by the training data selection means, outputs a classification result in accordance with the label attached to the training data item; and a training control means for controlling the training of the first, second and third neural networks by the training data selection means, the sampling means, and the first, second and third training means to be repeated while changing the training data selected by the training data selection means until a predetermined termination condition is established.

[0011] More preferably, each of the training data items is further provided with an attribute label indicating an attribute of a speaker, and the third training means trains the second neural network to classify the speaker's attribute label in addition to the emotion of the speaker of the training data item.

[0012] More preferably, the attribute label represents the gender of the speaker.

[0013] Preferably, the sampling means includes Gaussian distribution sampling means for obtaining a sample from a Gaussian distribution corresponding to the label of the training data item selected by the training data selection means from a plurality of different Gaussian distributions prepared in advance according to a plurality of types of labels.

[0014] More preferably, the multiple Gaussian distributions have different means.

[0015] More preferably, the multiple Gaussian distributions have equal variances.

[0016] Preferably, the means of the multiple Gaussian distributions differ from each other by one.

[0017] More preferably, the variances of the multiple Gaussian distributions are all less than half the smallest difference between the means of the multiple Gaussian distributions and greater than zero.

[0018] More preferably, the variances of any of the multiple Gaussian distributions are equal to or less than 0.4 times the minimum value of the means of the multiple Gaussian distributions.

[0019] Preferably, the output of the first neural network and the sample sampled by the sampling means are vectors of equal dimensions.

[0020] A training method according to a second aspect of the present invention is a training method for training an emotion classifier consisting of a neural network that classifies the emotion of a speaker represented by voice data, by a computer, the training method being used in connection with a training data storage device that stores a plurality of training data items, each of the plurality of training data items including voice data and a label indicating the emotion of the speaker represented by the voice data, the label being selected from a plurality of types of labels, the training method including the steps of: selecting a training data item from the plurality of training data items stored in the training data storage device by the computer; providing the data item selected in the selecting step to an input of a first neural network by the computer; acquiring a sample from a distribution corresponding to the label of the training data item selected in the selecting step, from a plurality of different distributions that are prepared in advance according to the plurality of types of labels by the computer; the computer inputs an output of the first neural network to the second neural network; the computer selectively inputs the output of the first neural network and the samples sampled in the sample obtaining step to the third neural network; and the computer trains the first, second, and third neural networks such that the third neural network identifies the samples obtained in the sample obtaining step and the training data item selected in the selecting step, and the second neural network estimates a label assigned to the training data item based on the output of the first neural network that has received the training data item selected in the selecting step, and classifies the training data item, wherein the second neural network becomes an emotion classifier when training in the training step is completed.

[0021] Preferably, the training step includes a first training step in which the computer trains the third neural network so that the third neural network identifies the sample acquired in the sample acquiring step as a first type of data; a second training step in which the computer trains the first and third neural networks so that the third neural network identifies the output of the first neural network, which has received the training data item selected in the selecting step as input, as a second type of data that is not the first type; a third training step in which the computer trains the first and second neural networks so that the second neural network, which has received the output of the first neural network for the training data item selected in the selecting step, outputs a classification result in accordance with the label attached to the training data item; and a repeating step in which the computer repeats the selecting step, the sample acquiring step, and the first, second and third training steps while changing the training data item selected in the selecting step until a predetermined termination condition is met.

[0022] More preferably, each of the training data items is further provided with an attribute label indicating a type of predetermined attribute of the speaker, and the third training step includes a step of the computer training the second neural network to classify the predetermined attribute of the speaker in addition to the emotion of the speaker of the training data items.

[0023] More preferably, the predetermined attribute is the gender of the speaker.

[0024] Preferably, the step of obtaining the sample includes a step of the computer obtaining the sample from a Gaussian distribution corresponding to the label of the training data item selected in the selecting step from among a plurality of different Gaussian distributions that are prepared in advance according to a plurality of types of labels.

[0025] The above and other objects, features, aspects and advantages of the present invention will become apparent from the following detailed description of the invention taken in conjunction with the accompanying drawings. [Brief description of the drawings]

[0026] [Figure 1] FIG. 1 is a schematic block diagram of a classifier training apparatus according to the first embodiment. [Diagram 2] FIG. 2 is a schematic block diagram of the encoder shown in FIG. [Diagram 3] FIG. 3 is a schematic block diagram of the preceding utterance feature extraction unit shown in FIG. [Figure 4] FIG. 4 is a schematic block diagram of the extended convolution block shown in FIG. [Diagram 5] FIG. 5 is a schematic block diagram of the classifier shown in FIG. [Figure 6] FIG. 6 is a schematic block diagram of the classifier shown in FIG. [Figure 7] FIG. 7 is a flowchart illustrating a control structure of a computer program for implementing, by a computer, the training of a classifier in accordance with the first embodiment. [Figure 8] FIG. 8 is an external view of a computer system for implementing each embodiment of the present invention. [Figure 9] FIG. 9 is a hardware block diagram of the computer system shown in FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0027] In the following description and drawings, the same parts are given the same reference numbers. Therefore, detailed description thereof will not be repeated. In the following description, a set of data consisting of a voice data sample and its label in the training data is referred to as a training data item.

[0028] 1. First embodiment 1. Configuration (1) Overall structure Referring to Fig. 1, a classifier training device 50 according to the first embodiment is used in connection with a training data storage device 60 that stores a large number of training data items, and includes a data selection unit 64 for selecting the training data items stored in the training data storage device 60 one by one when training an emotion classifier. Each training data item includes speech data and a label indicating the emotion of the speaker. In this embodiment, four types of labels are used: happy, sad, neutral, and angry. Of course, the types of labels are not limited to these four types, and any number of labels can be used.

[0029] The classifier training device 50 further includes an encoder 70 having an input configured to receive utterance data in the data items selected by the data selection unit 64, encoding the utterance data to output an intermediate representation which is a vector of a predetermined length representing information about the target utterance, a parameter storage unit 66 for storing parameters (the mean and variance of each distribution) specifying four types of Gaussian distributions prepared corresponding to the above-mentioned four types of labels, and a sampling unit 68 for obtaining samples from the Gaussian distributions corresponding to the labels included in the data items selected by the data selection unit 64.

[0030] The classifier training device 50 further includes a selection unit 72 having a first input connected to the output of the encoder 70 and a second input connected to the output of the sampling unit 68, for selecting and outputting an input to either the first input or the second input in response to a control signal given from the outside, a discriminator 74 having an input to receive the output of the selection unit 72, and trained to discriminate whether the input is an input from the encoder 70 or an input from the sampling unit 68, and a classifier 76 having an input to receive the output of the encoder 70, and trained to estimate which of the above four types of emotions the emotion of the speaker of the input voice data is classified into.

[0031] The classifier training device 50 further includes a first updating unit 78 for training the classifier 74 so that the classifier 74 correctly identifies an input provided to the classifier 74 from the sampling unit 68 via the selection unit 72 as an input from the sampling unit 68; a second updating unit 80 for training both the classifier 74 and the encoder 70 so that the classifier 74 correctly identifies an input provided to the classifier 74 from the encoder 70 via the selection unit 72 as an input from the encoder 70; a third updating unit 82 for training the classifier 76 and the encoder 70 so that the classifier 76 correctly classifies the input from the encoder 70 into any of the above four types of emotions; and a training control unit 62 for controlling the data selection unit 64, the sampling unit 68, the encoder 70, the selection unit 72, the classifier 74, the classifier 76, the first updating unit 78, the second updating unit 80, and the third updating unit 82, and for controlling each unit so that each unit repeatedly operates while changing the training data item selected from the training data storage device 60 until a predetermined termination condition is satisfied.

[0032] The training by the first update unit 78, the second update unit 80 and the third update unit 82 is all performed by the normal backpropagation method.

[0033] (2) Encoder 70 With reference to FIG. 2, the encoder 70 includes a current utterance feature extraction unit 124 for extracting from the training data 100 a feature matrix consisting of a time series of predetermined features of an utterance (current utterance) to be classified into emotions, a previous utterance feature extraction unit 120 for extracting the time series of the features from an utterance (previous utterance) of the same speaker which precedes the current utterance across the utterance of the other party in the conversation, and outputting the time series of the features as a feature matrix, and a partner utterance feature extraction unit 122 for extracting the time series of the features from the other party's utterance between the previous utterance and the current utterance, and outputting the time series of the features as a feature matrix.

[0034] The encoder 70 further includes a multiplication unit 126 for multiplying the feature matrix from the previous speech feature extraction unit 120 and the feature matrix from the partner speech feature extraction unit 122, a softmax calculation unit 128 for performing a Softmax calculation on each element of the output of the multiplication unit 126, a multiplication unit 130 for multiplying the output of the softmax calculation unit 128 and the output of the current speech feature extraction unit 124, and a concatenation unit 132 for concatenating the output of the multiplication unit 130 and the output of the current speech feature extraction unit 124.

[0035] (3) Previous utterance feature extraction unit 120 2 are all configured in the same manner. Therefore, the configuration of the previous utterance feature extraction unit 120 will be described here as an example.

[0036] 3, the preceding speech feature extraction unit 120 includes an acoustic feature extraction block 202 for framing speech data in each training data item included in the training data 100 by frames of a predetermined length and a predetermined stride, and extracting predetermined acoustic features (in this embodiment, MFCC (Mel-frequency cepstral coefficient), F0, zero crossing rate, and statistics thereof) for each frame as a time-series vector sequence, an extended convolution block 204 for performing extended convolution on the vector sequence output from the acoustic feature extraction block 202, and a convolution block 206 for performing normal convolution on the vector sequence output from the acoustic feature extraction block 202. A segment level feature 208 is obtained from the output of the extended convolution block 204 and the output of the convolution block 206.

[0037] The prior speech feature extraction unit 120 further includes a bidirectional GRU 212 that processes the segment-level features 208 at the utterance level to calculate and output the state in each segment, a dot-product attention block 214 that calculates attention between segments by performing a dot product operation between the bottleneck features 210 and the output of the bidirectional GRU 212, a global pooling layer 216 that performs global pooling on the output of the dot-product attention block 214, a global pooling layer 218 that performs global pooling on the output of the bidirectional GRU 212, and a multiplication block 220 that outputs a vector sequence with attention by multiplying the output of the global pooling layer 216 and the output of the global pooling layer 218.

[0038] The calculation in Fig. 3 will be described in more detail. First, the audio data is divided into frames by the acoustic feature extraction block 202, and a sequence of feature vectors s1, s2, ..., s n That is, each voice data is sample data x i is converted to

[0039]

number

[0040] The extended convolution blocks 204 and 206 shown in FIG. i To obtain the vector h i As shown in equation (3), each vector h i From the elements of the bottleneck layer in,a bottleneck feature 210, denoted by BT, is obtained.

[0041]

number

[0042] The bidirectional GRU 212 shown in Fig. 3 processes the utterance-level segment-level features 208 to obtain a matrix indicated by Key in the following formula (4). The following relates to the attention mechanism. That is, the dot-product attention block 214 applies a dot product between this matrix and the bottleneck feature 210 represented by BT described above to calculate the attention A in formula (4). The output of the global pooling layer 216 that received this attention is KQ shown below, the output of the global pooling layer 218 for the output Key of the bidirectional GRU 212 is V shown below, and the multiplication block 220 multiplies the matrix between KQ and V to obtain the attentioned hidden feature matrix HF.

[0043]

number

[0044] 4, the extended convolution block 204 shown in Fig. 3 includes a convolution layer 252 that performs convolution on the input, a convolution layer 254 that performs further convolution on the output of the convolution layer 252, and an extended convolution layer 256 that performs extended convolution on the output of the convolution layer 254. The output of the extended convolution layer 256 becomes the output of the extended convolution block 204.

[0045] 2, the other-party utterance feature extraction unit 122 and the current utterance feature extraction unit 124 have the same configuration as the previous utterance feature extraction unit 120. From these, therefore, the hidden feature of the speaker's previous utterance, the hidden feature of the previous conversation partner's utterance, and the speaker's current (target) hidden feature (HF pre , H.F. Int , H.F. cur These are concatenated by a multiplier 126, a softmax calculator 128, a multiplier 130, and a concatenator 132 as shown in the following formula (5), and the speech of the training data items of the training data 100 is converted into an intermediate representation Z.

[0046]

number

[0047] (4) Classifier 74 5, the classifier 74 shown in Fig. 2 includes a fully connected layer 302 arranged to receive an input to the classifier 74, and a fully connected layer 304 that receives an output of the fully connected layer 302. The output of the fully connected layer 304 becomes the output of the classifier 74.

[0048] (5) Classifier 76 6, the classifier 76 includes fully connected layers 352 and 354 arranged in series for classifying the emotion of the speaker from the input, and fully connected layers 356 and 358 arranged in series for classifying a certain attribute of the speaker (gender in this case) that does not change over time like emotion from the input. The output of the fully connected layer 354 is a vector with four elements, and these values ​​indicate the scores of the four types of emotions. The emotion corresponding to the maximum score is the classification result. The output of the fully connected layer 358 is a vector with two elements, and these values ​​indicate the score of the speaker's gender. The larger score indicates the speaker's gender.

[0049] The objective of this embodiment is to classify emotions from voice. To achieve this, the classifier 76 only needs to have fully connected layers 352 and 354 for classifying emotions. However, as shown in this embodiment, it is known that the accuracy of the intended classification can be improved by training the classifier to perform not only one intended classification, but also two types of classification including a classification that is not directly related to the objective (multitask learning). Therefore, in this embodiment, the classifier 76 is also configured to classify gender in addition to emotion.

[0050] (6) Program Structure FIG. 7 shows a control structure of a computer program for implementing classifier training apparatus 50 having the above configuration by computer hardware.

[0051] 7, this program includes step 400 of initializing each variable area used in the program after startup, step 402 of repeatedly executing step 404 until a predetermined termination condition is met, and step 406 of performing a predetermined termination process after step 402 is completed. The termination process includes a process of saving the parameters learned in step 402, particularly a set of parameters of at least classifier 76 shown in FIG. 1, in a predetermined storage device. The termination condition here is that a predetermined number of epochs have been completed, with one epoch being defined as a process of executing the following processes using all training data items included in the training data.

[0052] In this embodiment, training is performed in a mini-batch format, but parameters are updated for each sample. That is, step 404 includes step 420 of sampling training data items from the training data by the number of data items in the batch, and step 422 of updating the parameters of the encoder 70, the discriminator 74, and the classifier 76 by backpropagation using the errors calculated for the training data items in the batch, by performing step 424 for each training data item sampled in step 420.

[0053] Step 424 includes step 450 of extracting a label included in the training data item to be processed, step 452 of selecting a Gaussian distribution corresponding to the type of this label and performing sampling, step 454 of inputting the sample sampled in step 452 to a discriminator 74, and step 456 of updating the parameters of the discriminator 74 based on the output of the discriminator 74. Here, the discriminator 74 is trained so that when a sample from a Gaussian distribution is input to the discriminator 74, the output becomes false, whereas, as will be described later, when data from the encoder 70 is input to the discriminator 74, the output becomes true. That is, in step 456, the parameters of the discriminator 74 are updated so that the error between the output of the discriminator 74 and a value representing a false is reduced.

[0054] In this embodiment, the loss function L used in step 456 is D is expressed by the following formula (6). Here, ε is a vector randomly sampled from a Gaussian distribution corresponding to the label. The dimension of this sample is the same as the dimension of the intermediate representation Z described above. D() indicates the output of the classifier 74.

[0055]

number

[0056] Step 404 further includes step 458 of inputting the speech data of the training data items sampled in step 450 to the encoder 70 to obtain an intermediate representation thereof at the output of the encoder 70, step 460 of inputting the data obtained as the output of the encoder 70 in step 458 to the discriminator 74 to obtain a classification result, and step 462 of updating parameters of both the encoder 70 and the discriminator 74 so as to reduce the error between the classification result obtained from the discriminator 74 in step 460 and a value representing the true value. As described above, the purpose in this example is to train the discriminator 74 so that the discriminator 74 determines that the input obtained from the actual training data items is true, but at the same time, the encoder 70 is also trained in a similar manner.

[0057] Step 404 further includes step 464 of inputting the speech data of the training data item sampled in step 450 to the encoder 70 to obtain an intermediate representation thereof, step 466 of inputting the intermediate representation obtained in step 464 to the classifier 76 to obtain classification results of both gender and emotion by the classifier 76, and step 468 of updating the parameters of the encoder 70 and the classifier 76 by backpropagation so that the errors between the two classification results obtained in step 466 and the correct labels are both reduced, thereby terminating the processing for this training data item.

[0058] The loss function Lc in training the classifier 76 is expressed by the following equation (7).

[0059]

number

[0060] The loss function L used in training the encoder 70 in this embodiment is enc is expressed as the following equation (8).

[0061]

number

[0062] In step 402, step 404 is repeated on the entire training data while changing the training data items sampled in step 450, until a predetermined end condition is met. In this embodiment, training is terminated when the processing of step 404 is performed a predetermined number of times on all training data items. A classifier that classifies the emotion and gender of a speaker from speech voice data can be realized by the parameters of classifier 76 at the end of this training and a group of programs that realize a neural network that is the substance of classifier 76.

[0063] 2.Operation The classifier training device 50 according to the first embodiment operates as follows. A large number of training data items are prepared in advance in the training data storage device 60. Each of these training data items includes utterance data constituting a dialogue, a label representing the emotion of the speaker, and a gender label indicating the gender of the speaker. In this example, the utterance data includes an utterance to be classified, an utterance by the same speaker preceding the utterance, and an utterance by the other party in the dialogue existing between the two utterances. In addition, the parameter storage unit 66 stores parameters (mean and variance) specifying four types of Gaussian distributions corresponding to four types of labels indicating emotions.

[0064] The training control unit 62 repeats the following process until the end condition is met. Note that since the following process is all under the control of the training control unit 62, in order to make the description easier to understand, no further reference will be made to the control of the training control unit 62.

[0065] First, the data selection unit 64 selects one training data item from the training data storage device 60 (step 450 in FIG. 7. In the following description, each "step" is as shown in FIG. 7). The sampling unit 68 checks the label included in the training data item selected by the data selection unit 64 via the training control unit 62, and samples from a Gaussian distribution specified by the parameters corresponding to the label stored in the parameter storage unit 66 (step 452). The sampling unit 68 provides the sampled sample to the selection unit 72. The selection unit 72 provides the sample from the sampling unit 68 to the classifier 74 (step 454). The classifier 74 performs an internal operation on this input and outputs the result. Since the input to the classifier 74 is from a Gaussian distribution, the first update unit 78 updates the parameters of the classifier 74 by the backpropagation method so that the output of the classifier 74 approaches a value indicating a false positive (step 456).

[0066] Next, the data selection unit 64 provides the speech data of the selected training data item to the encoder 70 (step 458). In step 458, the speech data is processed to generate an intermediate representation, which is provided to the selection unit 72. The selection unit 72 selects the intermediate representation from the encoder 70 and provides it to the classifier 74 (step 460). The classifier 74 processes the intermediate representation and outputs the result. The second update unit 80 updates the parameters of the encoder 70 and the classifier 74 by backpropagation so as to reduce the error between the output of the classifier 74 and the value indicating the true value (step 462).

[0067] The data selection unit 64 again provides the selected training data item to the encoder 70. The encoder 70 performs an operation on this input using the updated parameters and outputs an intermediate representation thereof. This intermediate representation is input to the classifier 76 (step 466). The classifier 76 processes the input and outputs the result. The third update unit 82 updates the outputs of the encoder 70 and the classifier 76 by backpropagation so that the emotion classification result output by the classifier 76 is close to the value representing the label attached to the training data item selected by the data selection unit 64, and so that the gender classification result output by the classifier 76 is close to the value representing the gender label attached to the training data item.

[0068] Each unit of the classifier training device 50 repeats the above-mentioned training until the processing using all the training data items stored in the training data storage device 60 is executed a predetermined number of times. A classifier capable of classifying the emotion of a speaker based on input voice data is realized by the parameters of the classifier 76 at the time of completion of the training and a program having an algorithm for implementing the neural network of the classifier 76. As can be seen from the above-mentioned training method, this classifier can also classify the gender of the speaker.

[0069] 3. Computer implementation Each unit of the classifier training apparatus 50 according to the first embodiment is realized by a program having a control structure shown in FIG. 7, which is executed on computer hardware on which an operating system having appropriate program execution and data handling functions runs.

[0070] Fig. 8 is an external view of a computer system for implementing each of the above embodiments, and Fig. 9 is a hardware block diagram of the computer system shown in Fig. 8.

[0071] 8, this computer system 950 includes a computer 970 having a DVD (Digital Versatile Disc) drive 1002, and a keyboard 974, a mouse 976, a monitor 972, a pair of speakers 980, and a microphone 982 for interacting with a user, all of which are connected to the computer 970. Of course, these are just one example of a configuration for when user interaction is required, and any general hardware and software (e.g., a touch panel, a general pointing device) that can be used for user interaction can be used.

[0072] 9, the computer 970 includes, in addition to a DVD drive 1002, a CPU (Central Processing Unit) 990, a GPU (Graphics Processing Unit) 992, a bus 1010 connected to the CPU 990, the GPU 992, and the DVD drive 1002, a ROM (Read-Only Memory) 996 connected to the bus 1010 and storing a boot-up program of the computer 970, a RAM (Random Access Memory) 998 connected to the bus 1010 and storing instructions constituting a program, a system program, working data, and the like, and a SSD (Solid State Drive) 1000 which is a non-volatile memory connected to the bus 1010. The SSD 1000 is for storing programs executed by the CPU 990 and the GPU 992, and data used by the programs executed by the CPU 990 and the GPU 992, and the like. The computer 970 further includes a network I / F (Interface) 1008 that provides a connection to a network 986 enabling communication with other terminals, and a USB port 1006 to which a USB (Universal Serial Bus) memory 984 can be attached / detached and that provides communication between the USB memory 984 and each component within the computer 970.

[0073] The computer 970 further includes an audio I / F 1004 that is connected to the microphone 982, the speaker 980, and the bus 1010, and that reads out audio signals, video signals, and text data generated by the CPU 990 and stored in the RAM 998 or the SSD 1000 in accordance with instructions from the CPU 990, performs analog conversion and amplification processing to drive the speaker 980, and digitizes the analog audio signal from the microphone 982 and stores it in any address of the RAM 998 or the SSD 1000 specified by the CPU 990.

[0074] In the above embodiment, parameters of the encoder 70, the discriminator 74, the classifier 76, etc., shown in Fig. 1, as well as training data, etc., are all stored in, for example, the SSD 1000, the RAM 998, the DVD 978, or the USB memory 984 shown in Fig. 9, or a storage medium of an external device (not shown) connected via the network I / F 1008 and the network 986. Typically, these data, parameters, etc. are written into the SSD 1000 from the outside, for example, and loaded into the RAM 998 when the computer 970 is executed.

[0075] A computer program for operating this computer system to realize the functions of the training data storage device 60, the training control unit 62, the data selection unit 64, the encoder 70, the selection unit 72, the discriminator 74, the classifier 76, the first update unit 78, the second update unit 80, and the third update unit 82 of the classifier training device 50 shown in Fig. 1 and each of the components thereof is stored in a DVD 978 inserted in a DVD drive 1002 and transferred from the DVD drive 1002 to the SSD 1000. Alternatively, these programs are stored in a USB memory 984, and the USB memory 984 is inserted in a USB port 1006 and the programs are transferred to the SSD 1000. Alternatively, the programs may be transmitted to the computer 970 via a network 986 and stored in the SSD 1000.

[0076] The program is loaded into RAM 998 at the time of execution. Of course, a source program may be input using keyboard 974, monitor 972, and mouse 976, and the compiled object program may be stored in SSD 1000. In the case of a script language, a script input using keyboard 974 or the like may be stored in SSD 1000. In the case of a program that runs on a virtual machine, a program that functions as a virtual machine must be installed in computer 970 in advance. However, since training using classifier training device 50 involves a large amount of calculation, it is preferable to realize each part of the embodiment of the present invention as an object program consisting of native computer code rather than a script language.

[0077] The CPU 990 reads a program from the RAM 998 according to an address indicated by an internal register called a program counter (not shown) and interprets the instruction. The CPU 990 further reads data required for executing the instruction from the RAM 998, the SSD 1000, or other devices according to an address specified by the instruction, and executes the process specified by the instruction. The CPU 990 stores the execution result data at an address specified by the program, such as the RAM 998, the SSD 1000, or a register in the CPU 990. At this time, the value of the program counter is also updated by the CPU 990 operating according to the program. The computer program may be directly loaded into the RAM 998 from the DVD 978, the USB memory 984, or via a network. Note that, among the programs executed by the CPU 990, some tasks (mainly numerical calculations that can be executed in parallel) are executed by the GPU 992 according to instructions included in the program or according to the analysis results obtained when the CPU 990 executes the instructions.

[0078] The program for implementing the functions of each unit according to each embodiment described above by the computer 970 includes a plurality of instructions written and arranged to operate the computer 970 to implement those functions. Some of the basic functions required to execute the instructions are provided by the operating system or third-party programs running on the computer 970, or by modules of various tool kits installed on the computer 970, and are linked to the object program by dynamic linking at the time of execution. Therefore, the program does not necessarily include all the functions required to implement the system and method of this embodiment. The program may include only instructions to execute the operations of each of the above-mentioned devices and their components by calling the appropriate functions or functions of the "programming tool kit" in a controlled manner to obtain the desired results. The method of operating the computer 970 for that purpose is well known, and will not be repeated here.

[0079] The GPU 992 is capable of parallel processing, and can simultaneously execute a large amount of calculations associated with machine learning in parallel or in a pipelined manner. For example, parallel calculation elements found in a program when the program is compiled, or parallel calculation elements found when the program is executed, are issued from the CPU 990 to the GPU 992 at any time, and executed, and the results are returned to the CPU 990 directly or via a predetermined address in the RAM 998, and assigned to a predetermined variable in the program.

[0080] 4.Effects According to the first embodiment, it has been confirmed by experiments, as described below, that even when the number of existing training data items is insufficient, classification accuracy can be improved without artificially increasing the number of training data items. As a method for training a classifier, an effect is obtained in that the speaker's emotion can be classified from voice data with sufficient accuracy by a relatively simple method not previously used.

[0081] Second Second Embodiment In the first embodiment, the classifier 74 is first trained using samples sampled from a Gaussian distribution, then the classifier 74 and the encoder 70 are trained using selected training data items, and then the encoder 70 and the classifier 76 are trained in this order. However, the present invention is not limited to such an embodiment. As long as the above three types of training are performed as one set for one training data item, the order is not limited to that of the first embodiment.

[0082] In the second embodiment, first the encoder 70 and the discriminator 74 are trained, then the encoder 70 and the classifier 76 are trained, and finally the discriminator 74. The classifier 76 trained in this manner can also achieve the same effects as in the first embodiment.

[0083] Third Experiment 1. Setup In order to test the performance of the classifier training apparatus 50 according to the first embodiment, the following experiment was carried out. In the experiment, two types of data sets called IEMOCAP and MELD were used.

[0084] IEMOCAP is used in Non-Patent Document 2 and consists of videos of one-to-one two-way dialogues between 10 speakers. Each video is divided into two dialogues, one for each utterance, and an emotion label is attached. In this experiment, four emotion labels, happiness, sadness, neutral, and anger, were used as described above. In the original dataset, the labels excitement and dissatisfaction were also attached, but as in many studies, excitement was merged into happiness and dissatisfaction into sadness. As in Non-Patent Document 2, 80% of the data was randomly divided into a training dataset and 20% into a test set, and five-fold cross-validation was performed. The training samples were 5946 samples and the test samples were 1487 samples. In this experiment, the results disclosed in Non-Patent Document 2 were used for comparison.

[0085] MELD is a multi-party conversation dataset collected from American TV series. In this experiment, we extracted samples from the original dataset in the categories of "joy", "sadness", "neutral", and "anger" to ensure consistency in the comparison with the training data of IEMOCAP in the experiment. To ensure that cross-corpus validation can be performed with the same data structure, we selected 2000 samples from the pre-selected training dataset that include the speaker's preceding utterance and the interlocutor's utterance. The number of training samples is 8292. Since this dataset does not include information indicating the speaker's gender, we only classify the speaker's emotions when using this dataset.

[0086] In this experiment, the kernel sizes in the dilated convolution block were set to 512, 512, and 128 for each layer. The sizes of each kernel were 1, 3, and 1. The dilated size in the dilated convolution was set to 2. The number of nodes in the bottleneck was set to 512, and the number of cells in BiGRU was set to 256. The batch size was 32. Adam was used as the optimizer, and the learning rate was set to 10 -4 The means of the four Gaussian distributions corresponding to each emotion label were set to 0, 1, 2, and 3, respectively, and the variances were all set to 0.3. The values ​​of the means and variances should be such that the Gaussian distributions can be separated to some extent from each other. Here, the difference between the means of adjacent Gaussian distributions was set to 1, but it may be larger than this, or may be smaller than this to reduce the variance. When the difference between the means is set to 1, as in this experiment, a variance of 0.5 or less will allow for some separation, and a variance of 0.4 or less will allow for further separation. In this experiment, the variance was set to 0.3, and therefore the separation between the Gaussian distributions is considered to be substantially sufficient.

[0087] In this embodiment, the differences in the means of adjacent Gaussian distributions are all equal, being 1. However, the present invention is not limited to such an embodiment, and the differences in the means of adjacent Gaussian distributions may differ from each other, or may only be partially different. In this case, however, it is desirable to set the variance to half or less of the difference in the means of adjacent Gaussian distributions. It is even more desirable to set the variance to half or less of the minimum difference in the means of adjacent Gaussian distributions.

[0088] Performance was measured using UA (Unweighted Accuracy) and WA (Weighted Accuracy). UA means the average overall accuracy, and WA means the percentage of correctly classified samples. In the experiment, we performed model performance on the test dataset and cross-corpus validation (a validation method in which the performance of a model trained on one corpus is tested on data from the other corpus) for each epoch of training, and saved the best model.

[0089] As described above, in this experiment, the performance of the model described in Non-Patent Document 2 was used as a baseline. Furthermore, in order to investigate the effect of regularizing the intermediate representation by mapping it to Gaussian distributions separated from each other, a model using a normal Gaussian distribution (mean 1.0, variance 0.3) was prepared (MAEC). Separately from this, a model without a classifier 74 was also prepared (MEC).

[0090] 2.Results The experimental results are shown in Table 1 below, together with the results of the baseline model described in Non-Patent Document 2. In Table 1, ISO-MAEC shows the model according to the above embodiment.

[0091] [Table 1]

[0092] In Table 1, for example, the number in the cell at the intersection of the row labeled "IEMOCAP" and the column labeled "Training results with IEMOCAP" indicates the performance of each model trained with IEMOCAP in an experiment using IEMOCAP as test data. The number in the cell at the intersection of the row labeled "IEMOCAP" and the column labeled "Training results with MELD" indicates the performance of each model trained with MELD in an experiment (cross-corpus validation) using IEMOCAP as test data. The same is true for the row labeled "MELD."

[0093] With reference to Table 1, when trained with IEMOCAP, the MAEC model showed about 2.8% and 2.4% performance improvement in WA and UA, respectively, compared to MEC by mapping to a single Gaussian distribution. However, the ISO-MAEC model according to the above embodiment 1 showed an additional 1.87% and 2.23% performance improvement in WA and UA, respectively, by mapping the intermediate representation to multiple Gaussian distributions separated from each other according to the label. In cross-corpus validation, the model according to embodiment 1 showed a performance improvement of more than 9% and 1% in WA and UA, respectively, compared to both MAEC and MEC.

[0094] When trained with MELD, the MEC model performed 49.65% and 57.17% better in WA and UA, respectively. The MAEC model, by incorporating a regularization function that maps the intermediate representation to a Gaussian distribution, performed 3.53% and 1.08% better in WA and UA, respectively, than the MEC model. The ISO-MAEC model according to the above embodiment performed 0.95% better in WA. In cross-corpus validation, the MEC model performed 33.31% and 48.21% better in WA and UA, respectively. In contrast, MAEC performed 3.24% and 0.89% better in WA and UA, respectively, than the MEC model. The ISO-MAEC according to the above embodiment performed an additional 9.05% and 3.48% better in WA and UA, respectively, by adopting a regularization function that maps the intermediate representation to a separated Gaussian distribution.

[0095] As described above, this experiment shows that not only does regularizing the input by simply mapping the intermediate representation to a Gaussian distribution improve performance, but also that applying the method of the above embodiment, which maps the intermediate representation to separate and mutually separated Gaussian distributions for each label, improves performance even further.

[0096] The embodiments disclosed herein are merely illustrative, and the present invention is not limited to the above-described embodiments. The scope of the present invention is defined by the claims of the appended claims, taking into consideration the detailed description of the invention, and includes all modifications within the scope and meaning equivalent to the words described therein. [Explanation of symbols]

[0097] 50 Classifier training device 60 Training Data Storage Device 62 Training Control Unit 64 Data Selection Section 66 Parameter storage section 68 Sampling section 70 Encoder 72 Selection section 74 Classifier 76 Classifier 78 1st update part 80 2nd update part 82 Third update part 100 training data 120 Preceding speech feature extraction unit 122 Extraction of features of other party's speech 124 Current speech feature extraction unit 126, 130 Multiplication section 128 Softmax Calculation Unit 132 Connection part 202 Acoustic feature extraction block 204 Extended Convolution Block 206 Convolution Blocks 208 Segment-level features 210 Bottleneck Features 212 Bidirectional GRU 214 Dot Product Attention Block 216, 218 Global Pooling Layer 220 Multiplication Block 252, 254 convolutional layer 256 dilated convolutional layers

Claims

1. A training device for training an emotion classifier made of a neural network that classifies an emotion of a speaker represented by voice data, comprising: in connection with a training data storage device that stores training data including a plurality of training data items; Each of the plurality of training data items includes voice data and a label indicating an emotion of a speaker represented by the voice data, the label being selected from a plurality of types of labels; The training device comprises: training data selection means for selecting training data items from the training data; a first neural network connected to receive as input the training data items selected by said training data selection means; a sampling means for acquiring a sample from a distribution corresponding to the label of the training data item selected by the training data selection means from a plurality of different distributions prepared in advance according to the plurality of types of labels; a second neural network that receives as an input the output of the first neural network; a third neural network connected to selectively receive as inputs the output of the first neural network and the samples sampled by the sampling means; training means for training the first and third neural networks so that the third neural network distinguishes between the samples acquired by the sampling means and an output of the first neural network that has received the training data item selected by the training data selection means, and training means for training the first and second neural networks so that the second neural network estimates a label assigned to the training data item based on the output of the first neural network that has received the training data item selected by the training data selection means, and classifies the training data item; When training by the training means is completed, the second neural network is the emotion classifier.

2. The training means includes: a first training means for training the third neural network to identify samples obtained by the sampling means as a first type of data; a second training means for training the first neural network and the third neural network so that the third neural network identifies an output of the first neural network that has received as input the training data items selected by the training data selection means as a second type of data that is not the first type; a third training means for training the first and second neural networks so that the second neural network, which receives an output of the first neural network for the training data item selected by the training data selection means, outputs a classification result according to the label attached to the training data item; and 2. The training device according to claim 1, further comprising: a training control means for controlling the training of the first, second, and third neural networks by the training data selection means, the sampling means, and the first, second, and third training means to be repeated while changing the training data items selected by the training data selection means until a predetermined termination condition is met.

3. Each of the training data items is further provided with an attribute label indicating an attribute of a speaker; 3. The training apparatus of claim 2, wherein the third training means trains the second neural network to classify the sentiment of a speaker of the training data items as well as the attribute labels of the speaker.

4. The training device of claim 3 , wherein the attribute label represents a gender of a speaker.

5. 5. The training device according to claim 1, wherein the sampling means includes Gaussian distribution sampling means for obtaining a sample from a Gaussian distribution corresponding to the label of the training data item selected by the training data selection means, from a plurality of different Gaussian distributions prepared in advance according to the plurality of types of labels.

6. The training device of claim 5 , wherein the plurality of Gaussian distributions have different means.

7. The training device of claim 6 , wherein the plurality of Gaussian distributions have equal variances.

8. 8. The training device according to claim 6 or 7, wherein the means of the plurality of Gaussian distributions differ from each other by one.

9. The training device according to claim 6 , wherein the variances of the plurality of Gaussian distributions are all equal to or less than half the minimum value of the difference between the means of the plurality of Gaussian distributions and are greater than zero.

10. The training device of claim 9 , wherein the variances of the plurality of Gaussian distributions are all less than or equal to 0.4 times the minimum value of the means of the plurality of Gaussian distributions.

11. 11. The training device according to claim 1, wherein the output of the first neural network and the sample sampled by the sampling means are vectors of equal dimensions.

12. A method for training an emotion classifier, which is made of a neural network, by a computer to classify an emotion of a speaker represented by speech data, comprising the steps of: in connection with a training data storage device that stores training data including a plurality of training data items; Each of the plurality of training data items includes voice data and a label indicating an emotion of a speaker represented by the voice data, the label being selected from a plurality of types of labels; The training method includes: a computer selecting a training data item from the plurality of training data items stored in the training data storage device; providing the data items selected in the selecting step to an input of a first neural network by the computer; A step of acquiring samples from a distribution corresponding to the label of the training data item selected in the selecting step, from among a plurality of different distributions prepared in advance according to the plurality of types of labels by a computer; A computer inputs an output of the first neural network into a second neural network; A step of selectively inputting the output of the first neural network and the sample sampled in the step of obtaining the sample into a third neural network by a computer; a step of training the first and third neural networks by a computer so that the third neural network distinguishes between the samples acquired in the step of acquiring the samples and an output of the first neural network that has received the training data item selected in the step of selecting, and a step of training the first and second neural networks by a computer so that the second neural network estimates a label assigned to the training data item based on the output of the first neural network that has received the training data item selected in the step of selecting, and classifies the training data item; The method of training, wherein the second neural network becomes the emotion classifier upon completion of training in the training step.

13. The training step includes: a first training step in which a computer trains the third neural network so that the third neural network identifies samples acquired in the step of acquiring samples as a first type of data; a second training step in which a computer trains the first neural network and the third neural network so that the third neural network identifies an output of the first neural network that has received as input the training data items selected in the selecting step as a second type of data that is not the first type; a third training step in which a computer trains the first and second neural networks so that the second neural network, which receives an output of the first neural network for the training data item selected in the selecting step, outputs a classification result according to the label attached to the training data item; 13. The training method of claim 12, further comprising: a repeating step in which the computer repeats the selecting step, the sample obtaining step, and the first, second, and third training steps while varying the training data items selected in the selecting step, until a predetermined termination condition is met.

14. Each of the training data items is further provided with an attribute label indicating a type of a predetermined attribute of the speaker; 14. The method of claim 13, wherein the third training step includes a step of training the second neural network by a computer to classify the predetermined attributes of speakers of the training data items in addition to their emotions.

15. The training method of claim 14 , wherein the predetermined attribute is a gender of a speaker.

16. 16. The training method according to claim 12, wherein the step of obtaining the sample includes a step of obtaining, by a computer, a sample from a Gaussian distribution corresponding to the label of the training data item selected in the selecting step, from a plurality of different Gaussian distributions that are prepared in advance according to the plurality of types of labels.

Images