Classifier training method, training device, and computer program

Abstract

To enable model training with a high reliability about data of a modality in which it is difficult to accurately know human internal conditions.SOLUTION: A classifier training method includes the steps of: training an encoder 90 so as to generate data Z from an input X; training a decoder 94 so as to minimize an error to X of a new sample X' for decoder training generated from a decode result X'ε by the decoder 94 of a sample ε from a predetermined distribution and a decode result X'Z by the decoder 94, of an output Z of the encoder to X; and training a classifier 162 so as to maximize a probability of classifying X' and X, and training the encoder 90 and the decoder 94 so as to maximize a probability that the classifier 162 erroneously classifies X' and X. Any of a plurality of kinds of teacher labels is allocated to each of the sample ε and X.SELECTED DRAWING: Figure 10

Description

[Technical field]

[0001] The present invention relates to a classifier for classifying the inner state, such as the emotion, of a speaker from voice or the like, a training method and training device therefor, a computer program therefor, and an emotion classifier. [Background technology]

[0002] Recently, research on emotion estimation has been progressing. Emotion estimation is the estimation of a person's internal state based on the person's voice, facial expression, body movements, content of speech, etc. when they are communicating with other people. By estimating a person's emotion, it is possible to control the behavior of an agent that is conversing with a person, for example, or conversely, to convey to the other party what emotion the agent is feeling by controlling the agent's behavior.

[0003] Currently, machine learning using neural networks and other such technologies is the mainstream for estimating such internal states of people. In the case of text or images, it is relatively easy to collect a large amount of such data, classify it for learning, and create a database. However, in the case of voice, it is difficult to collect a large amount of such data and classify it reliably. Conventionally, for example, actors were asked to speak in a way that expresses a specific emotion, and the resulting voice was used. However, such intentionally expressed emotions are often exaggerated and too strong compared to expressions in everyday communication. Therefore, when data obtained from such utterances is used, there is a problem that the expression of emotions associated with everyday communication cannot be accurately captured. Therefore, recently, many researchers have been using voice obtained from everyday communication, rather than data obtained from intentionally expressed emotions, as basic data for emotion estimation.

[0004] However, it is extremely difficult for others to accurately classify the internal state of a speaker from speech obtained from such everyday communication. Even if it were possible to accurately classify such data, it would be biased and would not be possible to cover the entire actual distribution of what kind of speech results from emotions. Therefore, even if such data were used to train a model for emotion estimation, for example by machine learning, there is a problem that it is difficult to make a reliable estimation. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Patent Publication No. 2020-154332 [Non-patent literature]

[0006] [Non-Patent Document 1] Alireza Makhzani, Jonathon Shlens, Navdeep Jaitly, Ian Goodfellow, and Brendan Frey. 2015. Adversarial Autoencoders. arXiv preprint arXiv:1511.05644 (2015). [Non-Patent Document 2] Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative Adversarial Nets. In Advances in Neural Information Processing Systems, pages 2672-2680, 2014. [Non-Patent Document 3] Lukas Stappen, Alice Baird, Georgios Rizos, Panagiotis Tzirakis, Xinchen Du, Felix Hafner, Lea Schumann, Adria Mallol-Ragolta, Bjorn W Schuller, Iulia Lefter, et al. 2020. MuSe 2020-The First International Multimodal Sentiment Analysis in Real-life Media Challenge and Workshop. arXiv preprint arXiv:2004.14858 (2020). Summary of the Invention [Problem to be solved by the invention]

[0007] The aforementioned Patent Document 1 discloses a technology for setting an analysis interval of a voice, determining a pattern of change in the features of the voice data included in the analysis interval, and estimating the emotions of a speaker by using the voice data classified into the change patterns of the features corresponding to each emotion as training data.

[0008] However, the technology disclosed in Patent Document 1 also assumes the collection of a large amount of speech data, and it is necessary to determine a pattern of change in feature quantities corresponding to each emotion. To do this, it is necessary to accurately know the emotion of the speaker when he or she speaks. However, as mentioned above, it is difficult to collect a large amount of data for learning. If there is insufficient data, there is a problem that the reliability of the obtained model is low.

[0009] Therefore, an object of the present invention is to enable reliable model training for data of a modality, such as speech, in which it is difficult to accurately know the internal state of a human being. [Means for solving the problem]

[0010] A method for training a classifier according to a first aspect of the present invention includes the steps of: training an encoder by a computer to generate data from an input according to a predetermined distribution; training a decoder by the computer to minimize an error, relative to the first training data, of new samples for training the decoder generated from a result of decoding by the decoder of samples from the predetermined distribution and a result of decoding, using the decoder, of an output of the encoder for first training data; and training a decoder by the computer to minimize an error, relative to the first training data, of new samples for training the decoder generated from a result of decoding by the decoder of samples from the predetermined distribution and a result of decoding, using the decoder, of an output of the encoder for second training data. Ta and and training the encoder and the decoder to maximize the probability that the classifier correctly classifies the first training data, and each of the samples from the predetermined distribution and the first training data is assigned one of a plurality of types of teacher labels.

[0011] Preferably, each of the predetermined distribution, the first training data and the second training data includes speech voice data, and the plurality of types of teacher labels are labels indicating emotions of a speaker represented by the speech voice data.

[0012] More preferably, the step of training the encoder includes a step of training the predetermined classifier by a computer to maximize the probability that the predetermined classifier correctly classifies a sample from a predetermined distribution and the output of the encoder for the first training data, and training the encoder to maximize the probability that the predetermined classifier incorrectly classifies.

[0013] More preferably, the step of training the decoder includes the steps of: a computer decoding a sample from a predetermined distribution using the decoder to obtain a first decoded result; a computer decoding an output of the encoder for the first training data using the decoder to obtain a second decoded result; a computer generating a new decoded sample for training the classifier by performing a predetermined operation between the first decoded result and the second decoded result; and a computer updating parameters of the decoder using an error backpropagation method so that an error between the new decoded sample for training the classifier and the first training data is reduced.

[0014] Preferably, the step of training the decoder further includes a step of the computer repeating the steps of obtaining a first decoding result, obtaining a second decoding result, and updating parameters of the decoder until a predetermined termination condition is met.

[0015] More preferably, the step of generating a new decoded sample includes the step of the computer generating the new decoded sample by calculating an average of the first decoded result and the second decoded result.

[0016] More preferably, the step of training the classifier includes the steps of: a computer decoding a sample from a predetermined distribution with a decoder to obtain a first decoded result for training the classifier; a computer decoding an output of the encoder for second training data with a decoder to obtain a second decoded result for training the classifier; a computer generating a new decoded sample for training the classifier by performing a predetermined operation between the first decoded result for training the classifier and the second decoded result for training the classifier; a computer classifying the second training data or the new decoded sample for training the classifier with the classifier to obtain a classification result; and a computer updating parameters of the classifier so as to reduce the error by an error backpropagation method using an error between the classification result and a correct classification.

[0017] Preferably, the step of training the classifier further includes a step in which the computer repeats the steps of obtaining a first decoding result for training the classifier, obtaining a second decoding result for training the classifier, and updating parameters of the classifier until a predetermined termination condition is met.

[0018] More preferably, the step of generating new decoded samples for training the classifier includes a step of the computer generating new decoded samples for training the classifier by calculating an average of the first decoded result for training the classifier and the second decoded result for training the classifier.

[0019] A classifier according to a second aspect of the present invention is a classifier trained by any of the above-mentioned methods for training a classifier.

[0020] A computer program according to a third aspect of the present invention causes a computer to function as each of the means of any one of the above-mentioned devices.

[0021] A classifier training device according to a fourth aspect of the present invention includes encoder training means for training an encoder to generate data from an input according to a predetermined distribution; decoder training means for training the decoder to minimize an error, relative to the first training data, of new samples for decoder training generated from the results of decoding by the decoder of samples from the predetermined distribution and the results of decoding, using the decoder, of the encoder output for first training data; and decoder training means for training the decoder to minimize an error, relative to the first training data, of new samples for decoder training generated from the results of decoding by the decoder of samples from the predetermined distribution and the results of decoding, using the decoder, of the encoder output for second training data. Ta and and a classifier training means for training the classifier to maximize the probability of correctly classifying the first training data, and training the encoder and the decoder to maximize the probability of the classifier misclassifying the first training data, wherein each of the samples from the predetermined distribution and the first training data is assigned one of a plurality of types of teacher labels.

[0022] An emotion classifier according to a fifth aspect of the present invention includes emotion classification means for outputting an emotion label indicating the emotion represented by input data, and discrimination means for outputting a discrimination label indicating whether the data is real data or fake data generated by a predetermined data generation means.

[0023] Preferably, the input data is speech.

[0024] The above and other objects, features, aspects and advantages of the present invention will become apparent from the following detailed description of the invention taken in conjunction with the accompanying drawings. [Brief description of the drawings]

[0025] [Figure 1] FIG. 1 is a block diagram that illustrates the training process of a general emotion recognition model. [Diagram 2] FIG. 2 is a block diagram of a system that trains an encoder to learn the distribution of real data using adversarial learning. [Diagram 3]FIG. 3 is a diagram illustrating a training data expansion method according to the present invention. [Figure 4] FIG. 4 illustrates a schematic diagram of an arrangement for augmenting training data and a classifier training device for training a classifier of emotions on the training data, according to an embodiment of the present invention. [Diagram 5] FIG. 5 is a diagram illustrating a schematic configuration of the encoder shown in FIG. [Figure 6] FIG. 6 is a diagram illustrating a schematic configuration of the decoder shown in FIG. [Figure 7] FIG. 7 is a diagram illustrating a schematic configuration of the discriminator illustrated in FIG. [Figure 8] FIG. 8 is a diagram illustrating a schematic configuration of the classifier illustrated in FIG. [Figure 9] FIG. 9 is a flowchart illustrating a control structure of a computer program for implementing, by a computer, the training of the classifier training apparatus shown in FIG. [Figure 10] FIG. 10 is a schematic diagram for explaining the training process of the classifier training apparatus shown in FIG. [Figure 11] FIG. 11 is a schematic diagram showing the configuration of the classifier during testing. [Figure 12] FIG. 12 is a graph showing the composition of the training data obtained in the 10th iteration of the full training of the classifier. [Figure 13] FIG. 13 is a graph showing the composition of the training data obtained at the 100th iteration of the full training of the classifier. [Figure 14] FIG. 14 is a graph showing the composition of the training data obtained at the 500th iteration of the full training of the classifier. [Figure 15] FIG. 15 is a diagram showing the appearance of a computer system for implementing an embodiment of the present invention. [Figure 16] FIG. 16 is a block diagram showing the hardware configuration of the computer system whose external appearance is shown in FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0026] In the following description and drawings, the same parts are given the same reference numbers, and therefore detailed descriptions thereof will not be repeated.

[0027] <Conventional Technology> One approach to improving the accuracy of a model is to add new training samples from the available training samples.

[0028] 1 shows a schematic configuration of an emotion recognition model training system 50 that uses a conventional technique for adding training samples. This conventional technique is disclosed in the above-mentioned Non-Patent Document 1.

[0029] 1, the emotion recognition model training system 50 includes a real data storage device 60 consisting of samples obtained from real voice data, a training unit 64 that trains a generation model 62 of emotion-labeled voice so as to learn the distribution of sample data by adversarial learning using the samples stored in the real data storage device 60, an extended sample storage device 66 for storing extended samples (extended voice data) generated by the generation model 62, and a training unit 68 for training an emotion recognition model 70 using the sample data stored in the real data storage device 60 and the extended voice data stored in the extended sample storage device 66 as teacher data. When given voice data 72, the emotion recognition model 70 thus trained outputs an emotion label 74 indicating the emotion of the speaker when the voice was uttered. The basic idea of ​​adversarial learning is disclosed in Non-Patent Document 2.

[0030] FIG. 2 shows the configuration of the training unit 64 shown in FIG. 1. The training unit 64 trains the generative model 62 by adversarial learning. Referring to FIG. 2, the training unit 64 includes an encoder 90 that, when a sample X is given, performs an operation according to internal parameters to output a representation vector Z representing the sample X, a discriminator 92 that, when a noise ε sampled from a predetermined distribution or a representation vector Z from the encoder 90 is given, determines whether they are obtained from a predetermined distribution (real) or obtained from a sample by the encoder 90 (fake) and outputs a real / fake discrimination result, and a decoder 94 that is trained so that the extended sample X′ obtained by receiving and decoding the representation vector Z from the encoder 90 is equal to the sample X (so that the error is minimized). That is, the encoder 90 and the decoder 94 form an autoencoder.

[0031] In this training unit 64, a discriminator 92 is trained so as to correctly determine that noise ε is data obtained from a predetermined distribution and that the representation vector Z is the output of the encoder 90 obtained from the sample X, and at the same time, the encoder 90 is trained so as to maximize the probability that the discriminator 92 erroneously determines that the representation vector Z is noise ε, thus performing so-called adversarial learning. Furthermore, the decoder 94 is trained so that the extended sample X′ obtained by decoding the representation vector Z from the encoder 90 by the decoder 94 is equal to the sample X to the encoder 90. These two phases of training are performed until a predetermined termination condition is met.

[0032] As a result of such training, the decoder 94 learns the distribution implied by the noise ε, and when an input is given, it outputs an augmented sample X′ according to the distribution that gave rise to the noise ε. Thus, the sample is augmented by the augmented sample X′.

[0033] However, this conventional technology has the following problems. With reference to Fig. 3(A), it is generally considered that the distribution 122 of the test samples does not completely overlap with the distribution 120 of the training samples for training the emotion recognition model 70. That is, as shown in Fig. 3(A), the distribution 122 of the test samples partially overlaps with the distribution 120 of the training samples, but a portion of it is considered to be outside the distribution 120 of the training samples.

[0034] In the above-mentioned conventional technology, the distribution of the training sample distribution 120 that the noise ε follows is trained on the emotion recognition model 70. As a result, as shown in FIG. 3B, the number of samples in the training sample distribution 120 is certainly large. However, most of them remain within the training sample distribution 120, and are not considered to cover the entire test sample distribution 122. Therefore, if a test is performed using the test sample distribution 122, it is considered that a sufficient improvement in performance will not be shown.

[0035] The embodiment described below is intended to ensure that the real-world predicted sample distribution 124 represented by the augmented data extends outside the training sample distribution 120 and includes as much of the testing sample distribution 122 as possible, as shown in FIG. 3(C).

[0036] <Embodiment> [composition] 4 shows a block diagram of a classifier training apparatus 150 according to this embodiment. In the classifier training apparatus 150, unlike the training unit 64 in FIG. 2, a decoder 94 separately decodes the noise ε and the representation vector Z output from the encoder 90 to generate a first decoded result X′. ε and the second decoded result Χ′ Z The classifier training device 150 further outputs the first decoded result X′ ε and the second decoded result Χ′ Z2 in that it newly includes an average calculation unit 160 that calculates an extended sample X′ which is the arithmetic mean of the average calculation unit 160 and the sample X that is input to the encoder 90, and a classifier 162 that receives the extended sample X′ output by the average calculation unit 160 and the sample X that is input to the encoder 90, and outputs a label indicating whether the input data is a true sample X (real) or an extended sample X′ (fake) obtained by decoding by the decoder 94, and a label indicating which of a plurality of emotion categories C1 / C2 / ... each data belongs to. The decoder 94 receives a noise ε and an expression vector Z in order.

[0037] Fig. 5 shows the configuration of the encoder 90 shown in Fig. 4. Referring to Fig. 5, the encoder 90 includes a bidirectional GRU layer 182 suitable for processing time-series data, which receives as input a power spectrogram 180 extracted from a speech signal, and a three-layer fully connected layer 184 provided after the bidirectional GRU layer 182. The vector representation of the final hidden layer of the fully connected layer 184 is the representation vector Z.

[0038] In this embodiment, the number of GRU cells in bidirectional GRU layer 182 is 64, and the numbers of units in the three hidden layers are 512, 256, and 128, respectively.

[0039] 6, the decoder 94 includes a simple three-layer fully connected layer 200 that receives the noise ε or the representation vector Z as input. The number of units in each layer is 64, 64, and 38,400. The output of the final layer of the fully connected layer 200 is the reconstructed spectrogram.

[0040] 7, the classifier 92 includes a simple three-layer fully connected layer 210 that receives the noise ε or the representation vector Z as an input, similar to the decoder 94. The number of units in the three fully connected layers 210 is 512, 256, and 1, respectively. The output of the final layer indicates real / fake.

[0041] 8, the classifier 162 includes a three-layer convolutional network 232 that receives as input a power spectrogram 230, which is either a sample X or an extended sample X′, a bidirectional GRU layer 234 that receives the output of the convolutional network 232, and three independent fully connected layers 236, 238, and a fully connected layer 240 that receive the output from the bidirectional GRU layer 234. The number of filters in each layer of the convolutional network 232 is 512, 256, and 128, respectively, and the kernel sizes are 3, 1, and 3, respectively. The stride of each filter is 1. The number of GRU cells in the bidirectional GRU layer 234 is 64.

[0042] The fully connected layers 236, 238, and the fully connected layer 240 are all neural networks consisting of three fully connected layers. The fully connected layer 236 is for discriminating between real and fake, the fully connected layer 238 is for outputting a label indicating arousal level, which is one dimension of emotion, and the fully connected layer 240 is for outputting a label indicating valence, which is another dimension of emotion.

[0043] The numbers of units in the three fully connected layers of fully connected layer 236 are 512, 32, and 1, respectively. In both fully connected layer 238 and fully connected layer 240, the numbers of units in the three fully connected layers are 512, 32, and 3, respectively.

[0044] Fig. 9 shows, in the form of a flowchart, a control structure of a computer program for training the classifier training apparatus 150 shown in Fig. 4. Referring to Fig. 9, this program includes step 300 of repeatedly training the encoder 90 and the discriminator 92 shown in Fig. 4 for N epochs by so-called adversarial learning, step 302 of repeatedly training the encoder 90 and the decoder 94 for N epochs by a normal backpropagation method so that the output of the decoder 94 is equal to the input to the encoder 90 (the error is small), step 304 of repeatedly training the decoder 94 and the classifier 162 for N epochs by adversarial learning, and step 306 of determining whether or not the parameters of each neural network constituting the classifier training apparatus 150 have converged after step 304, and terminating the execution of this program when they have converged, and returning control to step 300 to restart the training of the classifier training apparatus 150 when they have not converged.

[0045] The following formula (1) shows the symbols used in the description of this embodiment.

[0046]

number

[0047]

number

[0048] Simultaneously with such training, a new training sample obtained by extending the real data is obtained as the extended sample X' output by the average calculation unit 160, and the training sample is used to train the classifier 162. After the training is completed, the classifier 162 can be used as a classifier that receives a sample X, which is speech audio data, as shown in Fig. 11, and outputs two types of labels (+ real / fake output for the sample X) that indicate the emotion of the speaker when he / she spoke.

[0049] [Operation] The above-described classifier training device 150 operates as follows: It is assumed that samples of real data have been stored in advance in a storage device.

[0050] In step 300 of FIG. 9, as shown in FIG. 10(A), adversarial learning of the encoder 90 and the discriminator 92 is performed. In this learning, for example, the encoder 90 outputs a representation vector Z for a sample X, and a noise ε obtained from a predetermined distribution or a representation vector Z from the encoder 90 is given to the discriminator 92. In the adversarial learning between the discriminator 92 and the encoder 90, parameters are learned so that the discriminator 92 correctly discriminates the noise ε as real and the representation vector Z as fake, and so that the discriminator 92 erroneously discriminates the representation vector Z as real for the encoder 90. This training is repeated for N epochs, with training using all the training data being one epoch. The parameters of the encoder 90 and the discriminator 92 are updated by this training.

[0051] After the training in step 300 is completed, in step 302 in Fig. 9, the encoder 90 and the decoder 94 are trained so that the error between the output of the decoder 94 and the input to the encoder 90 is reduced, as shown in Fig. 10(B). That is, a noise ε is provided to the decoder 94, which decodes it to obtain a first decoded result X' ε The sample X is then fed to an encoder 90 which encodes it and outputs a representation vector Z. The representation vector Z is then decoded by a decoder 94 to produce a second decoded result X′.Z The mean calculation unit 160 calculates the extended sample X', which is the average of the two. The encoder 90 and the decoder 94 are trained so that the extended sample X' is equal to the sample X (so that the error between the two is small).

[0052] This training is repeated for N epochs, with one epoch being training that uses all the training data. This training updates the parameters of the encoder 90 and the decoder 94. New samples are also generated during this process.

[0053] After the training in step 302 is completed, in step 304 of Fig. 9, the encoder 90, the decoder 94, and the classifier 162 are trained. At this time, the sample X and the extended sample X' are separately provided to the classifier 162. The classifier 162 is trained to correctly determine that the sample X is real when it is input, and that the extended sample X' is fake when it is input, while the encoder 90 and the decoder 94 are trained to maximize the probability that the classifier 162 erroneously determines that the extended sample X' is sample X. This training is repeated for N epochs. As a result, the parameters of the encoder 90, the decoder 94, and the classifier 162 are updated. In this process, in addition to the original sample X, the new extended sample X' generated in step 302 is used as a training sample.

[0054] After this process is executed, in step 306 of Fig. 9, it is determined whether or not the parameters of the encoder 90, discriminator 92, decoder 94, and classifier 162 have converged. If they have converged, the parameters of the encoder 90, discriminator 92, decoder 94, and classifier 162 at this time are stored in a predetermined storage device, and the execution of this program is terminated. If the parameters have not converged, control returns to step 300 again, and the above-mentioned process is further repeated.

[0055] The termination condition may be based on how many times the entire repetition has been performed, rather than on whether or not the parameters have converged.

[0056] After training has been completed in this manner, when voice data is input to the classifier 162 during testing, the classifier 162 outputs two types of labels indicating the emotion of the speaker when the voice was spoken. Note that during training, the classifier 162 is configured to output a label indicating whether the input is real or fake. Therefore, the classifier 162 can also output information indicating whether the input is real or fake during testing, but this information itself is meaningless during testing and is not used.

[0057] In the example shown in Fig. 9, the same number of epochs was used for training in steps 300, 302, and 304. However, the present invention is not limited to such an embodiment. Different numbers of epochs may be used, or only one number of epoch may be used that is different from the others.

[0058] Furthermore, the number of cells in the bidirectional GRU layer constituting the encoder 90, discriminator 92, decoder 94, classifier 162, etc., the number of layers in the neural network, and the number of units in each layer are not limited to those described above.

[0059] [Experimental Results] The accuracy of the classifier 162 trained by the classifier training device 150 according to the above embodiment was confirmed by an experiment using the MuSe-Topic dataset introduced in Non-Patent Document 3. For this dataset, two tasks were provided for car promotional videos. The first was domain-specific prediction of 10 classes of emotion, and the second was prediction of 3 classes of valence and arousal. In the experiment, only the task of predicting emotion using only audio from the video was performed.

[0060] The original data was divided into 4207 training samples and 1335 test samples, and the classifier training device 150 described in the above embodiment was trained, and the trained classifier 162 was tested. The classifier 162 was evaluated using a combination of (0.34×) Unweighted Average Recall (UAR) and (0.66×) F1 score in addition to each prediction result. This is the same criterion as the baseline model.

[0061] For preprocessing of the audio signal, we extracted a power spectrogram from each segment of the video signal and used it as a feature for training. The maximum length of each utterance was set to 6.8 seconds. Any utterances longer than that were cut off at 15 seconds, and shorter utterances were padded with zeros. The sampling rate was 16000Hz. A fast Fourier transform of length 1024 and hop length 512 was performed on each frame.

[0062] In the experiment, 10 epochs were repeated in step 300 in FIG. 9, 20 epochs in step 302, and 20 epochs in step 304. The entire process was terminated when the number of repetitions reached 500.

[0063] Summary statistics of the sample data used in the experiment are shown in Table 1 below.

[0064] [Table 1] Various settings were used in the experiments, which are shown in Table 2 below.

[0065] [Table 2] In this table, AAEC denotes the classifier trained in the above embodiment. D and C in the "component" section denote a discriminator and a classifier, respectively. G denotes the encoder-decoder in the above embodiment, G orishows the conventional encoder-decoder shown in FIG. 2. AAE_C is a result of deleting the discriminator D in order to confirm whether the discriminator D is unnecessary with only the classifier C. AAE+C is a result of confirming whether the performance of the classifier C is improved by the adversarial learning for the classifier C, in which the classifier C is removed during repeated learning, and the classifier C is trained after the end of the repeated learning. AAEC' is a result of comparing the effect of the encoder-decoder of the above embodiment with the conventional encoder-decoder shown in FIG. 2. In addition, in order to confirm the effect of AAEC, the result of training only the classifier C is shown by "Classifier".

[0066] Table 2 shows the results reported in Non-Patent Document 3 as a baseline. The baseline shows the results of emotion recognition by SVM using only audio, emotion recognition by a neural network using LSTM and self-attention (LSTM+self-ATT) using only audio, and emotion recognition by a multimodal model (MMT) using audio + image + text. Note that "MMT" is an abbreviation for "Multimodal Transformer".

[0067] In Table 2, the numbers in bold indicate the highest performance. Referring to Table 2, it can be seen that the classifier trained using the adversarial learning according to the above embodiment, or a similar adversarial learning, shows higher performance than the classifier alone. The classifier trained using the augmented data (AAE+C) also shows a 2.6% improvement in performance compared to the classifier alone. In addition, comparing the results of AAEC and AAE+C, AAEC shows a 2.2% improvement in performance, indicating that the adversarial learning according to the above embodiment shows high effectiveness.

[0068] Furthermore, when comparing the results of AAEC and AAE_C, the model incorporating the discriminator shows an improvement of about 0.8% in performance in terms of the combination score. This result shows that although the discriminator and the classifier partially overlap, it is better to have a discriminator.

[0069] From the comparison between AAEC and AAEC', as shown in the above embodiment, the noise ε and the representation vector Z are input to the decoder separately, and the first decoded result X' is obtained. ε and the second decoded result X′ Z We can see that the performance is improved by averaging the extended samples Χ′. The value (40.33%) is about 1.5% higher than the performance of AAEC′ (38.31%).

[0070] The decoder according to the above embodiment randomly generates samples at 10, 100, and 500 training iterations, and the distribution of the original data and the extended data is visualized by principal component analysis in Figures 12, 13, and 14, respectively. These figures are a little difficult to understand because they are in black and white, but the inverted Y-shaped symbols indicate the original samples, the X-shaped symbols indicate the test samples, and the +-shaped symbols indicate the extended samples (the darkest areas in the figures).

[0071] As is clear from these figures, the original training samples do not completely cover the test samples. This is as described above as a problem of the prior art. At the beginning of the training iteration, the expanded samples are generated in a region that overlaps with the original training samples to a large extent, but as the number of iterations increases, the expanded samples are generated more outside the area of ​​the training samples. As a result, it was confirmed that the expanded samples generated by the above-mentioned embodiment are closer to the distribution of real-world data, and therefore a classifier trained using such expanded samples shows better performance than a classifier trained using only the original training samples.

[0072] [Computer implementation] Fig. 15 is an external view of a computer system for implementing each of the above-described embodiments, and Fig. 16 is a hardware block diagram of the computer system shown in Fig. 15.

[0073] 15, this computer system 950 includes a computer 970 having a DVD drive 1002, and a keyboard 974, a mouse 976, and a monitor 972 for interacting with a user, all of which are connected to the computer 970. Of course, these are just one example of a configuration for when user interaction is required, and any general hardware and software available for user interaction (e.g., a touch panel, voice input, or a general pointing device) can be used.

[0074] 16, the computer 970 includes, in addition to the DVD drive 1002, a CPU 990, a GPU 992, a bus 1010 connected to the CPU 990, the GPU 992, and the DVD drive 1002, a ROM 996 connected to the bus 1010 and storing a boot-up program of the computer 970, a RAM 998 connected to the bus 1010 and storing instructions constituting a program, a system program, working data, and the like, and a HDD 1000 which is a non-volatile memory connected to the bus 1010. The HDD 1000 is for storing programs executed by the CPU 990 and the GPU 992, and data used by the programs executed by the CPU 990 and the GPU 992, and the like. The computer 970 further includes a network I / F 1008 which provides a connection to a network 986 which enables communication with other terminals, and a USB port 1006 to which a USB memory 984 can be attached and detached and which provides communication between the USB memory 984 and each part in the computer 970.

[0075] The computer 970 further includes an audio I / F 1004 that is connected to the microphone 982, the speaker 980, and the bus 1010, and that reads out audio signals, video signals, and text data generated by the CPU 990 and stored in the RAM 998 or the HDD 1000 in accordance with instructions from the CPU 990, performs analog conversion and amplification processing to drive the speaker 980, and digitizes the analog audio signal from the microphone 982 and stores it in any address of the RAM 998 or the HDD 1000 specified by the CPU 990.

[0076] In the above embodiment, the encoder 90, discriminator 92, decoder 94, and classifier 162 shown in Fig. 4, as well as data and parameters such as speech signals, video signals, and text data, are all stored in, for example, the HDD 1000, RAM 998, DVD 978, or USB memory 984 shown in Fig. 16, or in a storage medium of an external device (not shown) connected via the network I / F 1008 and network 986. Typically, these data, parameters, and the like are written into the HDD 1000 from outside, for example, and loaded into the RAM 998 when the computer 970 is executed.

[0077] 4, the classifier training device 150 that trains them by adversarial learning, the emotion recognition device using the trained classifier 162, and a computer program for operating this computer system to realize the functions of each of its components are stored in a DVD 978 that is inserted into a DVD drive 1002, and transferred from the DVD drive 1002 to the HDD 1000. Alternatively, these programs are stored in a USB memory 984, and the USB memory 984 is inserted into a USB port 1006 and the programs are transferred to the HDD 1000. Alternatively, the programs may be transmitted to the computer 970 via a network 986 and stored in the HDD 1000.

[0078] The program is loaded into RAM 998 at the time of execution. Of course, a source program may be input using keyboard 974, monitor 972, and mouse 976, and the compiled object program may be stored in HDD 1000. In the case of a script language, a script input using keyboard 974 or the like may be stored in HDD 1000. In the case of a program that runs on a virtual machine, a program that functions as a virtual machine must be installed in computer 970 in advance. However, since training using classifier training device 150 involves a large amount of calculation, it is preferable to realize each part of the embodiment of the present invention as an object program consisting of native computer code rather than a script language.

[0079] The CPU 990 reads a program from the RAM 998 according to an address indicated by an internal register called a program counter (not shown), interprets the instruction, reads data required for executing the instruction from the RAM 998, HDD 1000, or other device according to an address specified by the instruction, and executes the process specified by the instruction. The CPU 990 stores the execution result data in an address specified by the program, such as the RAM 998, HDD 1000, or a register in the CPU 990. At this time, the value of the program counter is also updated by the program. The computer program may be directly loaded into the RAM 998 from the DVD 978, the USB memory 984, or via a network. Note that some tasks (mainly numerical calculations) among the programs executed by the CPU 990 are dispatched to the GPU 992 according to instructions included in the program or according to the analysis results obtained when the CPU 990 executes the instructions.

[0080] The program for implementing the functions of each unit according to each embodiment described above by the computer 970 includes a plurality of instructions written and arranged to operate the computer 970 to implement those functions. Some of the basic functions required to execute the instructions are provided by an operating system (OS) or a third-party program running on the computer 970, or by modules of various tool kits installed on the computer 970. Thus, the program does not necessarily include all of the functions required to implement the system and method of this embodiment. The program may include only instructions to execute the operations of each of the above-mentioned devices and their components by calling appropriate functions or functions of a "programming tool kit" in a controlled manner to obtain a desired result. The method of operating the computer 970 for that purpose is well known, and will not be repeated here.

[0081] The GPU 992 is capable of parallel processing, and can simultaneously execute a large amount of calculations associated with machine learning in parallel or in a pipelined manner. For example, parallel calculation elements found in a program when the program is compiled, or parallel calculation elements found when the program is executed, are dispatched from the CPU 990 to the GPU 992 as needed, and executed, and the results are returned to the CPU 990 directly or via a predetermined address in the RAM 998, and assigned to a predetermined variable in the program.

[0082] The embodiments disclosed herein are merely illustrative, and the present invention is not limited to the above-described embodiments. The scope of the present invention is defined by the claims of the appended claims, taking into consideration the detailed description of the invention, and includes all modifications within the scope and meaning equivalent to the words described therein. [Explanation of symbols]

[0083] 50 Emotion recognition model training system 60 Actual Data Storage Device 62 Generative Model 64, 68 Training Department 66 Extended Sample Memory 70 Emotion Recognition Model 72 Audio Data 74 Emotion Labels 90 Encoder 92 Discriminator 94 Decoder 120 Distribution of training samples 122 Distribution of test samples 124 Expected real-world sample distributions 150 Classifier training device 160 Average calculation section 162 Classifier 180, 230 Power Spectrogram 182, 234 bidirectional GRU layer 184, 200, 210, 236, 238, 240 fully connected layers 232 Convolutional Networks

Claims

1. A computer training an encoder to generate data from an input according to a predetermined distribution; training the decoder by a computer to minimize an error, relative to the first training data, of new samples for decoder training generated from the results of decoding by the decoder of samples from the predetermined distribution and the results of decoding, using the decoder, of the output of the encoder relative to first training data; training a classifier to maximize the probability of correctly classifying new samples for training a classifier generated from the result of decoding a sample from the predetermined distribution by the decoder and the result of decoding an output of the encoder for second training data using the decoder, and training the encoder and the decoder to maximize the probability of incorrect classification by the classifier; A method for training a classifier, wherein each of the samples from the predetermined distribution and the first training data is assigned one of a plurality of types of teacher labels.

2. 2. The method for training a classifier according to claim 1 , wherein each of the predetermined distribution, the first training data, and the second training data includes speech voice data, and the plurality of types of teacher labels are labels indicating emotions of a speaker represented by the speech voice data.

3. 3. The method for training a classifier according to claim 1, wherein the step of training the encoder includes a step of: training the predetermined classifier by a computer to maximize a probability that a predetermined classifier correctly classifies a sample from the predetermined distribution and an output of the encoder for the first training data, and training the encoder to maximize a probability that the predetermined classifier erroneously classifies the sample.

4. The step of training the decoder comprises: a computer decoding the samples from the predetermined distribution with the decoder to obtain a first decoded result; A computer decoding an output of the encoder for the first training data with the decoder to obtain a second decoded result; generating new decoded samples for training the classifier by performing a predetermined operation between the first decoded result and the second decoded result by a computer; 4. The method for training a classifier according to claim 1, further comprising: updating parameters of the decoder by a backpropagation method so that an error between a new decoded sample for training the classifier and the first training data is reduced.

5. 5. The method of training a classifier of claim 4, wherein the step of training the decoder further comprises a step of: a computer repeating the steps of obtaining the first decoding result, obtaining the second decoding result, and updating parameters of the decoder until a predetermined termination condition is met.

6. 6. The method for training a classifier according to claim 4 or 5, wherein the step of generating a new decoded sample comprises a step of generating the new decoded sample by a computer by calculating an average of the first decoded result and the second decoded result.

7. The step of training the classifier comprises: A computer decoding the samples from the predetermined distribution with the decoder to obtain a first decoded result for training a classifier; A computer decoding, by the decoder, an output of the encoder for the second training data to obtain a second decoded result for training a classifier; A computer generates a new decoded sample for training the classifier by performing a predetermined operation between the first decoded result for training the classifier and the second decoded result for training the classifier; A computer classifies the second training data or the new decoded sample for training the classifier using the classifier to obtain a classification result, for the second training data or the new decoded sample for training the classifier; The method for training a classifier according to any one of claims 1 to 6, further comprising: updating, by a computer, parameters of the classifier by a backpropagation method using an error between the classification result and a correct classification so that the error is reduced.

8. 8. The method for training a classifier according to claim 7, wherein the step of training the classifier further includes a step of: by a computer repeating the steps of obtaining a first decoding result for training the classifier, obtaining a second decoding result for training the classifier, and updating parameters of the classifier until a predetermined termination condition is met.

9. 9. The method for training a classifier according to claim 7 or 8, wherein the step of generating new decoded samples for training the classifier includes a step of generating new decoded samples for training the classifier by calculating an average of a first decoded result for training the classifier and a second decoded result for training the classifier.

10. A computer program product causing a computer to carry out the training method according to any one of claims 1 to 9.

11. an encoder training means for training the encoder to generate data from an input according to a predetermined distribution; a decoder training means for training the decoder so as to minimize an error, relative to the first training data, of new samples for decoder training generated from a result of decoding by the decoder of samples from the predetermined distribution and a result of decoding, using the decoder, of an output of the encoder relative to first training data; a classifier training means for training a classifier to maximize a probability of correctly classifying new samples for classifier training generated from a result of decoding a sample from the predetermined distribution by the decoder and a result of decoding an output of the encoder for second training data using the decoder, and training the encoder and the decoder to maximize a probability of incorrect classification by the classifier; A classifier training apparatus, wherein each of the samples from the predetermined distribution and the first training data is assigned one of a plurality of types of teacher labels.

Images