Generation apparatus, generation method, and generation program

Abstract

To enable the recognition of emotions contained in spoken speech from the tone of the speech, even if the speech does not contain any content related to those emotions. [Solution] The generation device includes a storage unit that stores a set of training datasets which are combinations of speech signals representing spoken speech and text representing positive or negative expressions of emotion; a speech coding unit that encodes the speech signals and generates speech feature vectors based on first learning parameters; a language coding unit that encodes the text and generates language feature vectors based on second learning parameters; and an update unit that updates the first and second learning parameters by comparative learning using a combination of the speech feature vectors generated by the speech coding unit and the language feature vectors generated by the language coding unit.

Description

Technical Field

[0001] The present invention relates to a generation device, a generation method, and a generation program for generating data.

Background Art

[0002] In a wide range of industries such as contact centers, retail / counter services, and nursing care, labor shortages are becoming serious due to the declining birthrate and aging population, as well as severe labor such as emotional labor. There is a growing expectation that advanced and standardized customer service can be achieved by voice emotion recognition technology, and customer satisfaction can be improved efficiently. For example, in contact centers, retail services, counter services, etc., it is required to efficiently improve the quality of staff responses. Conventionally, improvements have been made based on experience and intuition. However, if the emotion of a customer can be automatically recognized from voice by voice emotion recognition, the recognition result can be presented to the staff to prompt improvement of the response.

[0003]

[0004] Non-patent document 1 states, "The predicted emotional state of the utterance i for the dimension k is modeled by the linear combination of standardized acoustic parameters of the utterance i (x_i1k . . . x_i6k) as follows: y_ik = Σ _j∈FS β_jk x_ijk + ε_ik, (1) It states that "where x_ijk is one of the six acoustic features (e.g., f0 range, f0 slope, ...) that is normalized by subtracting its mean and dividing by its standard deviation." [Prior art documents] [Patent Documents]

[0005] [Patent Document 1] Japanese Patent Publication No. 2022-65309 [Non-patent literature]

[0006] [Non-Patent Document 1] H. Mori, T. Satake, M. Nakamura, and H. Kasuya, “Constructing a spoken dialogue corpus for studying paralinguistic information in expressive conversation and analyzing its statistical / acoustic characteristics,” Speech Communication, vol. 53, no. 1, pp. 36-50, 2011. [Overview of the Initiative] [Problems that the invention aims to solve]

[0007] However, the emotions that customers want to recognize vary, and often it is necessary to recognize new emotions that are not included in the categories predetermined during the training of the emotion recognition model. In such cases, Non-Patent Literature 1 states that it is necessary to prepare a new emotion recognition model corresponding to the newly added category and retrain that model. Therefore, it is necessary to collect new training data consisting of the label of the newly added emotion category and the corresponding audio signal pair.

[0008] The present invention aims to enable the recognition of emotions contained in speech from the tone of speech, even if the speech itself does not contain any emotional content. [Means for solving the problem]

[0009] A generating device representing one aspect of the invention disclosed in this application is characterized by comprising: a storage unit that stores a set of training datasets which are combinations of speech signals representing spoken speech and text representing positive or negative expressions of emotion; a speech coding unit that encodes the speech signals and generates speech feature vectors based on first learning parameters; a language coding unit that encodes the text and generates language feature vectors based on second learning parameters; and an update unit that updates the first and second learning parameters by comparative learning using a combination of the speech feature vectors generated by the speech coding unit and the language feature vectors generated by the language coding unit. [Effects of the Invention]

[0010] According to a typical embodiment of the present invention, even if the speech does not contain any emotional content, it is possible to recognize the emotion contained in the speech from the tone of the speech. Problems, configurations, and effects other than those described above will be clarified by the following description of the embodiments. [Brief explanation of the drawing]

[0011] [Figure 1] Figure 1 is a block diagram showing an example of the hardware configuration of the generation device. [Figure 2] Figure 2 is a block diagram showing an example of the functional configuration of the generating device. [Figure 3] Figure 3 is a block diagram showing an example of the functional configuration of the learning unit. [Figure 4] Figure 4 is a flowchart showing an example of the learning process procedure in the learning unit. [Figure 5] Figure 5 is an explanatory diagram showing an example of comparative learning by the update unit (step S408). [Figure 6] Figure 6 is a block diagram showing an example of the functional configuration of the generation unit. [Figure 7] Figure 7 is a flowchart showing an example of the generation process procedure in the generation unit. [Modes for carrying out the invention]

[0012] <Figure 1: Example of hardware configuration of the generation device> Figure 1 is a block diagram showing an example of the hardware configuration of a generation device. The generation device 100 includes a processor 101, a storage device 102, an input device 103, an output device 104, and a communication interface (communication IF) 105. The processor 101, storage device 102, input device 103, output device 104, and communication IF 105 are connected by a bus 106. The processor 101 controls the generation device 100. The storage device 102 serves as the work area for the processor 101. The storage device 102 is a non-temporary or temporary recording medium that stores various programs and data. Examples of storage devices 102 include ROM (Read Only Memory), RAM (Random Access Memory), HDD (Hard Disk Drive), and flash memory. The input device 103 inputs data. Examples of input devices 103 include a keyboard, mouse, touch panel, numeric keypad, scanner, microphone, and sensor. The output device 104 outputs data. Output devices 104 include, for example, a display, a printer, and a speaker. The communication interface 105 connects to a network and sends and receives data.

[0013] <Figure 2: Example of Functional Configuration of Generation Device 100> Figure 2 is a block diagram showing an example of the functional configuration of the generation device 100. The generation device 100 includes a training dataset DB 201, a learning unit 202, a neural network model 203, and an inference unit 204. Specifically, the training dataset DB 201 is stored, for example, in the storage device 102 shown in Figure 1, or in another computer capable of communicating with the generation device 100. Specifically, the learning unit 202, the neural network model 203, and the inference unit 204 are implemented, for example, by having the processor 101 execute a program stored in the storage device 102 shown in Figure 1.

[0014] The training dataset DB201 is a storage unit that stores one or more training datasets. A training dataset is a combination of training data and correct answer data. The training dataset DB201 stores training datasets DS1 to DSn. When not distinguishing between the training datasets DS1 to DSn, they are denoted as training dataset DSi (where i is any one of 1 to n).

[0015] The training datasets DS1 to DSn have training signal time waveform sets T1 to Tn and explanatory texts G1 to Gn. When not distinguishing between the training signal time waveform sets T1 to Tn, they are denoted as training signal time waveform set Ti. When not distinguishing between the explanatory texts G1 to Gn, they are denoted as explanatory text Gi. The training signal time waveform set Ti is a set of training signal time waveforms. A training signal time waveform is training data indicating the time waveform of a training signal. A training signal is a signal of human speech.

[0016] The explanatory text Gi is text that affirms or negates human emotion. Examples of text that affirms human emotion include, for example, "I am feeling comfortable", and examples of text that negates human emotion include "I am feeling uncomfortable".

[0017] Note that as the explanatory text Gi included in the training signal time waveform set Ti, for example, it is considered that emotion can be expressed in the following 6 axes proposed in Non-Patent Document 1.

[0018] Unpleasant-pleasant Sleepy-aroused Submissive-dominant Doubtful-credible Indifferent-interested Negative-positive

[0019] And the following texts representing a total of 12 corresponding to the positive and negative two directions of each of the above 6 axes are used.

[0020] Unpleasant-pleasant: "I feel uncomfortable," "I feel comfortable." Sleepy-aroused: "I feel sleepy," "I feel awake." Submissive-dominant: "I feel submissive," "I feel dominant." Doubtful-credible: "I am skeptical," "I trust." Indifferent-interested: "I am indifferent," "I am interested." Negative-positive: "I am negative," "I am positive"

[0021] The training signal time waveform set Ti contains at least one training signal time waveform corresponding to these 12. Note that these descriptive sentences are not sentences that the speaker actually speaks aloud, but rather what is called the "inner voice" that the speaker does not actually utter.

[0022] The training signal time waveform set Ti and the explanatory text Gi are used as the ground truth data for the neural network model 203. The neural network model 203 is a model consisting of a neural network that accepts the training signal and explanatory text as inputs, outputs an audio feature vector from the training signal, and outputs a linguistic feature vector from the explanatory text.

[0023] If the input description is a description that expresses the emotion contained in the tone of the audio of the input training signal, the learning parameters of the neural network model 203 are learned so that the cosine similarity between the audio feature vector output by the neural network model 203 and the language feature vector approaches 1.

[0024] On the other hand, if the input description is a description that negates the emotion contained in the tone of the input training signal, the learning parameters of the neural network model 203 are learned so that the cosine similarity between the speech feature vector output by the neural network model 203 and the language feature vector approaches -1.

[0025] If the input description is neither an emotion contained in the tone of the input training signal nor a description expressing its negation, the learning parameters of the neural network model 203 are learned such that the cosine similarity between the speech feature vector output by the neural network model 203 and the language feature vector approaches 0.

[0026] As a result of the above learning, the following inferences regarding speech emotion recognition become possible.

[0027] Inference Example 1: That is, when human speech is input to the neural network model 203, the neural network model 203 outputs a speech feature vector. Also, when the neural network model 203 is given the aforementioned 12 texts, the neural network model 203 outputs a language feature vector corresponding to each text. Of these 12 language feature vectors, the text corresponding to the language feature vector with the highest cosine similarity (closest to 1) to the aforementioned speech feature vector can be considered to be an explanatory text that expresses the emotion contained in the tone of that speech. Therefore, it is possible to recognize what emotion a person's speech expresses based solely on the tone of the speech, even if the content related to that emotion is not included in the speech.

[0028] Inference Example 2: Another important point is that during inference, you can also provide any other text that was not included in the 12 texts used during training as described above. For example, if you input "I want to buy" and "I don't want to buy," which represent purchasing intent, two linguistic feature vectors corresponding to each text will be output. If the cosine similarity between the linguistic feature vector corresponding to "I want to buy" and the speech feature vector is higher, it is considered that purchasing intent is high. If the cosine similarity between the linguistic feature vector corresponding to "I don't want to buy" and the speech feature vector is higher, it is considered that purchasing intent is low. In this way, even if an emotion category was not defined during training, it can be recognized without needing to retrain the model.

[0029] The learning unit 202 randomly selects the training dataset DSi from among the n training datasets DS1 to DSn. As described above, the training dataset DSi consists of a training signal time waveform set Ti and an explanatory text Gi. Furthermore, the learning unit 202 randomly selects one element from the training signal time waveform set Ti of the training dataset DSi and sets it as the training signal time waveform 300 as an explanatory variable, and sets the explanatory text Gi as the target variable.

[0030] The learning unit 202 trains the neural network model 203. Specifically, for example, the learning unit 202 calculates the value of the loss function based on the difference between the output data, which is produced as a result of inputting the set explanatory variables into the neural network model 203, and the explanatory text Gi, and updates the learning parameters of the neural network model 203 so that the value of the loss function is minimized.

[0031] The inference unit 204, upon inputting the recognition target signal time waveform 241, the recognition target emotion affirmative paraphrase text, and the recognition target emotion negative paraphrase text, outputs the emotion recognition result 244 obtained from the neural network model 203. The recognition target signal time waveform 241 is a waveform representing the recognition target speech signal. The recognition target is, for example, the source of the speech signal.

[0032] The signal time waveform 241 to be recognized may also be the training signal time waveform 300. The paraphrased text for affirming the emotion to be recognized is text that affirms the emotion to be recognized. The paraphrased text for denying the emotion to be recognized is text that denies the emotion to be recognized. As in the inference example 1 above, the paraphrased text for affirming the emotion to be recognized and the paraphrased text for denying the emotion to be recognized may also be descriptive sentences Gi included in the training dataset, such as "I feel uncomfortable" and "I feel comfortable." Alternatively, they may be descriptive sentences not included in the training dataset, such as "I want to buy" and "I don't want to buy" in the inference example 2 above.

[0033] <Figure 3: Example of functional configuration of the learning unit 202> Figure 3 is a block diagram showing an example of the functional configuration of the learning unit 202. The learning unit 202 includes a frame division unit 301, a window function multiplication unit 302, a frequency domain signal generation unit 303, a speech coding unit 304, a language coding unit 311, and an update unit 320.

[0034] The frame division unit 301 divides the training signal time waveform 300 into frames. Each of the divided training signal time waveforms is referred to as a frame division signal.

[0035] The window function multiplication unit 302 performs window function multiplication on the frame division signal, converting each of the frame division signals into a window function multiplied signal.

[0036] The frequency domain signal generation unit 303 performs a short-time Fourier transform on each of the window function multiplied signals to convert them into time-frequency domain signals. The frequency domain signal generation unit 303 can also use frequency transformation methods such as the constant Q transform (CQT) instead of the short-time Fourier transform.

[0037] The speech coding unit 304 calculates a feature vector (hereinafter referred to as the speech feature vector) based on the frequency domain signal. The speech coding unit 304 is typically a neural network encoder with multiple convolutional layers, activation functions, and pooling layers stacked and skip connections in between. Alternatively, the speech coding unit 304 may be a recurrent neural network having layers such as a known Transformer model, Long-Short-Term-Memory (LSTM), bidirectional LSTM, Gated recurrent unit (GRU), or bidirectional GRU.

[0038] The language coding unit 311 calculates the feature vector of the explanatory text Gi (hereinafter referred to as the language feature vector). The language coding unit 311 is an encoder similar to the speech coding unit 304. The language coding unit 311 calculates the feature vector of the explanatory text Gi by embedding.

[0039] In Figure 3, for the sake of explanation, the speech coding unit 304 and the language coding unit 311 are shown separately from the neural network model 203. However, the speech coding unit 304 and the language coding unit 311 constitute the neural network model 203. Note that the dimensionality of the speech feature vector and the language feature vector are the same.

[0040] The update unit 320 learns the neural network model 203 based on the speech feature vector from the speech coding unit 304 and the language feature vector from the language coding unit 311. In other words, the update unit 320 updates the learning parameters of the speech coding unit 304 and the language coding unit 311.

[0041] Specifically, for example, the update unit 320 uses the contrast loss calculated from a contrast matrix consisting of the cosine similarity between the speech feature vector and the language feature vector as the loss function, and updates the learning parameters of the speech coding unit 304 and the language coding unit 311 so that the loss function becomes smaller, that is, so that the diagonal terms of the contrast matrix become large and the off-diagonal terms become small.

[0042] <Figure 4: Learning process procedure of learning unit 202> Figure 4 is a flowchart showing an example of the learning process procedure of the learning unit 202.

[0043] (Step S401) The update unit 320 determines whether the value of the loss function converges. Specifically, for example, the update unit 320 determines whether the convergence condition is met, or whether the number of iterations C1 is greater than the threshold ThC. The convergence condition is, for example, the condition that the convergence determination function becomes smaller than a predetermined threshold.

[0044] If the convergence conditions are not met, and the number of iterations C1 does not exceed the threshold ThC (step S401: No), proceed to step S402. If the convergence conditions are met, or if the number of iterations C1 exceeds the threshold ThC (step S401: Yes), it is determined that the value of the loss function has converged, and proceed to step S419.

[0045] (Step S402) The learning unit 202 randomly selects the training dataset DSi from the training dataset DB201. Furthermore, the learning unit 202 randomly selects one element from the training signal time waveform set T1 and sets it as the training signal time waveform 300 as an explanatory variable, and sets the explanatory text Gi as the target variable.

[0046] (Step S403) The frame division unit 301 divides the training signal time waveform 300 into frames.

[0047] (Step S404) The window function multiplication unit 302 performs window function multiplication on the frame division signal, converting each of the frame division signals into a window function multiplied signal.

[0048] (Step S405) The frequency domain signal generation unit 303 performs a short-time Fourier transform on each of the window function multiplied signals to convert them into time-frequency domain signals.

[0049] (Step S406) The speech coding unit 304 calculates a speech feature vector based on the frequency domain signal.

[0050] (Step S407) The language coding unit 311 calculates a language feature vector based on the explanatory text Gi.

[0051] (Step S408) The update unit 320 updates the learning parameters of the neural networks of the speech coding unit 304 and the language coding unit 311 by comparative learning of speech feature vectors and language feature vectors.

[0052] (Step S409) The update unit 320 calculates the convergence conditions.

[0053] (Step S410) The update unit 320 increments the iteration count C1. Then, it returns to step S410.

[0054] (Step S411) Step S404: If Yes, the update unit 320 stores the learning parameters updated in step S408 as parameters for the neural network model 203 in the memory device 102.

[0055] <Figure 5: Comparative learning by the update unit 320 (step S408)> Figure 5 is an explanatory diagram showing an example of comparative learning (step S408) by the update unit 320. In Figure 5, the speech feature vectors Va1 to Van are feature vectors generated by the speech coding unit 304 from different training signal time waveforms 300. If the speech feature vectors Va1 to Van are not distinguished, they are referred to as speech feature vector Va. The language feature vectors Vt1 to Vtn are feature vectors generated by the language coding unit 311 from explanatory sentences G1 to Gn. If the language feature vectors Vt1 to Vtn are not shown, they are referred to as language feature vector Vt.

[0056] The contrast matrix 500 is an n x n square matrix. The training signal time waveform 300, which is the source of the i-th speech feature vector Vai, is a spoken speech that expresses an emotion corresponding to the affirmation or negation of the emotion indicated by the descriptive sentence Gi, which is the source of the i-th language feature vector Vti.

[0057] Each element of the symmetric matrix 500 represents the similarity between the speech feature vector Va and the language feature vector Vt. The similarity is expressed as the Euclidean distance or cosine similarity between the speech feature vector Va and the language feature vector Vt. A smaller Euclidean distance and a cosine similarity closer to 1 indicate a higher similarity between the speech feature vector Va and the language feature vector Vt.

[0058] The update unit 320 updates the learning parameters of the speech coding unit 304 and the language coding unit 311 so that the similarity between the speech feature vector Vai and the language feature vector Vti, indicated by the diagonal component Mii of the contrast matrix 500, is increased, and the similarity between the speech feature vector Vai and the language feature vector Vti, indicated by the off-diagonal component Mjk (where j and k are integers from 1 to n, excluding (j,k)=(i,i)), is decreased.

[0059] <Figure 6: Example of Functional Configuration of Inference Unit 204> Figure 6 is a block diagram showing an example of the functional configuration of the inference unit 204. The inference unit 204 includes a frame division unit 601, a window function multiplication unit 602, a frequency domain signal generation unit 603, a speech coding unit 604, a generation unit 610, language coding units 611, 621, average distance calculation units 612, 622, and a classification unit 630.

[0060] The frame division unit 601, window function multiplication unit 602, frequency domain signal generation unit 603, speech coding unit 604, and language coding units 611 and 621 have the same configuration as the frame division unit 301, window function multiplication unit 302, frequency domain signal generation unit 303, speech coding unit 304, and language coding unit 311 of the learning unit 202.

[0061] The speech coding unit 604 is set with the learning parameters updated by the speech coding unit 304. The language coding units 611 and 621 are set with the learning parameters updated by the language coding unit 311. For the sake of explanation, the speech coding unit 604 and the speech coding unit 304 are shown as having separate configurations, but the speech coding unit 604 may be the same as the speech coding unit 304.

[0062] The generation unit 610 generates a paraphrased text of the recognized emotion from the recognized emotion text 600, which is a paraphrased expression of the recognized emotion text 600 in a positive sense, and a paraphrased text of the recognized emotion, which is a paraphrased expression of the recognized emotion text 600 in a negative sense. The recognized emotion text 600 is text that expresses the emotion to be recognized (for example, the desire to buy).

[0063] For example, if the recognized emotion text 600 is "desire to buy," then the group of texts 242 that affirm the recognized emotion would be "I want to buy it," "I want it," "I want to get it," "The price is cheap," etc., while the group of texts 243 that negate the recognized emotion would be "I don't want to buy it," "I don't need it," "The price is high," etc.

[0064] The language coding unit 611 sets updated learning parameters based on the language feature vector Vti of the texts with positive expressions among the diagonal components Mii. The language coding unit 611 calculates a language feature vector for each of the target emotion-affirming paraphrase texts in the group of target emotion-affirming paraphrase texts 242.

[0065] The language coding unit 621 sets updated learning parameters based on the language feature vector Vti of the negative expression text among the diagonal components Mii. The language coding unit 621 calculates a language feature vector for each of the negative emotion paraphrase texts in the group of negative emotion paraphrase texts 243.

[0066] The average distance calculation unit 612 calculates the Euclidean distance between the speech feature vector output from the speech coding unit 604 and the language feature vector output from the language coding unit 611 for each combination of speech feature vectors and language feature vectors. Then, the average distance calculation unit 612 calculates the average value of the multiple Euclidean distances obtained for each combination.

[0067] The average distance calculation unit 622 calculates the Euclidean distance between the speech feature vector output from the speech coding unit 604 and the language feature vector output from the language coding unit 611 for each combination of speech feature vectors and language feature vectors. Then, the average distance calculation unit 622 calculates the average value of the multiple Euclidean distances obtained for each combination.

[0068] The classification unit 630, based on the average distance output from the average distance calculation unit 612 and the average distance output from the average distance calculation unit 622, classifies the emotion of the utterance expressed in the recognition target signal into either an utterance that affirms the recognition target emotion or an utterance that denies the recognition target emotion, and outputs an emotion recognition result 244. The emotion recognition result is determined, for example, based on the calculation result Logit of the following formula (1).

[0069] Logit=exp(-d_pos) / exp(-d_neg)···(1)

[0070] d_pos is the average distance output from the average distance calculation unit 612, and d_neg is the average distance output from the average distance calculation unit 622. If Logit is greater than or equal to a pre-set threshold, the emotion recognition result 244 indicates that the utterance expressed by the recognition target signal has been classified as an utterance that affirms the recognition target emotion. If it is less than the threshold, the emotion recognition result 244 indicates that the utterance expressed by the recognition target signal has been classified as an utterance that denies the recognition target emotion.

[0071] Furthermore, not limited to the above formula (1), the classification unit 630 may, if d_pos is greater than or equal to d_neg, provide content indicating that the utterance expressed by the recognition target signal has been classified as an utterance that affirms the recognition target emotion, and if d_pos is less than d_neg, provide content indicating that the emotion recognition result 244 has been classified as an utterance that negates the recognition target emotion.

[0072] Furthermore, although the above explanation uses Euclidean distance in equation (1), cosine similarity may also be used. In other words, regardless of whether Euclidean distance or cosine similarity is used, the classification unit 630 should classify the expression into the one with the higher similarity between the positive and negative expressions.

[0073] <Figure 7: Generation process procedure of the inference unit 204> Figure 7 is a flowchart showing an example of the generation process procedure of the inference unit 204.

[0074] (Step S701) The inference unit 204 reads out the neural network model 203.

[0075] (Steps S702~S705) The inference unit 204 performs the same processing as in steps S403 to S406 on the input recognition target signal time waveform 241.

[0076] (Step S706) The inference unit 204 generates a group of paraphrased texts 242 that affirm the target emotion and a group of paraphrased texts 243 that deny the target emotion from the target emotion text 600 using the generation unit 610. If the group of paraphrased texts 242 that affirm the target emotion and the group of paraphrased texts 243 that deny the target emotion have been prepared in advance, step S706 is unnecessary.

[0077] (Step S707) The inference unit 204 generates a language feature vector for each of the emotion-affirming paraphrase texts in the group of emotion-affirming paraphrase texts 242, using the language coding unit 611.

[0078] (Step S708) The inference unit 204 calculates the Euclidean distance between the speech feature vector of the signal to be recognized from the speech coding unit 604 and the language feature vector of the emotion-affirming paraphrase text to be recognized, using the average distance calculation unit 612. It then calculates the average distance d_pos, which is the average value of the Euclidean distances calculated for each combination of the speech feature vector and the emotion-affirming paraphrase text to be recognized.

[0079] (Step S709) The inference unit 204 generates a language feature vector for each of the emotion-negating paraphrase texts in the group 243 of emotion-negating paraphrase texts, using the language encoding unit 621.

[0080] (Step S710) The inference unit 204 calculates the Euclidean distance between the speech feature vector of the signal to be recognized from the speech coding unit 604 and the linguistic feature vector of the emotion-negating paraphrase text to be recognized, using the average distance calculation unit 622. It then calculates the average distance d_neg, which is the average value of the Euclidean distances calculated for each combination of the speech feature vector and the emotion-negating paraphrase text to be recognized.

[0081] (Step S711) The inference unit 204, based on the average distance d_pos output from the average distance calculation unit 612 and the average distance d_neg output from the average distance calculation unit 622, classifies the emotion of the utterance represented by the recognition target signal into either an utterance that affirms the recognition target emotion or an utterance that denies the recognition target emotion, and outputs an emotion recognition result 244.

[0082] Thus, the above-described generation device 100 makes it possible to recognize what emotion a spoken voice expresses based solely on the tone of the voice, even if the content relating to that emotion is not included in the speech.

[0083] In other words, it is possible to generate a neural network model 203 that can recognize what emotion an utterance expresses based solely on the tone of the speech, even if the content relating to that emotion is not included in the utterance. Furthermore, by using such a neural network model 203, it is possible to recognize the emotion of an input utterance from the tone of the input utterance, even if the content expressing that emotion is not included in the input utterance.

[0084] Another important point is that, according to the generator 100 described above, since it is given other arbitrary text that was not included in the text input during training, it is possible to recognize any emotion category that was not defined during training, such as "I want to buy," which means the desire to buy, without needing to retrain the model. In other words, it is possible to generate a neural network model 203 that can recognize any emotion category that was not defined during training, such as "I want to buy," which means the desire to buy, without needing to retrain the model. Furthermore, by using such a neural network model 203, it is possible to recognize any emotion category that was not defined during training, such as "I want to buy," which means the desire to buy, without needing to retrain the model.

[0085] In the above-described embodiment, the generation device 100 has a learning unit 202 and an inference unit 204, but it may also have either the learning unit 202 or the inference unit 204, with the other being another computer capable of communicating with the generation device 100.

[0086] It should be noted that the present invention is not limited to the embodiments described above, but includes various modifications and equivalent configurations within the spirit of the attached claims. For example, the embodiments described above are described in detail to make the present invention easier to understand, and the present invention is not necessarily limited to having all of the described configurations. Furthermore, some of the configurations of one embodiment may be replaced with those of another embodiment. Furthermore, some of the configurations of one embodiment may be added to those of another embodiment. Furthermore, some of the configurations of each embodiment may be added, deleted, or replaced with other configurations.

[0087] Furthermore, each of the aforementioned configurations, functions, processing units, and processing means may be implemented in hardware, for example, by designing them as integrated circuits, or they may be implemented in software by having a processor interpret and execute programs that realize each function.

[0088] Information such as programs, tables, and files that implement each function can be stored in memory, hard disks, SSDs (Solid State Drives), or on recording media such as IC (Integrated Circuit) cards, SD cards, and DVDs (Digital Versatile Discs).

[0089] Furthermore, the control lines and information lines shown are those deemed necessary for explanation purposes and do not necessarily represent all control lines and information lines required for implementation. In reality, it can be assumed that almost all components are interconnected. [Explanation of Symbols]

[0090] 100 generator 202 Learning Department 203 Neural Network Models 204 Reasoning section 244 Emotion recognition results 304 Speech Encoding Unit 311 Language encoding section 320 Update Department 500 contrasting matrices 604 Speech Coding Unit 610 Generation part 611,621 Language encoding section 612,622 Average distance calculation part 630 Classification Department

Claims

1. A memory unit that stores a set of training datasets, which are combinations of audio signals representing spoken speech and text representing positive or negative expressions of emotion. A speech coding unit that encodes the speech signal and generates a speech feature vector based on a first learning parameter, A language coding unit that encodes the text and generates a language feature vector based on a second learning parameter, An update unit updates the first learning parameter and the second learning parameter by comparative learning using a combination of the speech feature vector generated by the speech coding unit and the language feature vector generated by the language coding unit. A generating apparatus characterized by having the following features.

2. A generating apparatus according to claim 1, The memory unit stores a first training dataset which is a combination of a first audio signal representing a first utterance and a first text representing an affirmative or negative expression of a first emotion of the first utterance, and a second training dataset which is a combination of the first audio signal and a second text representing an affirmative or negative expression of a second emotion different from the first emotion. The audio encoding unit encodes the first audio signal to generate a first audio feature vector, The language coding unit encodes the first text to generate a first language feature vector, encodes the second text to generate a second language feature vector, The update unit updates the first learning parameter and the second learning parameter so that the first similarity between the first speech feature vector and the first language feature vector is higher than the second similarity between the first speech feature vector and the second language feature vector. A generating apparatus characterized by the following features.

3. A generating apparatus according to claim 1, The updated speech encoding unit, which has updated the first learning parameters by the update unit, encodes the speech signal representing the speech to be recognized to generate a speech feature vector of the recognized target. The updated language encoding unit, which has updated the second learning parameters by the update unit, encodes the text of the positive expression of the emotion to be recognized to generate a language feature vector of the positive expression of the emotion to be recognized, encodes the text of the negative expression of the emotion to be recognized to generate a language feature vector of the negative expression of the emotion to be recognized, A classification unit that classifies the speech of the recognized object into either an affirmative or negative expression based on the similarity between the speech feature vector of the recognized object and the language feature vector of the affirmative expression of the recognized object, and the similarity between the speech feature vector of the recognized object and the language feature vector of the negative expression of the recognized object. A generating apparatus characterized by having the following features.

4. A generating device having a storage unit that stores a set of training datasets, which are combinations of speech signals representing spoken language and text representing positive or negative expressions of emotion, A speech coding process that encodes the speech signal and generates a speech feature vector based on a first learning parameter, A language coding process that encodes the text and generates a language feature vector based on a second learning parameter, An update process is performed to update the first learning parameter and the second learning parameter by comparative learning using a combination of the speech feature vector generated by the speech coding process and the language feature vector generated by the language coding process. A generation method characterized by performing the following.

5. A processor in a generating device having a memory unit that stores a set of training datasets, which are combinations of speech signals representing spoken language and text representing positive or negative expressions of emotion, A speech coding process that encodes the speech signal and generates a speech feature vector based on a first learning parameter, A language coding process that encodes the text and generates a language feature vector based on a second learning parameter, An update process is performed to update the first learning parameter and the second learning parameter by comparative learning using a combination of the speech feature vector generated by the speech coding process and the language feature vector generated by the language coding process. A generation program characterized by causing the execution of a specific action.

Images