Training apparatus, inference apparatus, training method, and program

Abstract

This training apparatus is for training a neural network model, and comprises a training unit that updates a model parameter of the model on the basis of the error between second modal output information, which is output from the model to which first modal input information has been input, and correct second modal output information, and the error between first modal output information, which is output from the model to which second modal input information has been input, and correct first modal output information.

Description

Learning device, inference device, learning method, and program

[0001] The present invention relates to speech recognition, which takes speech as input and outputs the content of the speech, speech synthesis, which takes text as input and outputs speech that reads the text, and machine learning.

[0002] Known prior art related to speech synthesis and speech recognition is, for example, the technology disclosed in Non-Patent Document 1. The technology disclosed in Non-Patent Document 1 converts continuous speech into discrete speech tokens, and then uses a language model to treat the speech tokens in the same way as text, thereby solving a variety of tasks including speech synthesis and speech recognition.

[0003] T. Wang, L. Zhou, Z. Zhang, Y. Wu, S. Liu, Y. Gaur, Z. Chen, J. Li, and F. Wei, "VioLA: Unified codec language models for speech recognition, synthesis, and translation," arXiv:2305.16107, 2023.

[0004] In the above-mentioned conventional technology, the parts that input or output mel-spectrograms for speech synthesis and speech recognition are configured using separate models, and discrete tokens are used as input / output for the language model, which results in a degradation of processing quality such as speech synthesis quality and speech recognition performance.

[0005] Furthermore, since the above-mentioned parts are modeled independently for speech synthesis and speech recognition, parameters are required for each model, and knowledge sharing (parameter sharing) through simultaneous learning of speech synthesis and speech recognition is not possible. Note that this type of problem can arise in tasks other than speech synthesis and speech recognition.

[0006] The present invention has been made in consideration of the above points, and aims to provide a technology that enables improving the quality of processing in a model that performs multiple types of transformations while sharing parameters between the multiple types of transformations.

[0007] According to the disclosed technology, there is provided a learning device that performs learning of a neural network model, comprising a learning unit that updates model parameters of the model based on an error between second modal output information output from the model to which first modal input information has been input and the correct second modal output information, and an error between first modal output information output from the model to which second modal input information has been input and the correct first modal output information.

[0008] The disclosed technology provides a technology that enables a model that performs multiple types of transformations to improve the quality of processing while sharing parameters between the multiple types of transformations.

[0009] FIG. 1 is a diagram showing an example of the configuration of an information processing device 10 in a first embodiment. FIG. 2 is a flowchart for explaining the operation of the information processing device 10 during inference. FIG. 3 is a flowchart for explaining the operation of the information processing device 10 during learning. FIG. 4 is a diagram showing an example of the configuration of an information processing device 10 in a second embodiment. FIG. 5 is a diagram showing the configuration of a system including a learning device 100 and an inference device 200. FIG. 6 is a diagram showing the configuration of the learning device 100. FIG. 7 is a diagram showing the configuration of the inference device 200 in Example 1. FIG. 8 is a diagram showing the configuration of the inference device 200 in Example 2. FIG. 9 is a diagram showing experimental results. FIG. 10 is a diagram showing an example of the hardware configuration of the device.

[0010] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. The embodiment described below is merely an example, and the embodiment to which the present invention is applied is not limited to the following embodiment.

[0011] Specifically, first to third embodiments will be described. In the first to second embodiments, an information processing device 10 that performs speech synthesis and speech recognition will be described. In the third embodiment, an example of a device configuration that incorporates the technical ideas of the first and second embodiments will be described.

[0012] In the first to third embodiments, the inference device performs the processing of the components based on a trained model in which parameters have been trained in advance by machine learning. Furthermore, the learning device may use a general learning method in machine learning for learning, converting input data based on a machine learning model and training the model so that the output data is close to the correct output for a predetermined purpose.

[0013] Furthermore, each of the first to third embodiments is an embodiment that is incorporated into a practical application, provides specific improvements over the prior art, and represents an advancement in the field of technology. Specific improvements are described at the end of each embodiment.

[0014] First Embodiment First, a first embodiment will be described. In the first embodiment, an encoder is shared between speech synthesis processing and speech recognition processing, and a decoder is also shared between speech synthesis processing and speech recognition processing.

[0015] (First embodiment: device configuration) Fig. 1 shows an example configuration of an information processing device 10 in the first embodiment. As shown in Fig. 1, the information processing device 10 in the first embodiment includes a model parameter learning unit 11, a speech pre-processing unit 12, a text pre-processing unit 13, a common encoder unit 14, a common decoder unit 15, a text post-processing unit 16, a speech post-processing unit 17, a text decoder pre-processing unit 18, a speech decoder pre-processing unit 19, and a style estimation unit 20.

[0016] In this embodiment, the "speech pre-processing unit 12, text pre-processing unit 13, common encoder unit 14, common decoder unit 15, text post-processing unit 16, speech post-processing unit 17, text decoder pre-processing unit 18, speech decoder pre-processing unit 19, and style estimation unit 20" are a neural network model.

[0017] In addition, in the case where the voice of one person is used in the voice synthesis, the style estimation unit 20 may not be provided.

[0018] 1 is a device that performs both learning and inference. Focusing on learning, the information processing device 10 may be called a learning device. Focusing on inference, the information processing device 10 may be called an inference device.

[0019] The function of each unit will be explained below. In explaining each unit, the input and output will be clearly stated, and then the processing content will be explained.

[0020] <Model Parameter Learning Unit 11> The input to the model parameter learning unit 11 is speech recognition learning data D=(X １ , Y １ ), ..., (X ｜Ｄ｜ , Y ｜Ｄ｜ ), speech synthesis training data D'=(X' １ , Y' １ ), ..., (X' ｜Ｄ´｜ , Y' ｜Ｄ´｜ ) where X and X' respectively represent mel spectrograms, Y and Y' respectively represent text, |D| and |D'| respectively represent the number of data. The output from the model parameter learning unit 11 is the model parameter θ.

[0021] In the following description, when a model parameter θ is input to each functional unit, the model parameter θ is the model parameter for that functional unit.

[0022] The model parameter learning unit 11 learns and outputs model parameters for the "speech pre-processing unit 12, text pre-processing unit 13, common encoder unit 14, common decoder unit 15, text post-processing unit 16, speech post-processing unit 17, text decoder pre-processing unit 18, speech decoder pre-processing unit 19, and style estimation unit 20" from the speech recognition training data and speech synthesis training data. Conventional learning algorithms and error functions can be used for training speech synthesis and speech recognition.

[0023] Here, the "speech pre-processing unit 12, text pre-processing unit 13, common encoder unit 14, common decoder unit 15, text post-processing unit 16, speech post-processing unit 17, text decoder pre-processing unit 18, speech decoder pre-processing unit 19, and style estimation unit 20" are referred to as the learning target model.

[0024] For example, in the case of speech synthesis, the model parameter learning unit 11 can input text of the speech content into the learning model and learn the model parameters of the learning model using the L1 error between the estimated speech output from the learning model and the correct speech.

[0025] Furthermore, with regard to speech recognition, the model parameter learning unit 11 can input speech (more specifically, a mel spectrogram) to the training model and learn the model parameters of the training model using the cross-entropy error between the estimated text output from the training model and the correct text.

[0026] More specifically, the model parameter learning unit 11 can update the model parameters by using error backpropagation with respect to an error function obtained by adding, with arbitrary weighting, the error between correct speech and estimated speech in speech synthesis (e.g., the above-mentioned L1 error) and the error between correct text and estimated text in speech recognition (e.g., the above-mentioned cross-entropy error). In other words, the model parameters can be updated so that the value of the error function is minimized.

[0027] For example, if the error in speech synthesis is Error 1 and its weight is W1, and the error in speech recognition is Error 2 and its weight is W2, then the error function can be expressed as "W1 × Error 1 + W2 × Error 2." W1 and W2 can be set arbitrarily, and for example, W1 = W2 = 1.

[0028] <Speech Pre-Processing Unit 12> The input to the speech pre-processing unit 12 is a mel spectrogram a of the observed speech and model parameters θ. The output from the speech pre-processing unit 12 is speech intermediate features e.

[0029] The speech preprocessing unit 12 estimates speech intermediate features e from the mel spectrogram a of the input observed speech and the model parameters θ. The method for converting the mel spectrogram a into the speech intermediate features e is not limited to a specific method. For example, a neural network with a fully connected layer can be used as the speech preprocessing unit 12, and the conversion can be performed by the neural network.

[0030] <Text Pre-Processing Unit 13> The input to the text pre-processing unit 13 is the text b of the utterance content and the model parameters θ. The output from the text pre-processing unit 13 is the text intermediate feature f.

[0031] The text preprocessing unit 13 estimates intermediate text features f from the input utterance text b and the model parameters θ. The method for converting the utterance text b into intermediate text features f is not limited to a specific method. For example, a neural network with a convolutional layer can be used as the text preprocessing unit 13, and the conversion can be performed by the neural network.

[0032] <Common Encoder Unit 14> The input to the common encoder unit 14 is the "speech intermediate feature e or text intermediate feature f" and the model parameter θ. The output from the common encoder unit 14 is the encoder intermediate feature g.

[0033] During speech recognition, the common encoder unit 14 outputs encoder intermediate features g from speech intermediate features e and model parameters θ. During speech synthesis, the common encoder unit 14 outputs encoder intermediate features g from text intermediate features f and model parameters θ.

[0034] The model used as the common encoder unit 14 is not limited to a specific model. For example, a model in which multiple Transformer encoder blocks are stacked can be used as the common encoder unit 14. The common encoder unit 14 receives either the speech intermediate feature e or the text intermediate feature f without distinguishing between them, and performs common processing on both the speech intermediate feature e and the text intermediate feature f.

[0035] More specifically, the common encoder unit 14 is a functional unit that receives features such as text and speech as input and outputs features (specifically, vectors) that are useful to the decoder. Useful features are features that are useful for a specific task (e.g., speech recognition, speech synthesis), and are features that make it easier for the model to generate a correct output for the target task during the learning phase. Note that the above-mentioned "decoder" is the "common decoder unit 15" in the first embodiment, and the "text decoder unit 21" and the "speech decoder unit 22" in the second embodiment.

[0036] <Common decoder unit 15> The inputs to the common decoder unit 15 are the “audio decoder preprocessing output k or text decoder preprocessing output j”, the encoder intermediate feature g, and the model parameter θ. The output from the common decoder unit 15 is the decoder intermediate feature h.

[0037] During speech recognition, the common decoder unit 15 outputs a decoder intermediate feature h from a text decoder preprocessing output j, an encoder intermediate feature g, and a model parameter θ. During speech synthesis, the common decoder unit 15 outputs a decoder intermediate feature h from a speech decoder preprocessing output k, an encoder intermediate feature g, and a model parameter θ.

[0038] There is no particular limitation to the model used as the common decoder unit 15. For example, the common decoder unit 15 may be configured by stacking a plurality of transformer decoder blocks.

[0039] <Text Post-Processing Unit 16> The inputs to the text post-processing unit 16 are the decoder intermediate feature value h and the model parameter θ. The output from the text post-processing unit 16 is text v of the speech recognition content.

[0040] The text post-processing unit 16 outputs speech-recognized text v from the decoder intermediate features h and the model parameters θ. The model used as the text post-processing unit 16 is not limited to a specific model. For example, a fully connected layer neural network can be used as the text post-processing unit 16.

[0041] <Audio Post-Processing Unit 17> The inputs to the audio post-processing unit 17 are the decoder intermediate feature value h and the model parameter θ. The output from the audio post-processing unit 17 is the mel spectrogram w of the synthesized speech.

[0042] The speech post-processing unit 17 outputs a mel spectrogram w of the synthesized speech from the decoder intermediate feature value h and the model parameter θ. The model used as the speech post-processing unit 17 is not limited to a specific model. For example, a fully connected neural network can be used as the speech post-processing unit 17.

[0043] <Text decoder preprocessing unit 18> The input to the text decoder preprocessing unit 18 is the speech recognition content from the previous step and the model parameter θ. The output from the text decoder preprocessing unit 18 is the text decoder preprocessing output j.

[0044] The text decoder preprocessing unit 18 outputs the text decoder preprocessing output j for the next step based on the speech recognition content from the previous step and the model parameter θ. The model used as the text decoder preprocessing unit 18 is not limited to a specific model. For example, a neural network with a fully connected layer can be used as the text decoder preprocessing unit 18.

[0045] <Audio Decoder Preprocessing Unit 19> The input to the audio decoder preprocessing unit 19 is the mel spectrogram of the synthesized audio from the previous step and model parameters θ. The output from the audio decoder preprocessing unit 19 is audio decoder preprocessing output k.

[0046] The speech decoder preprocessing unit 19 outputs a speech decoder preprocessing output k for the next step of synthesized speech from the mel spectrogram of the previous step of synthesized speech and the model parameter θ. The model used as the speech decoder preprocessing unit 19 is not limited to a specific model. For example, a neural network with a fully connected layer can be used as the speech decoder preprocessing unit 19.

[0047] <Style Estimation Unit 20> The input to the style estimation unit 20 is a reference speech c and a model parameter θ. The output from the style estimation unit 20 is a style feature i. The style feature i is input to the speech decoder pre-processing unit 19.

[0048] The style estimation unit 20 estimates a style feature i that represents the style (speaker or speaking style) of the reference speech from the reference speech c and the model parameter θ. As the style estimation unit 20, for example, a model used as a style estimation unit in conventional speech synthesis technology can be used.

[0049] (Operation of the information processing device 10 in the first embodiment) The operation of the information processing device 10 during inference will be described with reference to the flowchart in Fig. 2. The flowchart in Fig. 2 is a flowchart common to speech recognition and speech synthesis. Here, it is assumed that a trained model parameter θ has already been set for each functional unit.

[0050] First, the operation for performing speech recognition will be described. In S101, a mel spectrogram a of observed speech is input to the speech pre-processing unit 12. In S102, the speech pre-processing unit 12 performs pre-processing and outputs speech intermediate features e.

[0051] In S103, the common encoder unit 14 performs encoding processing on the audio intermediate feature e and outputs an encoder intermediate feature g.

[0052] In S104, the common decoder unit 15 receives the text decoder preprocessing output j and the encoder intermediate feature g, performs decoding processing, and outputs decoder intermediate feature h. In S105, the text post-processing unit 16 receives the decoder intermediate feature h, performs post-processing (conversion to text), and outputs text, which is the speech recognition content v.

[0053] Next, the operation for performing speech synthesis will be described. In S101, text b of the utterance content is input to the text pre-processing unit 13. In S102, the text pre-processing unit 13 outputs intermediate text features f.

[0054] In S103, the common encoder unit 14 performs encoding processing on the text intermediate feature f and outputs the encoder intermediate feature g.

[0055] In S104, the common decoder unit 15 receives the speech decoder pre-processing output k and the encoder intermediate feature g, performs decoding processing, and outputs the decoder intermediate feature h. In S105, the speech post-processing unit 17 receives the decoder intermediate feature h, performs post-processing (conversion to a mel spectrogram), and outputs the mel spectrogram w of the synthesized speech.

[0056] In the operation of the information processing device 10 described above, during speech recognition, the decoder intermediate feature h output from the common decoder unit 15 is input to the text post-processing unit 16, and during speech synthesis, the decoder intermediate feature h output from the common decoder unit 15 is input to the speech post-processing unit 17. The method of performing such control is not limited to a specific method. For example, a control unit may be provided, and the control unit may identify the task to be executed and instruct the common decoder unit 15 to output the decoder intermediate feature h. Furthermore, the instruction may be made using specific information for identifying the task (e.g., a zero vector in the case of speech synthesis, or a start symbol in the case of speech recognition). The function corresponding to the control unit may be provided anywhere. For example, the control unit may be provided in any of the functional units of the common encoder unit 14, the common decoder unit 15, the text post-processing unit 16, the speech post-processing unit 17, the text decoder pre-processing unit 18, and the speech decoder pre-processing unit 19.

[0057] Next, the operation of the information processing device 10 during learning will be described with reference to the flowchart in Fig. 3. Here, the "speech pre-processing unit 12, text pre-processing unit 13, common encoder unit 14, common decoder unit 15, text post-processing unit 16, speech post-processing unit 17, text decoder pre-processing unit 18, speech decoder pre-processing unit 19, and style estimation unit 20" are referred to as a learning target model.

[0058] In S201, the model parameter learning unit 11 receives the training data D for speech recognition and the training data D' for speech synthesis.

[0059] In S202, the model parameter learning unit 11 inputs the mel spectrogram of the speech in the speech recognition training data D to the training model and obtains the speech recognition text output from the training model. The model parameter learning unit 11 also inputs the text of the utterance content in the speech synthesis training data D' to the training model and obtains the mel spectrogram of the synthetic speech output from the training model.

[0060] In S203, the model parameter learning unit 11 updates the parameters of the model to be learned by using error backpropagation so as to reduce the value of an error function obtained by weighting the error between the correct speech and the estimated speech in speech synthesis (e.g., the L1 error described above) and the error between the correct text and the estimated text in speech recognition (e.g., the cross-entropy error described above).

[0061] In S204, if the model parameter learning unit 11 determines that the learning has finished, it ends the process. If it does not determine that the learning has finished, it repeats the process from S202 using the next learning data.

[0062] The information processing device 10 of the first embodiment described above provides a specific improvement over conventional technology in that it shares an encoder and a decoder for speech synthesis and speech recognition, thereby reducing the required model size and enabling the construction of a more general-purpose model, and represents an advancement in the technical fields related to machine learning speech synthesis and speech recognition.

[0063] Second Embodiment Next, a second embodiment will be described. The second embodiment differs from the first embodiment in that a decoder is not shared between speech synthesis processing and speech recognition processing, but the other configurations are the same as those of the first embodiment.

[0064] That is, in the second embodiment, separate models are used for the decoder for speech synthesis and the decoder for speech recognition, while for the encoder, a common model is used for speech synthesis and speech recognition, as in the first embodiment.

[0065] (Second embodiment: device configuration) Fig. 4 shows an example of the configuration of an information processing device 10 in a second embodiment. Here, differences from the first embodiment will be described.

[0066] As shown in Fig. 4, in the second embodiment, a text decoder unit 21 and an audio decoder unit 22 are provided instead of the shared decoder unit 15 in the first embodiment. The configuration other than this part is the same as the configuration in the first embodiment. The functions of each unit that differ from the first embodiment are as follows.

[0067] <Text Decoder Unit 21> The inputs to the text decoder unit 21 are the text decoder preprocessing output j, the encoder intermediate feature g, and the model parameter θ. The output from the text decoder unit 21 is the text decoder intermediate feature l.

[0068] The text decoder unit 21 outputs a text decoder intermediate feature l from the text decoder preprocessing output j, the encoder intermediate feature g, and the model parameter θ.

[0069] There is no particular limitation to the model used as the text decoder unit 21. As the text decoder unit 21, for example, a plurality of Transformer decoder blocks stacked one on top of the other can be used.

[0070] <Audio Decoder Unit 22> The inputs to the audio decoder unit 22 are the audio decoder preprocessing output k, the encoder intermediate feature g, and the model parameter θ. The output from the audio decoder unit 22 is the audio decoder intermediate feature m.

[0071] The audio decoder unit 22 outputs an audio decoder intermediate feature m from the audio decoder preprocessing output k, the encoder intermediate feature g, and the model parameter θ.

[0072] There is no particular limitation to the model used as the audio decoder unit 22. For example, the audio decoder unit 22 may be configured by stacking a plurality of transformer decoder blocks.

[0073] (Operation of information processing device 10 in second embodiment) The operation during learning of the information processing device 10 in the second embodiment is the same as the operation during learning of the information processing device 10 in the first embodiment. However, the models to be learned are "speech pre-processing unit 12, text pre-processing unit 13, common encoder unit 14, text decoder unit 21, speech decoder unit 22, text post-processing unit 16, speech post-processing unit 17, text decoder pre-processing unit 18, speech decoder pre-processing unit 19, and style estimation unit 20."

[0074] The overall flow of the operation of the information processing device 10 during inference in the second embodiment is the same as the overall flow of the operation of the information processing device 10 during inference in the first embodiment, so the operation of the information processing device 10 during inference in the second embodiment will be described with reference to the flowchart of Figure 2 used in the first embodiment. The flowchart of Figure 2 is a flowchart common to speech recognition and speech synthesis. Here, it is assumed that a trained model parameter θ has already been set for each functional unit.

[0075] First, the operation for performing speech recognition will be described. In S101, a mel spectrogram a of observed speech is input to the speech pre-processing unit 12. In S102, the speech pre-processing unit 12 outputs a speech intermediate feature e.

[0076] In S103, the common encoder unit 14 performs encoding processing on the audio intermediate feature e and outputs an encoder intermediate feature g.

[0077] In S104, the text decoder unit 21 receives the text decoder preprocessing output j and the encoder intermediate feature g, performs decoding processing, and outputs the text decoder intermediate feature l. In S105, the text post-processing unit 16 receives the text decoder intermediate feature l, performs post-processing (conversion to text), and outputs text, which is the speech recognition content v.

[0078] Next, the operation for performing speech synthesis will be described. In S101, text b of the utterance content is input to the text pre-processing unit 13. In S102, the text pre-processing unit 13 outputs intermediate text features f.

[0079] In S103, the common encoder unit 14 performs encoding processing on the text intermediate feature f and outputs the encoder intermediate feature g.

[0080] In S104, the speech decoder unit 22 receives the speech decoder pre-processing output k and the encoder intermediate feature g, performs decoding processing, and outputs the speech decoder intermediate feature m. In S105, the speech post-processing unit 17 receives the speech decoder intermediate feature m, performs post-processing (conversion to a mel spectrogram), and outputs the mel spectrogram w of the synthesized speech.

[0081] The information processing device 10 of the second embodiment described above provides a specific improvement over conventional technology in that it shares an encoder between speech synthesis and speech recognition, thereby reducing the required model size and enabling the construction of a more general-purpose model, and represents an advancement in the technical fields related to machine learning speech synthesis and speech recognition.

[0082] Third Embodiment In the technology according to this embodiment, the learning device 100 and the inference device 200 may be provided separately, as shown in Fig. 5. An example configuration in which the learning device 100 and the inference device 200 are provided separately will be described as a third embodiment.

[0083] 5 , trained model parameters trained by the training device 100 are stored in the trained model DB 300. The inference device 200 reads the trained model parameters from the trained model DB 300 and sets the trained model parameters in each functional unit (neural network model).

[0084] <Learning Device 100> Fig. 6 shows an example of the configuration of the learning device 100. As shown in Fig. 6, the learning device 100 includes a learning unit 110 and a model 120 that is the learning target.

[0085] The learning unit 110 corresponds to, for example, the model parameter learning unit 11 in the first or second embodiment.

[0086] Furthermore, the model 120 to be trained is, for example, the "speech pre-processing unit 12, text pre-processing unit 13, common encoder unit 14, common decoder unit 15, text post-processing unit 16, speech post-processing unit 17, text decoder pre-processing unit 18, speech decoder pre-processing unit 19, and style estimation unit 20" in the first embodiment, or the "speech pre-processing unit 12, text pre-processing unit 13, common encoder unit 14, text decoder unit 21, speech decoder unit 22, text post-processing unit 16, speech post-processing unit 17, text decoder pre-processing unit 18, speech decoder pre-processing unit 19, and style estimation unit 20" in the second embodiment. However, without being limited to these, the model 120 may be "the first modal pre-processing unit 210, the second modal pre-processing unit 220, the modal common conversion unit 230, the second modal post-processing unit 240, and the first modal post-processing unit 250" in the inference device 200 of Example 1 described below, or "the first modal pre-processing unit 210, the second modal pre-processing unit 220, the modal common conversion unit 260, the second modal post-processing unit 270, and the first modal post-processing unit 280" in the inference device 200 of Example 2.

[0087] It is assumed that the first modal is speech and the second modal is text, but this is not limiting.

[0088] The learning unit 110 in the learning device 100 updates the model parameters of the model 120 based on an error (called a first error) between second modal output information output from the model 120 to which the first modal input information has been input and the correct second modal output information, and an error (called a second error) between the first modal output information output from the model 120 to which the second modal input information has been input and the correct first modal output information. When learning is completed, the learning unit 110 outputs the trained model 120.

[0089] For example, the learning unit 110 can update the parameters of the model 120 by using error backpropagation so as to reduce the value of an error function represented by a weighted sum of a first error and a second error. The model 120 also includes an intermediate layer that is used in common by the first modal and the second modal. This intermediate layer is, for example, the "common encoder unit 14 and common decoder unit 15" in the first embodiment, the common encoder unit 14 in the second embodiment, the modal common converter 230 in the third embodiment, or the modal common converter 260 in the third embodiment.

[0090] <Inference device 200> The following describes the inference device 200 of the third embodiment. The inference device 200 of the third embodiment has two examples: Example 1, which corresponds to the technical idea of ​​the first embodiment described above (both the decoder and the encoder are shared between tasks), and Example 2, which corresponds to the technical idea of ​​the second embodiment described above (the encoder is shared between tasks, and a decoder is provided for each task), and each will be described.

[0091] In the following description, it is assumed that the first modal is speech and the second modal is text, but this is not limiting. For example, the first modal and the second modal may each be a modal that is neither speech nor text.

[0092] <Example 1: Inference device 200> Fig. 7 shows an example configuration of inference device 200 of Example 1. As shown in Fig. 7, inference device 200 of Example 1 includes a first modal pre-processing unit 210, a second modal pre-processing unit 220, a modal common conversion unit 230, a second modal post-processing unit 240, and a first modal post-processing unit 250. "First modal pre-processing unit 210, second modal pre-processing unit 220, modal common conversion unit 230, second modal post-processing unit 240, and first modal post-processing unit 250" are a neural network model.

[0093] For example, the first modal pre-processing unit 210 corresponds to the speech pre-processing unit 12, the second modal pre-processing unit 220 corresponds to the text pre-processing unit 13, the modal common conversion unit 230 corresponds to the "common encoder unit 14 and common decoder unit 15", the second modal post-processing unit 240 corresponds to the "text post-processing unit 16 and text decoder pre-processing unit 18", and the first modal post-processing unit 250 corresponds to the "speech post-processing unit 17 and speech decoder pre-processing unit 19".

[0094] The first modal preprocessing unit 210 receives first modal input information as input and outputs first modal features. For example, the first modal input information is a mel spectrogram a of observed speech, and the first modal features are speech intermediate features e.

[0095] The second-modal pre-processing unit 220 receives second-modal input information as input and outputs second-modal features. For example, the second-modal input information is utterance content text b, and the second-modal features are text intermediate features f.

[0096] The modal common converter 230 receives the first modal feature or the second modal feature as input and outputs intermediate feature. For example, the intermediate feature is a decoder intermediate feature h. The modal common converter 230 may further receive a text decoder preprocessing output j or a speech decoder preprocessing output k.

[0097] The second-modal post-processing unit 240 receives the intermediate features converted from the first-modal features as input and outputs second-modal output information. The second-modal output information is, for example, text v representing the speech recognition content. The second-modal post-processing unit 240 may also output a text decoder pre-processing output j.

[0098] The first modal post-processing unit 250 receives the intermediate features converted from the second modal features and outputs first modal output information, which may be, for example, a mel spectrogram w of the synthesized speech. The first modal post-processing unit 250 may also output a speech decoder pre-processing output k.

[0099] The modal common converter 230 may include a common encoder unit used in common for the first modal and the second modal, and a common decoder unit used in common for the first modal and the second modal.

[0100] <Example 2: Inference device 200> Figure 8 shows an example configuration of inference device 200 of Example 2. As shown in Figure 8, inference device 200 of Example 2 includes a first modal pre-processing unit 210, a second modal pre-processing unit 220, a modal common conversion unit 260, a second modal post-processing unit 270, and a first modal post-processing unit 280. "The first modal pre-processing unit 210, the second modal pre-processing unit 220, the modal common conversion unit 260, the second modal post-processing unit 270, and the first modal post-processing unit 280" are a neural network model.

[0101] For example, the first modal pre-processing unit 210 corresponds to the speech pre-processing unit 12, the second modal pre-processing unit 220 corresponds to the text pre-processing unit 13, the modal common conversion unit 260 corresponds to the common encoder unit 14, the second modal post-processing unit 270 corresponds to the "text decoder unit 21, text post-processing unit 16, and text decoder pre-processing unit 18," and the first modal post-processing unit 280 corresponds to the "speech decoder unit 22, speech post-processing unit 17, and speech decoder pre-processing unit 19."

[0102] The first modal preprocessing unit 210 receives first modal input information as input and outputs first modal features. For example, the first modal input information is a mel spectrogram a of observed speech, and the first modal features are speech intermediate features e.

[0103] The second-modal pre-processing unit 220 receives second-modal input information as input and outputs second-modal features. For example, the second-modal input information is utterance content text b, and the second-modal features are text intermediate features f.

[0104] The modal common converter 260 receives the first modal feature or the second modal feature as input and outputs intermediate feature, for example, encoder intermediate feature g.

[0105] The second-modal post-processing unit 270 receives the intermediate features converted from the first-modal features and outputs second-modal output information, which is, for example, text representing the speech recognition content.

[0106] The first modal post-processing unit 280 receives the intermediate features converted from the second modal features and outputs first modal output information, which may be a mel spectrogram w of the synthesized speech, for example.

[0107] The modal common converter 260 may have a common encoder used in common for the first modal and the second modal. Furthermore, the first modal post-processor 280 may have a first modal decoder used for the first modal, and the second modal post-processor 270 may have a second modal decoder used for the second modal.

[0108] <Regarding the Number of Modals> In both Example 1 and Example 2, the first and second modals are used, but the number of modals is not limited to two. The number of modals may be three or more. For example, if a third modal is used in addition to the first and second modals, a third modal pre-processing unit and a third modal post-processing unit can be added to the inference device 200 (and the model to be trained). The common modal conversion unit performs conversion processing common to the first to third modals. The same applies when the number of modals is greater than three. Furthermore, configurations when the number of modals is three or more include configurations using the first and second modals, as shown in Figures 7 and 8.

[0109] Note that in order to be able to handle three or more modals, it is not necessarily necessary to add an x-th modal pre-processing unit and an x-th modal post-processing unit as a set (x is an integer equal to or greater than 3). For example, it is possible to add only an x-th modal pre-processing unit so that only the input side can handle the x-th modal.

[0110] Furthermore, even when two modalities are handled, it is possible to have only one modal on the output side, such as when two tasks, speech recognition and speech conversion, are handled simultaneously.

[0111] <Regarding Tasks> In the third embodiment, the inference device 200 can also perform conversion within one modality. For example, the inference device 200 can perform speech conversion or text conversion.

[0112] For example, pre-conversion speech is input to inference device 200 as first modal input information, and inference device 200 outputs the converted speech as first modal output information. Also, for example, pre-conversion text is input to inference device 200 as second modal input information, and inference device 200 outputs the converted text as second modal output information.

[0113] For example, the model parameters of the common modal converter 230 in Example 1 are shared between the speech conversion task and the text conversion task. For example, the model parameters of the common modal converter 260 in Example 2 are also shared between the speech conversion task and the text conversion task.

[0114] For learning, the learning device 100 may train the model 120 to be learned so as to perform the above tasks.

[0115] The learning device 100 and the inference device 200 of the third embodiment described above share a conversion unit (e.g., an encoder (or an encoder and a decoder)) between multiple modalities, thereby providing a specific improvement over conventional techniques in that the required model size is reduced and a more general-purpose model can be constructed, and represents an advancement in the technical field related to information conversion in machine learning.

[0116] (Experiment) Experiments were conducted on the technology of the first embodiment (proposed method: sharing the encoder and decoder) and the technology of the second embodiment (proposed method: sharing the encoder), and the experimental results are shown in FIG.

[0117] For comparison, the results of a prior art transformer-based speech recognition model (baseline speech recognition) and a transformer-based speech synthesis model (baseline speech synthesis) are also shown.

[0118] In the experiments, the character error rate (CER) was used as an evaluation index for speech recognition. The CER, Mel-cepstrum distortion (MCD), and root mean squared error (RMSE) of the synthesized speech were used as objective evaluation indexes for speech synthesis. In each case, the smaller the value, the better the performance.

[0119] As shown in Figure 9, the proposed method improves speech recognition performance compared to the baseline in both the first embodiment (sharing the "encoder and decoder") and the second embodiment (sharing the encoder). Furthermore, the method of the second embodiment (sharing the encoder) improves speech synthesis performance compared to the baseline.

[0120] (Hardware Configuration Example) Any of the devices described in this embodiment (information processing device 10, learning device 100, and inference device 200) can be realized, for example, by causing a computer to execute a program. This computer may be a physical computer or a virtual machine on the cloud.

[0121] That is, the device can be realized by executing a program corresponding to the processing performed by the device using hardware resources such as a CPU and memory built into a computer. The program can be recorded on a computer-readable recording medium (such as a portable memory) and stored or distributed. The program can also be provided via a network such as the Internet or email.

[0122] Fig. 10 is a diagram showing an example of the hardware configuration of the computer. The computer in Fig. 10 includes a drive device 1000, an auxiliary storage device 1002, a memory device 1003, a CPU 1004, an interface device 1005, a display device 1006, an input device 1007, an output device 1008, and the like, all of which are interconnected via a bus B. The computer may further include a GPU.

[0123] The program that realizes the processing on the computer is provided by a recording medium 1001, such as a CD-ROM or a memory card. When the recording medium 1001 storing the program is set in the drive device 1000, the program is installed from the recording medium 1001 to the auxiliary storage device 1002 via the drive device 1000. However, the program does not necessarily have to be installed from the recording medium 1001, but may be downloaded from another computer via a network. The auxiliary storage device 1002 stores the installed program as well as necessary files, data, etc.

[0124] The memory device 1003 reads and stores a program from the auxiliary storage device 1002 when an instruction to start the program is received. The CPU 1004 realizes functions related to the device in accordance with the program stored in the memory device 1003. The interface device 1005 is used as an interface for connecting to a network, etc. The display device 1006 displays a GUI (Graphical User Interface) or the like according to the program. The input device 1007 is composed of a keyboard, mouse, buttons, a touch panel, etc., and is used to input various operation instructions. The output device 1008 outputs the results of calculations.

[0125] (Summary, Effects, etc. of Technology According to the Embodiment) In the conventional technology disclosed in Non-Patent Document 1, speech recognition and speech synthesis of mel-spectrogram input and output are modeled separately. In contrast, in the technology according to the present embodiment, parameters of the encoder (or "encoder and decoder"), which are common parts between speech recognition and speech synthesis, are shared.

[0126] As described above, the technology according to this embodiment allows the encoder part (or both the encoder part and the decoder part) to be shared between speech recognition that uses a mel spectrogram as input and speech synthesis that uses a mel spectrogram as output. By sharing parameters that are common to speech recognition and speech synthesis, the required model size can be reduced compared to a method that prepares models for both speech recognition and speech synthesis, and a more general-purpose model can be constructed.

[0127] Furthermore, in the conventional technology disclosed in Non-Patent Document 1, input and output speech to a language model is converted into discrete tokens. This results in a lower amount of information compared to speech that is a continuous value, and the quality of speech synthesis and speech recognition performance deteriorates. In contrast, the technology according to this embodiment makes it possible to avoid this deterioration by using continuous speech as input and output to a common encoder (or common encoder-decoder).

[0128] Furthermore, the language model (Multi-Task Codec Language Modeling) in the prior art disclosed in Non-Patent Document 1 is a model that includes only a decoder. In a model that includes only a decoder, conversion from input tokens to features and estimation of output tokens are performed simultaneously. Therefore, when handling voice data, etc., the input and output become discrete systems, resulting in quality degradation. In contrast, the technology according to this embodiment includes a common encoder unit 14 in addition to the decoder, thereby avoiding quality degradation.

[0129] The following additional notes are provided regarding the above-described embodiments.

[0130] <Additional Notes> (Additional Item 1) A learning device that performs learning of a neural network model, comprising: a memory; and at least one processor connected to the memory, wherein the processor updates model parameters of the model based on an error between second modal output information output from the model to which first modal input information has been input and the correct second modal output information, and an error between first modal output information output from the model to which second modal input information has been input and the correct first modal output information. (Additional Item 2) The learning device according to Additional Item 1, wherein the model includes an intermediate layer that is used in common by the first modal and the second modal. (Supplementary Item 3) An inference device that performs inference using a neural network model, comprising: a memory; and at least one processor connected to the memory, wherein the processor executes, as processing of the model, a process of outputting a first modal feature when first modal input information is input, a process of outputting a second modal feature when second modal input information is input, a process of using the first modal feature or the second modal feature as input to output intermediate features, a process of using the first modal feature or the second modal feature as input to output second modal output information when the first modal feature is output, a process of using the intermediate feature converted from the first modal feature as input to output second modal output information, and a process of using the intermediate feature converted from the second modal feature as input to output first modal output information. (Supplementary Item 4) The inference device according to Supplementary Item 3, wherein the processor executes a common encoder process shared by the first modal and the second modal, and a common decoder process shared by the first modal and the second modal. (Supplementary Item 5) The inference device according to Supplementary Item 3, wherein the processor executes a common encoder process that is used in common for a first modal and a second modal, and executes a first modal decoder process for the first modal or a second modal decoder process for the second modal.(Supplementary Item 6) An inference device that performs inference using a neural network model, comprising: a memory; and at least one processor connected to the memory, wherein the processor performs, as processing of the model, a process of outputting a first modal feature when first modal input information is input; a process of outputting a second modal feature when second modal input information is input; a process of outputting an intermediate feature by performing processing common to the first modal and second modal with the first modal feature or the second modal feature as input; a process of outputting second modal output information with the intermediate feature converted from the first modal feature as input with the first modal feature as output; and a process of outputting first modal output information with the intermediate feature converted from the second modal feature as input with the second modal feature as output. (Supplementary Item 7) A learning method executed by a learning device that learns a neural network model, comprising a learning step of updating model parameters of the model based on an error between second modal output information output from the model to which first modal input information has been input and the correct second modal output information, and an error between first modal output information output from the model to which second modal input information has been input and the correct first modal output information. (Supplementary Item 8) A non-transitory storage medium storing a program for causing a computer to function as the learning device described in Supplementary Item 1 or 2. (Supplementary Item 9) A non-transitory storage medium storing a program for causing a computer to function as the inference device described in any one of Supplementary Items 3 to 6.

[0131] Although the present embodiment has been described above, the present invention is not limited to such a specific embodiment, and various modifications and changes are possible within the scope of the gist of the present invention described in the claims.

[0132] 10 Information processing device 11 Model parameter learning unit 12 Speech pre-processing unit 13 Text pre-processing unit 14 Common encoder unit 15 Common decoder unit 16 Text post-processing unit 17 Speech post-processing unit 18 Text decoder pre-processing unit 19 Speech decoder pre-processing unit 20 Style estimation unit 21 Text decoder unit 22 Speech decoder unit 100 Learning device 110 Learning unit 120 Model 200 Inference device 210 First modal pre-processing unit 220 Second modal pre-processing unit 230, 260 Modal common conversion unit 240, 270 Second modal post-processing unit 250, 280 First modal post-processing unit 300 Trained model DB 1000 Drive device 1001 Recording medium 1002 Auxiliary storage device 1003 Memory device 1004 CPU 1005 Interface device 1006 Display device 1007 Input device 1008 Output device

Claims

1. A learning device for learning a neural network model, comprising a learning unit that updates model parameters of the model based on the error between second modal output information output from the model to which first modal input information has been input and the correct second modal output information, and the error between first modal output information output from the model to which second modal input information has been input and the correct first modal output information.

2. The learning device according to claim 1, wherein the model comprises an intermediate layer that is used in common in the first modal and the second modal.

3. An inference device that performs inference using a neural network model, the model comprising: a first modal pre-processing unit that receives first modal input information as input and outputs first modal features; a second modal pre-processing unit that receives second modal input information as input and outputs second modal features; a modal common conversion unit that receives the first modal features or the second modal features as input and outputs intermediate features; a second modal post-processing unit that receives the intermediate features converted from the first modal features as input and outputs second modal output information; and a first modal post-processing unit that receives the intermediate features converted from the second modal features as input and outputs first modal output information.

4. An inference device as described in claim 3, wherein the common modal conversion unit has a common encoder unit used in common for the first modal and the second modal, and a common decoder unit used in common for the first modal and the second modal.

5. An inference device as described in claim 3, wherein the common modal conversion unit has a common encoder unit used in common for the first modal and the second modal, the first modal post-processing unit has a first modal decoder unit used for the first modal, and the second modal post-processing unit has a second modal decoder unit used for the second modal.

6. An inference device that performs inference using a neural network model, the model comprising: a first modal pre-processing unit that receives first modal input information as input and outputs first modal features; a second modal pre-processing unit that receives second modal input information as input and outputs second modal features; a modal common conversion unit that receives the first modal features or the second modal features as input and outputs intermediate features; a second modal post-processing unit that receives the intermediate features converted from the first modal features as input and outputs second modal output information; and a first modal post-processing unit that receives the intermediate features converted from the second modal features as input and outputs first modal output information, wherein the modal common conversion unit has a common encoder unit that is used in common for the first modal and the second modal.

7. A learning method executed by a learning device that learns a neural network model, comprising a learning step of updating model parameters of the model based on an error between second modal output information output from the model to which first modal input information has been input and the correct second modal output information, and an error between first modal output information output from the model to which second modal input information has been input and the correct first modal output information.

8. A program for causing a computer to function as the learning unit in the learning device according to claim 1 or 2, or as the model in the inference device according to any one of claims 3 to 5.

Images