Training data generation device, training data generation method, and program

Abstract

While it is effective for speech synthesis devices in voice dialogue systems to learn using the dialogue of a target speaker, it is extremely difficult to prepare a large volume of dialogue from the target speaker under various circumstances. [Solution] The learning data generation device according to the disclosed technology is a device that generates a group of dialogue audio data of a target speaker as data used for training a speech synthesizer so that a speech dialogue system can speak using the target speaker's synthesized voice, and includes a source data acquisition unit, a voice quality conversion unit, and a learning data output unit. The source data acquisition unit obtains a group of dialogue audio data spoken by various speakers as source data. The voice quality conversion unit converts the voice quality of each of the source dialogue audio data included in the source data to approximate the voice quality of the target speaker, thereby obtaining converted dialogue audio data. The learning data output unit obtains a group of converted dialogue audio data and outputs learning data.

Description

Technical Field

[0001] The disclosed technology relates to a technique for generating learning data used for learning a voice synthesis device included in a voice dialogue system so that the voice dialogue system can speak in a synthesized voice of a target speaker who is a single speaker.

Background Art

[0002] In recent years, voice dialogue systems have been used in various services. It is desirable that the utterances made by the voice dialogue system be human-like utterances. In particular, it is required that the utterances made by voice dialogue systems such as communication robots and CG dialogue agents be more human-like utterances. A human-like utterance is an utterance in which the interlocutor hardly feels the unnaturalness as a human utterance, and is an utterance with a high degree of naturalness as a human utterance.

[0003] As an element of the naturalness of human utterances, first, the naturalness as a response, that is, the appropriateness of the utterance content as a response to the utterance made by the interlocutor, can be cited. Regarding the naturalness of the responses of the utterances made by the voice dialogue system, it has been greatly improved in recent years due to the remarkable progress of large language models.

[0004] As an element of the naturalness of human utterances, the naturalness as dialogue voice (the voice of the utterance in the dialogue), that is, the high degree of compatibility of the way of speaking with the situation and context, can also be cited. In actual human-to-human conversations, for example, the content and way of speaking change according to the situation and context of the conversation, such as when speaking politely in a meeting and when speaking casually among friends. On the other hand, the utterances of the voice dialogue system are in a uniform way of speaking in a reading tone, and the degree of compatibility with the situation and context is not necessarily high. One of the reasons for this is considered to be that general voice synthesis devices are trained using a corpus in a reading tone prepared for voice synthesis.

[0005] As prior documents using dialogue voice for the learning of the voice synthesis device, there are Non-Patent Document 1 and Non-Patent Document 2. Non-patent document 1 describes a technique for fine-tuning a model trained on a text-to-speech corpus using speech from a dialogue corpus. Non-patent document 2 describes a technique for training a speech synthesis device using dialogue audio from unspecified speakers. Non-patent documents 1 and 2 demonstrate that using dialogue speech for training a speech synthesizer is effective in improving the naturalness of the speech produced by the speech synthesizer as dialogue speech. [Prior art documents] [Non-patent literature]

[0006] [Non-Patent Document 1] Takahisa Iizuka, Daiki Mori, "The influence of an agent speaking with synthesized speech using a spontaneous speech corpus on the behavior of the conversation partner," Proceedings of the Spring Meeting of the Acoustical Society of Japan, pp. 1283-1284, 2021. [Non-Patent Document 2] Hirotaka Nishino, Daiki Mori, "Improving the Quality of End-to-End Dialogue Speech Synthesis by Using Multiple Natural Dialogue Speech Corpora," Proceedings of the Spring Meeting of the Acoustical Society of Japan, pp. 1085-1086, 2022. [Overview of the project] [Problems that the invention aims to solve]

[0007] Voice dialogue systems used in communication robots, CG dialogue agents, and other similar systems need to speak in the voice of a predetermined target speaker. Therefore, the speech synthesizer included in such a system needs to synthesize a target speaker's voice that is highly natural as dialogue speech. From the findings of Non-Patent Documents 1 and 2, it is clear that training the speech synthesizer included in a voice dialogue system using a large volume of dialogue audio from the target speaker under various circumstances is effective. However, in reality, preparing a large volume of dialogue audio from the target speaker under various circumstances is extremely difficult. This problem is not resolved in either Non-Patent Document 1 or Non-Patent Document 2. The present invention aims to generate training data for training a speech synthesis device used in a speech dialogue system, which includes simulated dialogue audio of a target speaker under various circumstances. [Means for solving the problem]

[0008] To solve the above problems, the learning data generation device relating to the disclosed technology is a learning data generation device that generates a group of dialogue voice data of a target speaker as data (learning data) to be used to train the speech synthesis device of a speech dialogue system so that the speech dialogue system can speak using the synthesized voice of a single speaker (target speaker), and includes a source data acquisition unit, a voice quality conversion unit, and a learning data output unit. The source data acquisition unit obtains a group of dialogue audio data (source dialogue audio data) spoken by various speakers as source data. The voice quality conversion unit converts each of the source dialogue audio data included in the source data to approximate the voice quality of the target speaker, thereby obtaining converted dialogue audio data. The training data output unit acquires a group consisting of converted dialogue speech data and outputs the training data. [Effects of the Invention]

[0009] According to the disclosed technology, since simulated dialogue audio data of the target speaker can be generated without restriction as long as dialogue data is available, it becomes possible to generate training data that includes simulated dialogue audio of the target speaker under various circumstances, which can be used as training data for a speech synthesizer used in a speech dialogue system. Furthermore, by using a speech synthesis device trained with the learning data generated by the present invention in a speech dialogue system, the speech dialogue system can make utterances in the synthesized voice of the target speaker that correspond to the dialogue situation. [Brief explanation of the drawing]

[0010] [Figure 1] Functional block diagrams of the learning data generation device and voice dialogue system according to the first, second, and third embodiments. [Figure 2] A flowchart illustrating the operation of the learning data generation device according to the first, second, and fourth embodiments. [Figure 3] A flowchart illustrating the operation of the voice dialogue system according to the first, second, third, and fourth embodiments. [Figure 4] A flowchart illustrating the operation of the learning data generation device according to the third embodiment. [Figure 5] Functional block diagram of the learning data generation device and voice dialogue system according to the fourth embodiment. [Figure 6] A flowchart illustrating the operation of the voice quality conversion learning data acquisition unit and the voice quality conversion learning unit according to the fourth embodiment. [Figure 7] A diagram illustrating an example of a computer's functional configuration. [Modes for carrying out the invention]

[0011] The embodiments of the disclosed technology will be described in detail below. Components with the same function will be numbered identically, and redundant explanations will be omitted.

[0012] [First Embodiment] The learning data generation device according to the first embodiment generates learning data used for training a voice synthesis device included in a voice dialogue system from source data in order to cause a target speaker, who is a single speaker in the voice dialogue system, to speak using synthesized voice. The source data is a group of dialogue voice data uttered by various speakers. The learning data is a group of converted dialogue voice data obtained by performing voice quality conversion on each of the dialogue voice data included in the source data so as to approximate the voice quality of the target speaker.

[0013] FIG. 1 is a functional block diagram showing a configuration example of a learning data generation device 100 according to the first embodiment and a configuration example of a voice dialogue system 200 including a voice synthesis device 2200 that uses the learning data generated by the learning data generation device 100 for learning. FIG. 2 is a flowchart showing the operation of the learning data generation device 100, that is, a flowchart of the learning data generation method. Hereinafter, first, each part of the learning data generation device 100 according to the first embodiment and the operation of each part will be described according to FIGS. 1 and 2, and then each part of the voice dialogue system 200 and the operation of each part will be described.

[0014] [Learning Data Generation Device] The learning data generation device 100 includes a source data acquisition unit 110, a voice quality conversion unit 120, and a learning data output unit 130. The voice quality conversion unit 120 further includes a voice quality converter 1200.

[0015] <Processing of Source Data Acquisition Unit> The source data acquisition unit 110 acquires input data for the learning data generation device 100 and transmits it to the voice quality conversion unit 120. Specifically, the source data acquisition unit 110 obtains a group of dialogue voice data uttered by various speakers as source data (step S110). Further, the source data acquisition unit 110 outputs the source data to the voice quality conversion unit 120.

[0016] [Supplementary Explanation Regarding Dialogue Voice Data] A supplementary explanation will be given regarding the dialogue voice data. Dialogue audio data refers to the audio data of each utterance in a dialogue. A group of dialogue audio data consisting of utterances by various speakers refers to the audio data of a dialogue conducted by multiple speakers. Therefore, the audio data in question contains the audio data of each utterance in that dialogue, that is, each dialogue audio data in that dialogue. Hereafter, the dialogue audio data included in the source data will be referred to as the source dialogue audio data.

[0017] For audio data of dialogues involving multiple speakers, for example, audio data recorded from video chats can be used. In the inventors' experiment, digital audio data of one-on-one casual conversations using a web conferencing system, recorded at a sampling frequency of 16 kHz, was used as audio data of dialogues involving multiple speakers. Specifically, this one-on-one casual conversation consisted of 105 pairs of dialogues in which 15 speakers each spoke about their hobbies for 20 minutes. The number of utterances and responses included in these 105 pairs of dialogues was approximately 40,000.

[0018] The group consisting of dialogue audio data spoken by various speakers is not limited to those used in the aforementioned examples or experiments by the inventors, but can be any audio data of dialogues conducted by multiple speakers. Furthermore, the number of dialogue audio data included in the group consisting of dialogue audio data spoken by various speakers is not limited to the number used in the aforementioned experiments by the inventors, but can be any amount that is considered "large" by those skilled in the art, that is, an amount sufficient to adequately train the speech synthesizer, or at least the minimum amount necessary for training the speech synthesizer.

[0019] <<Supplementary information regarding obtaining source data>> This section provides supplementary information regarding the acquisition of source data by the training data generation device. The input data for the learning data generation device 100 may be recorded on a portable recording medium such as a magnetic recording device, optical disc, magneto-optical recording medium, or semiconductor memory, or it may be recorded on a recording device on a network such as a website, or it may be input sequentially, such as from a web conferencing system via a network.

[0020] If the input data for the learning data generation device 100 is recorded on a portable recording medium, the source data acquisition unit 110 is equipped with or capable of connecting a recording medium reader, and the source data acquisition unit 110 can obtain the source data by reading the input data from the recording medium using the reader. When input data to the learning data generation device 100 is received via a network, the source data acquisition unit 110 is equipped with or capable of connecting to an interface device that connects to the network, and the source data acquisition unit 110 can obtain the source data by acquiring the input data using the interface device.

[0021] When input data is directly input to the learning data generation device 100, the source data acquisition unit 110 is equipped with an interface for receiving data input, and the source data acquisition unit 110 can obtain the source data by acquiring the input data using this interface. If input data is input sequentially to the learning data generation device 100, the source data acquisition unit 110 may be provided with a recording unit, and the source data acquisition unit 110 may record the input data for a predetermined time in the recording unit, and obtain the input data for the predetermined time recorded in the recording unit as source data. Alternatively, if input data is input sequentially to the learning data generation device 100, the source data acquisition unit 110 may obtain the input data sequentially as is as source data.

[0022] <<Text-included dialogue audio data>> In the inventors' experiment, to train the speech synthesizer 2200, they acquired training data consisting of dialogue audio data plus text transcription of the dialogue audio along with time information. Therefore, similar to this experiment, the input data to the training data generation device 100 may be a set of data consisting of a group of dialogue audio data spoken by various speakers and text transcription of the dialogue audio with time information. In this case, the source data acquisition unit 110 obtains a group of dialogue audio data spoken by various speakers as source data, and also obtains text with time information corresponding to the source data (step S110A). When the source data acquisition unit 110 performs step S110A, the source data acquisition unit 110 outputs the source data to the voice quality conversion unit 120, and also outputs text with time information corresponding to the source data to the training data output unit 130, as shown by the dashed line in Figure 1.

[0023] <<Time-series audio data>> The input data to the learning data generation device 100 may be a single time-series audio data containing audio data of a dialogue performed by multiple speakers. Similarly, the source data obtained by the source data acquisition unit 110 may be a single time-series audio data containing audio data of a dialogue performed by multiple speakers.

[0024] The above is an explanation of the processing in the <source data acquisition unit>.

[0025] <Processing of the voice quality conversion section> The voice quality conversion unit 120 receives the source data output by the source data acquisition unit 110. The voice quality conversion unit 120 converts each of the source dialogue audio data so that it approaches the voice quality of the target speaker, thereby obtaining the converted dialogue audio (step S120). The voice quality conversion unit 120 also outputs each converted dialogue audio data to the learning data output unit 130.

[0026] The voice conversion performed by the voice conversion unit 120 can be any method that converts the input source dialogue audio data to approximate the voice quality of the target speaker and obtains converted dialogue audio data; any known voice conversion technique can be used. An example of a known voice conversion technique is Retrieval-based Voice Conversion (RVC) described in Reference 1. For example, the voice conversion unit 120 may be equipped with a voice converter 1200 that has been trained to convert the input audio data to approximate the voice quality of the target speaker and output converted audio data.

[0027] Reference 1: "Retrieval-based Voice Conversion", [Retrieved October 24, 2024], Internet<https: / / github.com / RVC-Project / Retrieval-based-Voice-Conversion-WebUI / releases> .

[0028] <<Supplementary information regarding voice quality transformation>> I will provide some additional explanation regarding voice quality transformation. "Performing voice conversion to approximate the voice quality of the target speaker" can be achieved, for example, by using a voice converter 1200 that has been trained with the goal of completely converting the voice quality of the input audio data to that of the target speaker. One example of "transforming the voice quality to approximate the target speaker's voice quality" is to transform the voice quality to be as close as possible to that of the target speaker, that is, to transform the voice quality so that it is approximately the same as that of the target speaker.

[0029] When expressing "voice quality conversion" in words, common phrases include "converting the voice quality to match that of the target speaker" or "converting the voice quality so that it is the same as that of the target speaker." "Performing voice conversion to approximate the target speaker's voice quality" is not limited to the examples mentioned above; at the very least, any voice conversion that results in a greater degree of similarity between the voice quality of the converted dialogue audio data and the target speaker's voice quality than the degree of similarity between the voice quality of the source dialogue audio data and the target speaker's voice quality is acceptable.

[0030] <<Text-included dialogue audio data>> Furthermore, if the source data acquisition unit 110 outputs text with time information corresponding to the source data to the learning data output unit 130, the voice quality conversion unit 120 should ensure that the time axis position of each source dialogue audio data and the time axis position of each converted dialogue audio data are the same.

[0031] <<Time-series audio data>> Furthermore, the multiple converted dialogue audio data obtained by the voice quality conversion unit 120 may be included in a single time-series audio data. In this case as well, the voice quality conversion unit 120 obtains the converted dialogue audio data by converting the voice quality of each of the source dialogue audio data so that it approaches the voice quality of the target speaker. For example, if the source data is a single time-series audio data containing audio data of a dialogue performed by multiple speakers, the voice quality conversion unit 120 can obtain a single time-series audio data containing multiple converted dialogue audio data sequentially by sequentially converting the voice quality of the multiple dialogue audio data contained in the source data, which is a single time-series audio data, to approximate the voice quality of the target speaker. In this case, the single time-series audio data obtained by the voice quality conversion unit 120 is originally audio data of a dialogue performed by multiple speakers, but it is a single time-series audio data that includes audio data in which the voice quality of each utterance has been made closer to that of the target speaker.

[0032] The above is an explanation of the <processing of the voice quality conversion section>.

[0033] <Processing of the training data output section> The learning data output unit 130 receives a group of converted dialogue audio data output by the voice quality conversion unit 120. Hereafter, this group of converted dialogue audio data will be collectively referred to as "converted data". The learning data output unit 130 obtains the input converted data as learning data (step S130). The learning data output unit 130 outputs the learning data as output data for the learning data generation device 100.

[0034] The training data, which is the output data of the training data generation device 100, is originally a group of dialogue audio data from multiple speakers, but it is a group of converted dialogue audio data in which the voice quality of each dialogue audio data is made to approximate the voice quality of the target speaker; in other words, it is a group of dialogue audio data from a pseudo-target speaker. The training data, which is the output data of the training data generation device 100, is used as a group of dialogue audio data from the target speaker for training the speech synthesizer 2200.

[0035] <<Text-included dialogue audio data>> When the source data acquisition unit 110 outputs text with time information corresponding to the source data to the training data output unit 130, the training data output unit 130 also receives text with time information corresponding to the source data. In this case, the training data output unit 130 obtains the input converted data plus the text with time information as training data (step S130A).

[0036] <<Time-series audio data>> The learning data obtained by the learning data output unit 130 may be contained in a single time-series audio data. If the converted data is a single time-series audio data, the learning data output unit 130 can simply use the converted data, which is a single time-series audio data, as the learning data, which is also a single time-series audio data. In this case, the single time-series audio data obtained by the learning data output unit 130 is originally audio data of a dialogue performed by multiple speakers, but the voice quality of each utterance has been adjusted to approximate the voice quality of the target speaker. This single time-series audio data contains a pseudo-target speaker dialogue audio data set and is used for training the speech synthesizer 2200 as a single time-series audio data set containing a target speaker dialogue audio data set.

[0037] If the source data is a single time-series audio data, and the converted data is also a single time-series audio data, and the voice quality conversion unit 120 ensures that the time axis position of each source dialogue audio data and the time axis position of each converted dialogue audio data are the same, then the learning data output unit 130 only needs to obtain the converted data, which is a single time-series audio data, plus the input text with time information itself as the learning data.

[0038] The above is an explanation of the processing of the <training data output section>.

[0039] [Voice Interaction System] The voice dialogue system 200 speaks in the synthesized voice of a target speaker as a response to the user's utterance. As shown in Figure 1, the voice dialogue system 200 includes a user utterance acquisition unit 210 and a system utterance generation unit 220. The system utterance generation unit 220 includes a speech synthesizer 2200 that obtains the synthesized voice of a target speaker corresponding to the input text. The voice dialogue system 200 performs steps S210 and S220 shown in Figure 3.

[0040] <Acquiring user utterances> The user utterance acquisition unit 210 receives an audio signal, including the user's utterance, which is the input to the voice dialogue system 200. The user utterance acquisition unit 210 performs speech recognition and utterance unit detection on the input audio signal to obtain user utterance text, which is the text of one unit of the user's utterance (step S210). The user utterance acquisition unit 210 outputs the user utterance text to the system utterance generation unit 220. Well-known methods can be used for speech recognition and utterance unit detection.

[0041] <Generating system utterances> The system utterance generation unit 220 receives the user utterance text output by the user utterance acquisition unit 210 as input. The system utterance generation unit 220 generates system utterance text, which is the text of the utterance that responds to the input user utterance text, and generates system utterance, which is the synthesized voice of the target speaker corresponding to the system utterance text (step S220). The system utterance generation unit 220 outputs the system utterance as the output of the voice dialogue system 200. The system utterance, which is the synthesized speech of the target speaker corresponding to the system utterance text, is generated by the speech synthesizer 2200 included in the system utterance generation unit 220. For this purpose, prior to the operation of the system utterance generation unit 220, the speech synthesizer 2200 performs training using the training data output by the training data generation unit 100 so that it can generate the synthesized speech of the target speaker. Known methods can be used for training. For example, as the speech synthesizer 2200, a neural network can be used in which the known speech synthesis model Tacotron 2 is used as the Mel spectrogram generation unit and HiFi-GAN is used as the waveform generation unit. In this example, the speech synthesizer 2200 is trained using backpropagation with Mel spectrograms output for Tacotron 2 and Generative adversarial networks with speech waveforms output for HiFi-GAN.

[0042] Furthermore, the target speaker's synthesized voice refers to the voice output by the speech synthesizer 2200, which has been trained to produce a voice quality as close as possible to that of the target speaker. In other words, it is a synthesized voice with a voice quality that is approximately the same as that of the target speaker.

[0043] The above is a description of the first embodiment.

[0044] [Second Embodiment] In the first embodiment, the converted data obtained by the voice conversion unit 120 may include low-quality converted dialogue audio data due to voice conversion errors. Therefore, in the second embodiment, instead of obtaining all of the converted dialogue audio data obtained by voice conversion as training data, as in the first embodiment, a group consisting of converted dialogue audio data that meets a predetermined quality standard is obtained as training data. This form will be described as the second embodiment, focusing on the differences from the first embodiment.

[0045] The learning data generation device 100 of the second embodiment, like the learning data generation device 100 of the first embodiment, includes a source data acquisition unit 110, a voice quality conversion unit 120, and a learning data output unit 130, as shown in Figure 1. The learning data generation device 100 of the second embodiment performs steps S110, S120, and S131, as shown in Figure 2.

[0046] Step S110 performed by the source data acquisition unit 110 in the second embodiment is the same as step S110 performed by the source data acquisition unit 110 in the first embodiment. Also as in the first embodiment, the source data acquisition unit 110 in the second embodiment may perform step S110A instead of step S110. Step S120 performed by the voice quality conversion unit 120 of the second embodiment is the same as step S120 performed by the voice quality conversion unit 120 of the first embodiment.

[0047] Step S131 performed by the learning data output unit 130 of the second embodiment is different from step S130 performed by the learning data output unit 130 of the first embodiment. Therefore, the following description will focus on the differences between the learning data output unit 130 of the second embodiment and the learning data output unit 130 of the first embodiment.

[0048] <Processing of the training data output section> The learning data output unit 130 receives the converted dialogue voice data output by the voice quality conversion unit 120. The learning data output unit 130 obtains a group of converted dialogue voice data that meet predetermined quality criteria from the input data as learning data (step S131). The learning data output unit 130 outputs the learning data as output data for the learning data generation device 100. The training data, which is the output data of the training data generation device 100, is originally a group of dialogue audio data from multiple speakers, but the voice quality of each dialogue audio is made to approximate the voice quality of the target speaker and the group of dialogue audio data that meet predetermined quality standards. In other words, it is a group of dialogue audio data from a pseudo target speaker and is used as a group of dialogue audio data from the target speaker for training the speech synthesizer 2200.

[0049] We will provide supplementary explanations regarding the specified quality standards. The specified quality standard is a criterion for obtaining a group of high-quality data from the input converted dialogue speech data to be used as training data. For example, the learning data output unit 130 may obtain a quality evaluation value for each of the input converted dialogue audio data and obtain a group of converted dialogue audio data whose quality evaluation value is greater than or equal to a predetermined standard value as learning data. Alternatively, it may obtain a group of converted dialogue audio data consisting of the number of data required for learning, starting with those with the highest quality evaluation values, as learning data. Alternatively, it may obtain a group of converted dialogue audio data consisting of those with the highest quality evaluation values ​​selected as learning data, so that their proportion to the total number of input converted dialogue audio data is a predetermined proportion.

[0050] For quality evaluation, any evaluation value representing the quality of the speech can be used; for example, the UTokyo-SaruLab MOS Prediction System (UTMOS) described in Reference 2 can be used. Alternatively, a speaker evaluation value, such as an evaluation value representing the degree of similarity to the target speaker's voice quality, may also be used as a quality evaluation value.

[0051] Reference 2: T. Saeki, et al., "UTMOS: UTokyo-Sarulab System for voiceMOS Challenge 2022", arXiv:2204.02152, 2022.

[0052] <<Time-series audio data>> The learning data obtained by the learning data output unit 130 may be contained in a single time-series audio data, similar to the learning data obtained by the learning data output unit 130 in the first embodiment. For example, the learning data output unit 130 may check whether each converted dialogue audio data contained in the converted data, which is a single input time-series audio data, meets a predetermined quality standard, and replace the portion of the converted dialogue audio data that does not meet the quality standard with silence to obtain a single time-series audio data as the learning data. Alternatively, for example, the learning data output unit 130 may check whether each converted dialogue audio data contained in the converted data, which is a single input time-series audio data, meets a predetermined quality standard, and delete the portion of the converted dialogue audio data that does not meet the quality standard to obtain a single time-series audio data as the learning data.

[0053] <<Text-included dialogue audio data>> When the source data acquisition unit 110 outputs text with time information corresponding to the source data to the learning data output unit 130, the learning data output unit 130 also receives the text with time information corresponding to the source data. In this case, the learning data output unit 130 obtains as learning data a group of converted dialogue voice data that meet predetermined quality standards, plus the text with time information (step S131A). If the source data is a single time-series audio data, and the converted data is also a single time-series audio data, and the voice quality conversion unit 120 ensures that the time axis position of each source dialogue audio data is the same as the time axis position of each converted dialogue audio data, then the learning data output unit 130 can, for example, perform the operation of either the first or second example below.

[0054] (Example 1) The learning data output unit 130 checks whether each converted dialogue audio data included in the converted data, which is a single time-series audio data input, meets a predetermined quality standard. It then obtains a single time-series audio data by replacing the portion of the converted dialogue audio data that does not meet the quality standard with silence, and adds the input text with time information itself to the obtained time-series audio data to obtain the learning data.

[0055] (Example 2) The learning data output unit 130 checks whether each converted dialogue audio data included in the converted data, which is a single time-series audio data input, meets a predetermined quality standard. It then removes the portion of the converted dialogue audio data that does not meet the quality standard to obtain a single time-series audio data. Finally, it adds the obtained time-series audio data to a modified text with time information, which is created by shifting the time information of the input text with time information forward by the amount of the portion that was removed. This modified text is then used as learning data.

[0056] The above is a description of the second embodiment. According to the learning data generation device 100 of the second embodiment, it is possible to obtain a group of high-quality converted dialogue speech data obtained by voice quality conversion, that is, a group that excludes as much as possible low-quality converted dialogue speech data caused by voice quality conversion errors, as learning data. Therefore, if a speech synthesizer trained using the learning data obtained by the learning data generation device 100 of the second embodiment is used in a speech dialogue system, it is possible to produce synthesized speech that is closer to the target speaker's voice quality and that is appropriate to the dialogue situation, compared to when a speech synthesizer trained using the learning data obtained by the learning data generation device 100 of the first embodiment is used in the speech dialogue system.

[0057] [Third Embodiment] When the gender of the speaker in the source dialogue audio and the target speaker are different, for example, when the speaker in the source dialogue audio is male and the target speaker is female, using only known voice conversion techniques may not result in a sufficiently voice-converted converted dialogue audio. Therefore, in the third embodiment, when the gender of the speaker in the source dialogue audio and the target speaker are different, the converted dialogue audio data is obtained by changing the key in addition to voice conversion. This form will be described as the third embodiment, focusing on the differences from the first and second embodiments.

[0058] The learning data generation device 100 of the third embodiment, like the learning data generation device 100 of the first and second embodiments, includes a source data acquisition unit 110, a voice quality conversion unit 120, and a learning data output unit 130, as shown in Figure 1. The learning data generation device 100 of the third embodiment performs the steps shown in Figure 4. Figure 4(a) shows a flow in which, in addition to steps S110, S120, and S130 of Figure 2, either or both of steps S1201 and S1202 are performed.

[0059] Step S110 performed by the source data acquisition unit 110 of the third embodiment is the same as step S110 performed by the source data acquisition unit 110 of the first and second embodiments. As with the first and second embodiments, the source data acquisition unit 110 of the third embodiment may perform step S110A instead of step S110.

[0060] The voice quality conversion unit 120 of the third embodiment performs either or both of steps S1201 and S1202, in addition to step S120 performed by the voice quality conversion unit 120 of the first and second embodiments.

[0061] Step S130 performed by the learning data output unit 130 in the third embodiment is the same as step S130 performed by the learning data output unit 130 in the first embodiment, step S130A performed by the learning data output unit 130 in the third embodiment is the same as step S130A performed by the learning data output unit 130 in the first embodiment, step S131 performed by the learning data output unit 130 in the third embodiment is the same as step S131 performed by the learning data output unit 130 in the second embodiment, and step S131A performed by the learning data output unit 130 in the third embodiment is the same as step S131A performed by the learning data output unit 130 in the second embodiment. Therefore, the following description will focus on the differences between the voice conversion unit 120 of the third embodiment and the voice conversion unit 120 of the first and second embodiments.

[0062] <Processing of the voice quality conversion section> The voice quality conversion unit 120 receives the source data output by the source data acquisition unit 110. In addition to step S120, the voice quality conversion unit 120 performs either or both of steps S1201 and S1202, which will be described later, to obtain each converted dialogue voice data from each source dialogue voice data. Steps S1201 and S1202 performed by the voice quality conversion unit 120 are as follows.

[0063] Figure 4(b) shows the detailed flow of step 1201. The voice conversion unit 120 determines whether the gender of the speaker of the source dialogue audio and the target speaker match for each input source dialogue audio data. If it is determined that they do not match (No in step S1201-1), it changes the key of the source dialogue audio data to make it subject to the operation in step S120 (step S1201-2). If it is determined that they match (Yes in step S1201-1), it makes the input source dialogue audio data itself subject to the operation in step S120.

[0064] Figure 4(c) shows the detailed flow of step 1202. The voice quality conversion unit 120 determines whether the gender of the speaker of the source dialogue audio and the target speaker match for each input source dialogue audio data. If it is determined that they do not match (No in step S1202-1), it changes the key of the converted dialogue audio data to obtain the final converted dialogue audio data from the voice quality conversion unit 120 (step S1202-2). If it is determined that they do match (Yes in step S1202-1), it uses the converted dialogue audio data obtained in step S120 itself as the final converted dialogue audio data from the voice quality conversion unit 120.

[0065] Let me provide some additional information about changing keys. Specifically, the key change performed by the voice quality conversion unit 120 involves raising the key when the speaker of the source dialogue audio is male and the target speaker is female, and lowering the key when the speaker of the source dialogue audio is female and the target speaker is male. For example, if the voice quality conversion unit 120 performs only one of step S1201 or step S1202, the action of raising the key should be to raise it by one octave, and the action of lowering the key should be to lower it by one octave. Also, for example, if the voice quality conversion unit 120 performs both step S1201 and step S1202, the action of raising the key in step S1201 and the action of raising the key in step S1202 should be combined to raise the key by one octave, and the action of lowering the key in step S1201 and the action of lowering the key in step S1202 should be combined to lower the key by one octave.

[0066] In other words, in addition to the operation in step S120, the voice conversion unit 120 also lowers the key of the source dialogue voice and / or the key of the converted dialogue voice if the speaker of the source dialogue voice is female and the target speaker is male, and raises the key of the source dialogue voice and / or the key of the converted dialogue voice if the speaker of the source dialogue voice is male and the target speaker is female. However, if the source dialogue voice is female and the target speaker is female, or if the source dialogue voice is male and the target speaker is male, the voice conversion unit 120 does not change the key of the source dialogue voice or the key of the converted dialogue voice, and only performs step S120.

[0067] The operation of changing the key specifically involves changing the fundamental frequency contained in the audio signal. Since general voice conversion methods target the conversion of the speech spectrum, when the voice conversion unit 120 uses a general voice conversion method, the perceived pitch of the sound is not converted. Therefore, by using the operation of changing the fundamental frequency contained in the audio signal in conjunction with voice conversion, it becomes possible to synthesize speech that has a speaker-like quality close to that of the target speaker.

[0068] The above describes the third embodiment. According to the learning data generation device 100 of the third embodiment, high-quality voice quality conversion can be performed even when the gender of the speaker of the source dialogue speech and the target speaker are different, making it possible to obtain high-quality learning data that is more like the voice of the target speaker. Therefore, if a speech synthesizer trained using the learning data obtained by the learning data generation device 100 of the third embodiment is used in a speech dialogue system, it becomes possible to produce synthesized speech that is closer to the speaker identity of the target speaker and that is appropriate to the dialogue situation, compared to when a speech synthesizer trained using the learning data obtained by the learning data generation device 100 of the first or second embodiment is used in the speech dialogue system.

[0069] [Fourth Embodiment] As described in the first embodiment, the voice conversion unit 120 only needs to be equipped with a voice converter 1200 that has been trained to convert the input dialogue audio data to approximate the voice quality of the target speaker and output the converted dialogue audio data. In the fourth embodiment, the training of the voice converter will be described.

[0070] The learning data generation device 100 of the fourth embodiment learns a voice converter using a group of speeches uttered by a target speaker before the operation of the learning data generation device 100 of the first to third embodiments. Figure 5 shows a functional block diagram of the learning data generation device 100 of the fourth embodiment. The learning data generation device 100 in Figure 5 also includes a voice quality conversion learning data acquisition unit 150 and a voice quality converter learning unit 160, which were not included in the learning data generation device 100 of the first to third embodiments. Before performing the operation shown in Figure 2, the learning data generation device 100 of the fourth embodiment performs steps S150 and S160 shown in Figure 6. Below, the voice quality conversion learning data acquisition unit 150 and the voice quality converter learning unit 160 included in the learning data generation device 100 of the fourth embodiment and their operations will be described.

[0071] <Processing of data acquisition unit for voice quality conversion learning> The voice quality conversion learning data acquisition unit 150 acquires a group of speeches uttered by the target speaker that are input to the learning data generation device 100, and transmits them to the voice quality conversion learning unit 160 as voice quality conversion learning data. Specifically, the voice conversion learning data acquisition unit 150 obtains a group of speeches uttered by the target speaker as voice conversion learning data (step S150). The voice conversion learning data acquisition unit 150 outputs the voice conversion learning data to the voice conversion converter learning unit 160.

[0072] In the inventors' experiment, 4,700 utterances from BASIC5000, a subset of the JSUT (see Reference 3), a Japanese text-to-speech corpus by a single female speaker, were used as the target speaker. In other words, the inventors' experiment used a single female speaker from JSUT as the target speaker.

[0073] Reference 3: R. Sonobe et al., "JSUT CORPUS: FREE LARGE-SCALE JAPANESE SPEECH CORPUS FOR END-TO-END SPEECH SYNTHESIS", arXiv:1711.00354, 2017.

[0074] Naturally, the group of sounds uttered by the target speaker is not limited to those used in the aforementioned experiments by the inventors, nor is it limited to a group of read-aloud sounds; it can be any group of sounds uttered by a single target speaker. Furthermore, the number of utterances included in the group consisting of speech uttered by the target speaker is not limited to the number in the aforementioned experiment by the inventors, but can be any quantity that is considered "large" among those skilled in the art, that is, any quantity that allows for sufficient training of the voice converter, or at least the minimum quantity necessary for training the voice converter.

[0075] <Processing in the voice quality converter learning unit> The voice quality converter learning unit 160 receives voice quality conversion learning data output by the voice quality conversion learning data acquisition unit 150. The voice quality converter learning unit 160 learns the voice quality converter using the input voice quality conversion learning data and obtains a voice quality converter that converts the input speech data to approximate the voice quality of the target speaker and outputs the converted speech data (step S160). The voice quality converter obtained by the voice quality converter learning unit 160 is output to the voice quality conversion unit 120. The voice quality conversion unit 120 uses the voice quality converter output by the voice quality converter learning unit 160 as the voice quality converter 1200.

[0076] In experiments conducted by the inventors, since the voice converter 2200 in the voice conversion unit 120 uses RVC, the voice converter learning unit 160 learned RVC using 4700 utterances from BASIC5000, a subset included in JSUT, over 30 learning epochs. The voice quality converter learning unit 160 is not limited to the method used in the experiments conducted by the inventors mentioned above. Any learning method and number of learning epochs may be used to enable the voice quality converter used in the voice quality converter unit 120 to convert the input speech data to approximate the voice quality of the target speaker and output the converted speech data.

[0077] The above is a description of the fourth embodiment.

[0078] [Programs, recording media] The functions realized by the components described herein may be implemented in a circuitry or processing circuitry, including general-purpose processors, application-specific processors, integrated circuits, ASICs (Application Specific Integrated Circuits), CPUs (a Central Processing Unit), conventional circuits, and / or combinations thereof, programmed to realize the functions described herein. A processor includes transistors and other circuits and is considered a circuitry or processing circuitry. A processor may be a programmed processor that executes a program stored in memory.

[0079] In this specification, circuitry, unit, and means are hardware programmed to perform or execute the functions described herein. Such hardware may be any hardware disclosed herein, or any hardware known to be programmed to perform or execute the functions described herein.

[0080] If the hardware is a processor that is considered to be a type of circuitry, then the circuitry, means, or unit is a combination of hardware and software used to constitute the hardware and / or processor.

[0081] The various processes described above can be carried out by loading a program that executes each step of the above method into the recording unit 2020 of the computer 2000 shown in Figure 7, and then causing the control unit 2010, input unit 2030, output unit 2040, display unit 2050, etc. to operate.

[0082] The program describing this process can be recorded on a computer-readable recording medium. Any computer-readable recording medium can be used, such as a magnetic recording device, optical disc, magneto-optical recording medium, or semiconductor memory.

[0083] Furthermore, this program may be distributed, for example, by selling, transferring, or lending portable recording media such as DVDs or CD-ROMs on which the program is recorded. Alternatively, the program may be stored in the storage device of a server computer and distributed by transferring the program from the server computer to other computers via a network.

[0084] A computer executing such a program may, for example, first store the program recorded on a portable storage medium or a program transferred from a server computer in its own storage device. Then, when processing is to be executed, the computer reads the program stored on its own storage medium and executes the processing according to the read program. Alternatively, the computer may directly read the program from the portable storage medium and execute the processing according to that program, or it may sequentially execute the processing according to the received program each time a program is transferred to it from a server computer. Furthermore, the processing may be executed by a so-called ASP (Application Service Provider) type service, where the processing function is realized only by execution instructions and result acquisition, without transferring the program from the server computer to this computer. Furthermore, the processing may be executed using a so-called SaaS (Software as a Service) type service, where a part of the server computer is made available to the user along with the program. In this form, the program includes information used for processing by an electronic computer that is equivalent to a program (data that is not a direct instruction to the computer but has the property of defining the computer's processing).

[0085] Furthermore, in this configuration, the device is configured by executing a predetermined program on a computer, but at least a part of these processes may be implemented in hardware. [Industrial applicability]

[0086] This invention can be used in voice dialogue systems. [Explanation of Symbols]

[0087] 100 Training Data Generator 110 Source data acquisition unit 120 Voice quality conversion unit 1200 Voice Converter 130 Training data output section 150 Voice quality conversion learning data acquisition unit 160 Voice Quality Converter Learning Unit 200 Voice Interaction Systems 210 User utterance acquisition unit 220 System speech generation unit 2200 Speech Synthesizer 2000 Computer 2010 Control Unit 2020 Records Department 2030 Input Section 2040 Output Section 2050 Display section

Claims

1. A learning data generation device that generates a set of dialogue voice data of a target speaker as data (learning data) to be used for training the speech synthesis device of a voice dialogue system, in order for the voice dialogue system to speak using the synthesized voice of a single speaker (target speaker), A source data acquisition unit obtains a group of dialogue audio data (source dialogue audio data) spoken by various speakers as source data, A voice quality conversion unit obtains converted dialogue audio data by converting each of the source dialogue audio data included in the source data so as to approximate the voice quality of the target speaker, A learning data output unit that acquires the group consisting of the converted dialogue voice data and outputs the learning data, A training data generation device that includes [specific data].

2. A learning data generation device according to claim 1, The learning data output unit outputs a group of converted dialogue voice data that meets predetermined quality standards as the learning data. A device for generating training data.

3. A learning data generation device according to claim 1, The voice conversion unit lowers the key of the source dialogue audio data and / or the key of the converted dialogue audio data if the source dialogue audio data is female and the target speaker is male; raises the key of the source dialogue audio data and / or the key of the converted dialogue audio data if the source dialogue audio data is male and the target speaker is female; and maintains the keys of the source dialogue audio data and the converted dialogue audio data if the source dialogue audio data is female and the target speaker is female, and if the source dialogue audio data is male and the target speaker is male. A device for generating training data.

4. A learning data generation device according to claim 3, The key change of the source dialogue audio data is performed before the voice quality conversion. The key change of the converted dialogue audio data is performed after the voice quality conversion. A device for generating training data.

5. A learning data generation device according to claim 3, The difference between the key of the source dialogue audio data output by the source data acquisition unit and the key of the converted dialogue audio data input to the learning data output unit is constant regardless of the number of times the key has been changed. A device for generating training data.

6. A learning data generation device according to claim 1, A voice conversion learning data acquisition unit obtains voice conversion learning data, which is the voice spoken by the target speaker. The system further includes a voice quality converter learning unit that learns a voice quality converter using the aforementioned voice quality conversion learning data, The voice conversion unit converts the source dialogue audio data into the converted dialogue audio data using the voice converter learned by the voice converter learning unit. A device for generating training data.

7. A method for generating training data, which generates a group of dialogue voices of a target speaker as training data (training data) to be used for training the speech synthesis device of a speech dialogue system, in order for a speech dialogue system to speak using the synthesized voice of a single speaker (target speaker), The data acquisition unit acquires a group of dialogue audio (source dialogue audio) spoken by various speakers as source data. The voice quality conversion unit converts each of the source dialogue audio included in the source data to approximate the voice quality of the target speaker and obtains the converted dialogue audio. The learning data output unit acquires the group consisting of the converted dialogue audio and outputs the learning data. Method for generating training data.

8. A program for causing a computer to function as a learning data generation device according to any one of claims 1 to 6.

Images