Voice conversion apparatus, voice conversion method, computer program product, and recording medium

Abstract

A voice conversion apparatus (1) includes: an input unit (11) that inputs designation of a conversion target voice; an extraction unit (12) that extracts time-series data including phonemes and pitch from a voice signal of a conversion source voice; an adjustment unit (13) that adjusts the height of the pitch to match the height of the conversion target voice designated; and a generation unit (14) that generates a voice signal in which the conversion target voice designated is synthesized in a deep learning model that synthesizes a voice of a designated person by inputting phonemes and pitch in time-series order to voice data capable of learning voices of multiple persons.

Description

Technical Field

[0001] This invention relates to speech conversion devices, speech conversion methods, computer program products, and recording media. Background Technology

[0002] With the expansion of services that publish images of computer-graphic characters (hereinafter referred to as avatars) operating in virtual space, there is a desire for voice conversion that matches the appearance of the avatar. For example, even if the gender and age of the publisher operating the avatar do not match the appearance of the avatar, the publisher's voice can be converted into a voice that matches the appearance of the avatar.

[0003] The quality of speech synthesis, including speech conversion, has improved significantly in recent years due to advancements in deep learning technology. Among these advancements, WaveNet, a deep learning model that employs an autoregressive method to generate speech samples bit by bit, is able to synthesize speech of almost indistinguishable quality from real speech. While WaveNet boasts high synthesis quality, it suffers from slow synthesis speed, a weakness that has been addressed by models such as Wave RNN.

[0004] Existing technical documents

[0005] Patent documents

[0006] Patent Document 1: Japanese Patent No. 6783475 Summary of the Invention

[0007] The problem that the invention aims to solve

[0008] One method for speech conversion using deep learning involves preparing paired data of source and target speech reading the same text aloud, and using this paired data as training data for speech conversion. However, this method requires recordings of the source speech reading multiple texts, and then using this audio data for deep learning, which is very time-consuming. The need for source speech data in deep learning speech conversion stems from the desire to solve speech conversion directly (end-to-end) through deep learning.

[0009] Furthermore, there is a demand for virtual avatars with the same appearance to speak with the same voice. That is, it is desired that the voice of anyone can be converted into the same voice. Moreover, if it is possible to convert the voice of anyone into the voice of various people, then it is possible to select the voice desired by the publisher as the voice of the virtual avatar, or for one or a few publishers to operate a large number of virtual avatars.

[0010] The present invention was made in view of the aforementioned circumstances, and its purpose is to convert the speech of any person into the speech of various people.

[0011] Methods for solving problems

[0012] One aspect of the speech conversion apparatus of the present invention comprises: an input unit that inputs a specified target speech for conversion; an extraction unit that analyzes the speech signal of the source speech for conversion to extract temporal data including phonemes and pitch; an adjustment unit that adjusts the pitch to match the pitch of the specified target speech for conversion; and a generation unit that generates a speech signal of the specified target speech for conversion by inputting the phonemes and pitch sequentially into a deep learning model capable of learning speech data of multiple people to synthesize the speech of the specified person.

[0013] In one aspect of the speech conversion method of the present invention, a computer inputs a specified target speech for conversion, analyzes the speech signal of the source speech for conversion to extract temporal data containing phonemes and pitches, makes the pitch height consistent with the specified target speech height, and generates a speech signal of the specified target speech by inputting the phonemes and pitches in temporal order into a deep learning model that can learn the speech data of multiple people to synthesize the speech of a specified person.

[0014] Invention Effects

[0015] According to the present invention, regardless of whose voice it is, it can be converted into the voice of various people. Attached Figure Description

[0016] Figure 1 This is a diagram illustrating an example of the structure of the speech conversion device according to this embodiment.

[0017] Figure 2 This is a diagram used to illustrate pitch adjustment.

[0018] Figure 3 This is a diagram used to illustrate the deep learning model of a speech conversion device.

[0019] Figure 4 This diagram illustrates a situation where speech conversion can be performed without limiting the source speech.

[0020] Figure 5 This is a flowchart illustrating an example of the processing flow of a speech conversion device.

[0021] Figure 6 This is a diagram illustrating an example of the structure of a modified version of the voice conversion device according to this embodiment.

[0022] Figure 7 This is an example of a screen displaying a web application that uses a speech-to-text conversion device.

[0023] Figure 8This is a diagram illustrating an example of a structure connecting a speed conversion device to a voice conversion device. Detailed Implementation

[0024] [structure]

[0025] Hereinafter, embodiments of the present invention will be described using the accompanying drawings.

[0026] Reference Figure 1 An example of the structure of the speech conversion device 1 in this embodiment will be described. Figure 1 The voice conversion device 1 shown includes an input unit 11, an extraction unit 12, an adjustment unit 13, and a generation unit 14. Each unit of the voice conversion device 1 can also be configured as a computer equipped with a processing unit, a storage unit, etc., and the processing of each unit is executed by a program. This program is stored in the storage unit of the voice conversion device 1, or it can be recorded on a recording medium such as a disk, optical disk, or semiconductor memory, or provided via a network.

[0027] The input unit 11 inputs a specified target speech for conversion. For example, the input unit 11 can input an identifier or name of the target speech, or it can input an attribute of the target speech (gender, adult voice, child voice, high voice, low voice, etc.). When an attribute of the target speech is input, the input unit 11 selects the target speech corresponding to that attribute from the candidates for target speech.

[0028] The extraction unit 12 receives the speech signal (hereinafter referred to as speech data) of the source speech for conversion, performs speech recognition on the source speech, and extracts temporal data containing phonemes (consonants + vowels) and pitch for each phoneme from the source speech. Pitch also includes speech information such as intonation, tone, and speech length. The extraction unit 12 can read a file containing speech data, input speech data using a microphone (not shown) provided with the speech conversion device 1, or input speech data from a device connected to an external terminal provided with the speech conversion device 1. The extraction unit 12 extracts phonemes and pitch from the speech data using existing speech recognition technology. For example, OpenJTalk can be used for phoneme extraction, and WORLD can be used for pitch extraction. Furthermore, the number of phonemes is determined by the content of the speech data (text content), and the number of pitches is determined by the length of the speech data; therefore, the phonemes and pitches may not be in a one-to-one correspondence.

[0029] The extraction unit 12 can also input an article containing the same content as the speech data along with the speech data. The extraction unit 12 can extract phonemes from the input article and can also use the input article to correct the speech recognition results of the speech data. By inputting both the speech and the article, both the accuracy of phoneme reading and the acquisition of pitch information can be achieved. For example, if incorrect phonemes are recognized due to unclear pronunciation or other reasons, adjustments can be made using the input article.

[0030] The extraction unit 12 sends phonemes to the generation unit 14 in chronological order and sends pitches to the adjustment unit 13. The pitches are adjusted by the adjustment unit 13 and then sent to the generation unit 14.

[0031] like Figure 2 As shown, the adjustment unit 13 performs a linear transformation on the pitch of each phoneme extracted by the extraction unit 12, making the pitch of the source speech to match the pitch of the target speech. For example, the adjustment unit 13 converts a lower pitch to a higher pitch, or vice versa. Furthermore, the pitch of the target speech is known and stored in the storage device provided in the speech conversion apparatus 1. The adjustment unit 13 can calculate the average pitch of each target speech and adjust the average pitch of the source speech to match the average pitch of the target speech.

[0032] The generation unit 14 inputs phonemes and converted pitches into a deep learning model that has learned from speech data of multiple people, and synthesizes a speech signal that is produced using the target speech specified by the input unit 11. The deep learning model maintained by the generation unit 14 outputs a speech signal produced using the speech specified by the input unit 11 when phonemes and pitches are input. For example, a Wave RNN can be used as the deep learning model. When extracting phonemes from the source speech data, the vocalization interval of each phoneme is extracted and attached to each phoneme. Each phoneme and pitch are input into the generation unit 14, so that the generation unit 14 can output speech that maintains the vocalization period of the source speech data. Regarding silent intervals, silent intervals can also be input into the generation unit 14, and silent intervals of the same length can be output.

[0033] The speech conversion device 1 may also include a learning unit 15. The learning unit 15 extracts phonemes and pitches from the speech data of multiple speakers that serve as the target speech for conversion, and learns a deep learning model capable of synthesizing the speech of the multiple speakers from the extracted phonemes and pitches respectively. For example, in this embodiment, phonemes and pitches are extracted from the high-quality speech data of 100 professional speakers, i.e., the JVS corpus, and a deep learning model is learned to synthesize and output the speech of a selected person among the 100 professional speakers when inputting phonemes and pitches. By performing deep learning on the speech of multiple speakers together, even with limited speech data for each speaker, it is possible to synthesize the speech of each speaker with high quality.

[0034] As explained above, in this embodiment, the source speech is decomposed into speaker-independent elements, and the target speech is synthesized from the decomposed elements, thereby enabling speech conversion without changing the waveform of the source speech. Specifically, as... Figure 3 As shown, during speech conversion, phonemes are extracted from the speech data as language information, and pitch and pronunciation timing are extracted as non-language information. The extracted phonemes and pitch are then input into a deep learning model to synthesize the target speech.

[0035] In this embodiment, since the source speech is decomposed into speaker-independent elements before speech synthesis, it is not necessary to learn paired data of the source and target speech, such as... Figure 4 As shown, regardless of whose voice it is, it can be converted into the voices of various people for learning purposes.

[0036] [action]

[0037] Next, refer to Figure 5 The flowchart illustrates the speech conversion operation of the speech conversion device 1.

[0038] In step S11, the speech conversion device 1 inputs the specified target speech for conversion.

[0039] In step S12, the speech conversion device 1 inputs speech data of the target speech to be converted and extracts phonemes and pitches from the speech data.

[0040] In step S13, the speech conversion device 1 converts the pitch extracted in step S12 into the target speech in a manner consistent with the target speech.

[0041] In step S14, the speech conversion device 1 inputs the phonemes and converted pitches into the deep learning model, synthesizes the converted target speech, and outputs it. When outputting the speech of multiple people, steps S13 and S14 are repeated to synthesize multiple converted target speeches.

[0042] [Variation Example]

[0043] Next, refer to Figure 6 An example of the structure of a modified version of the speech conversion device 1 of this embodiment will be described. Figure 6 The speech conversion device 1 shown includes an input unit 11, an adjustment unit 13, a generation unit 14, a phoneme acquisition unit 16, and a pitch generation unit 17. Figure 6 The voice conversion device 1 and Figure 1The speech conversion device 1 differs from the other in that it has a phoneme acquisition unit 16 and a pitch generation unit 17 instead of an extraction unit 12. Instead of inputting speech data, it inputs text and outputs a speech signal of the specified target speech.

[0044] The input unit 11 inputs the specified target language for conversion.

[0045] The phoneme acquisition unit 16 inputs text and acquires phonemes from the input text. For example, the phoneme acquisition unit 16 performs lexical analysis on the input text, generates a string of phonetic symbols representing speech using character codes, and acquires phonemes from the phonetic symbol string. The phoneme acquisition unit 16 maintains tone information such as words, and while acquiring phonemes from the text, it instructs the pitch generation unit 17 to generate pitch based on tone.

[0046] The pitch generation unit 17 generates pitches corresponding to phonemes. For example, the pitch generation unit 17 stores standard pitches in a storage device, reads them out, and outputs the pitches corresponding to the specified tones.

[0047] The adjustment unit 13 makes the pitch generated by the pitch generation unit 17 consistent with the pitch of the target speech to be converted.

[0048] The generation unit 14 inputs phonemes and linearly transformed pitches into a deep learning model and synthesizes a speech signal that is produced by the transformed target speech specified by the input unit 11.

[0049] [Example]

[0050] Next, an embodiment of the speech conversion device 1 of this embodiment will be described.

[0051] Figure 7 This is an example of a screen 100 showing a web application that converts input speech into multi-person speech. For example, when a user accesses a website providing speech conversion services via a browser on a mobile device or personal computer (PC), the screen 100 is displayed. Figure 7 The image is 100%.

[0052] The screen 100 includes a recording button 110, a text input field 120, target voice labels 130A-130D, a voice conversion button 140, and target voice playback buttons 150A-150D.

[0053] The user presses the record button 110 to input voice through a microphone connected to a portable terminal or PC. The user's voice data is then recorded.

[0054] The user enters text in the text input field 120 that contains the same content as the recorded voice message. For example, if the user recorded "Good morning," the user would enter "Good morning" in the text input field 120. Alternatively, the user can use the voice recognition function of a portable terminal or PC to automatically input text into the text input field 120 that contains the same content as the user's recorded voice message.

[0055] The labels representing the target speech are displayed in the target speech labels 130A to 130D. Figure 7 In the example, labels such as "Voice 1," "Voice 12," "Voice 31," and "Voice 99" are displayed. This indicates that the voice was converted to the voice of person number 1, 12, 31, and 99, respectively. The target voice for conversion can be predetermined or randomly selected. Alternatively, the user can choose the target voice for conversion.

[0056] When the user presses the voice conversion button 140, the voice conversion process begins. Specifically, the recorded voice data, the text entered into the text input field 120, and the voice identifiers shown in the target voice labels 130A-130D are input into the voice conversion device 1. The voice conversion device 1 extracts phonemes and pitch from the voice data, and also extracts phonemes from the text. The voice conversion device 1 can correct the phonemes extracted from the voice data using the phonemes extracted from the text, and can also use the phonemes extracted from the text in subsequent processing. The voice conversion device 1 adjusts the pitch and synthesizes the target voices shown in the target voice labels 130A-130D, and outputs voice data that converts the user's voice into the target voice.

[0057] After the voice conversion process, when the user presses the target voice playback button 150A-150D, the voice data corresponding to the voice playback button 150A-150D is played.

[0058] Next, an example of using the speech conversion device of this embodiment for speech speed conversion will be described. When the speech conversion device 1 is used for speech speed conversion, the input unit 11 receives the playback speed specification and inputs the temporal data including the phonemes and pitches extracted by the extraction unit 12, after compressing or lengthening them in the time direction, to the generation unit 14. For example, when playing at double speed, the vocal range of the phonemes extracted by the extraction unit 12 is compressed, and the adjustment unit 13 adjusts the pitch to the height of the target speech after compressing the pitch in the time direction, and inputs the phonemes and pitches to the generation unit 14. As a result, the input speech is played at double speed with a comfortable sound quality (target speech). Any speech can be selected as the target speech. If a speech close to the source speech is selected as the target speech, the playback speed of the speech can be changed more comfortably. When playing the input speech slowly, the vocal range of the phonemes is lengthened, and the pitch is lengthened in the time direction.

[0059] exist Figure 8 The diagram illustrates an example of connecting the speed conversion device 3 to the speech conversion device 1. The speed conversion device 3 receives speech (or a moving image) as input and changes the playback speed of the input speech to fast-forward or slow down. The pitch of the speech changes when the playback speed is changed, becoming higher or lower.

[0060] When speech with altered playback speed (and changed pitch) is input into speech conversion device 1, the device extracts phonemes and pitch from the speech data, linearly converts the extracted pitch into the height of the target speech, and inputs the phonemes and pitch into a deep learning model to synthesize speech based on the target speech. Thus, speech with altered pitch due to playback speed is played at the same timing as the target speech after the playback speed change. Furthermore, by inputting text data with the same content as the speech input to speech conversion device 1, the reduced recognition rate of fast-forwarded speech can be masked.

[0061] exist Figure 8 In this device, the speech conversion device 1 and the speed conversion device 3 are composed of different devices, but the speech conversion device 1 can also have the function of the speed conversion device 3. In addition, even without the speed conversion device 3, if speech played at double speed or slowly is input into the speech conversion device 1, the speed can be kept at double speed or slow and converted into natural speech at the normal speech level.

[0062] As explained above, the speech conversion apparatus 1 of this embodiment includes: an input unit 11 that inputs a specified target speech for conversion; an extraction unit 12 that analyzes the speech signal of the source speech to extract temporal data including phonemes and pitch; an adjustment unit 13 that adjusts the pitch to match the specified target speech pitch; and a generation unit 14 that generates a speech signal of the specified target speech by synthesizing a deep learning model that can learn from the speech data of multiple people to synthesize the speech of the specified person by sequentially inputting phonemes and pitches. In this embodiment, the source speech is decomposed into phonemes and pitches independent of the speaker, and the target speech is synthesized based on the phonemes and pitches, thereby enabling speech conversion without converting the waveform of the source speech. Thus, by learning a deep learning model for speech synthesis from phonemes and pitches, without using source speech data, anyone's speech can be converted into the target speech.

[0063] Explanation of reference numerals in the attached figures

[0064] 1. Voice conversion device

[0065] 11 Input Section

[0066] 12 Extraction Section

[0067] 13 Adjustment Department

[0068] 14 Production Department

[0069] 15. Study Department

[0070] 16-phoneme acquisition section

[0071] 17. Pitch generation section

[0072] 3. Speed ​​conversion device.

Claims

1. A voice conversion device, characterized in that, This voice conversion device has the following features: The input unit is responsible for inputting the specified target speech for conversion. The extraction unit parses and converts the speech data of the source speech to extract temporal data containing phonemes and pitch; An adjustment unit that makes the pitch level consistent with the specified pitch level of the target speech being converted; as well as The generation unit generates speech data for the specified target speech by inputting the phonemes and pitches sequentially into a deep learning model capable of learning speech data from multiple people to synthesize the speech of a specified person.

2. The speech conversion device according to claim 1, characterized in that, The speech conversion device includes a learning unit that extracts phonemes and pitches from speech data of multiple people that become the target speech for conversion, and learns a deep learning model that can synthesize the individual speech of the multiple people based on the phonemes and pitches.

3. The speech conversion device according to claim 1 or 2, characterized in that, The extraction unit inputs an article containing the same spoken content as the source speech, along with the speech data of the source speech, and parses the article to extract phonemes.

4. The speech conversion device according to claim 1 or 2, characterized in that, The extraction unit parses the text to extract phonemes instead of the speech data of the source speech, reads the pitch corresponding to the phonemes from the storage device and sends it to the adjustment unit.

5. The speech conversion device according to claim 1 or 2, characterized in that, The extraction unit extracts the vocal range of each phoneme and inputs the compressed or elongated vocal range into the generation unit. The adjustment unit compresses or lengthens the pitch in the time direction in accordance with the compression or lengthening of the sound-producing interval.

6. A speech conversion method, characterized in that, The computer input specifies the target speech for conversion. The speech data of the source speech is parsed to extract temporal data containing phonemes and pitch. Make the pitch level match the specified pitch level of the target speech. The speech data that synthesizes the specified target speech is generated by inputting the phonemes and pitches sequentially into a deep learning model that can learn from the speech data of multiple people to synthesize the speech of a specified person.

7. A computer program product comprising a program, characterized in that, This program causes the computer to perform the following processes: The specified processing steps for inputting the target speech; The process involves parsing and converting the source speech data to extract temporal data containing phonemes and pitch. The process of making the pitch level consistent with the specified pitch level of the target speech to be converted; as well as The process of generating speech data that synthesizes the specified target speech by inputting the phonemes and pitches sequentially into a deep learning model capable of learning speech data from multiple people to synthesize the speech of a specified person.

8. A recording medium that records a program, characterized in that, This program causes the computer to perform the following processes: The specified processing steps for inputting the target speech; The process involves parsing and converting the source speech data to extract temporal data containing phonemes and pitch. The process of making the pitch level consistent with the specified pitch level of the target speech to be converted; as well as The process of generating speech data that synthesizes the specified target speech by inputting the phonemes and pitches sequentially into a deep learning model capable of learning speech data from multiple people to synthesize the speech of a specified person.

Images