Speech synthesis device, speech synthesis method, and program

By introducing the structure of analytical section, encoder, pronunciation feature quantity decoder and speech waveform decoder in DNN speech synthesis, the problems of long response time and difficulty in pronunciation feature quantity processing are solved, and more efficient pronunciation synthesis and flexible Rhythm processing.

CN118891672APending Publication Date: 2024-11-01KK TOSHIBA
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202380027614.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2022-03-22
Filing Date
2023-03-20
Publication Date
2024-11-01

AI Technical Summary

Technical Problem

The existing DNN speech synthesis based on the encoder/decoder structure has a long problem in response time, and it is difficult to specifically process the input pronunciation feature quantity before generating the waveform.

Method used

Using a speech synthesis device composed of an analytical unit, a first processing unit and a second processing unit, the analytical unit generates a language feature quantity sequence, and the first processing unit generates an intermediate expression sequence and a rhythm feature quantity through an encoder and a rhythm feature quantity decoder. The second processing unit generates the speech waveform successively through the speech waveform decoder, and processes the rhythmic feature quantity during the generation process.

Benefits of technology

The response time until the waveform is generated is improved, and the pronunciation feature quantity is specifically processed before the waveform is generated, which improves the efficiency and flexibility of speech synthesis.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118891672A_ABST
    Figure CN118891672A_ABST
Patent Text Reader

Abstract

The response time until the waveform is generated can be improved, and the rhythm characteristic quantity based on the whole input can be specifically processed before the waveform is generated. A speech synthesis device according to an embodiment includes an analysis unit, a first processing unit, and a second processing unit. The analysis unit analyzes input text and generates a linguistic feature quantity sequence including one or more vectors representing linguistic feature quantities. The first processing unit is provided with: an encoder that converts the linguistic feature quantity sequence into an intermediate expression sequence by means of a first neural network, the intermediate expression sequence including one or more vectors representing potential variables; and a rhythm characteristic quantity decoder which generates a rhythm characteristic quantity through a second neural network according to the intermediate expression sequence. The second processing unit is provided with a voice waveform decoder that sequentially generates a voice waveform by means of a third neural network on the basis of the intermediate expression sequence and the rhythm feature amount.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] Embodiments of the present invention relate to a speech synthesis device, a speech synthesis method, and a program. Background Art

[0002] In recent years, speech synthesis devices using deep neural networks (DNN) have become known. Among them, many DNN speech synthesis devices based on encoder / decoder structures have been proposed.

[0003] For example, Patent Document 1 proposes a sequence-to-sequence recurrent neural network that takes a sequence of natural language characters as input and outputs a spectrogram of spoken speech. For another example, Non-Patent Document 1 proposes a DNN speech synthesis technology based on an encoder / decoder structure using a self-attention mechanism that takes a natural language phoneme transcription as input and outputs a Mel spectrogram or speech waveform via their respective durations, pitches, and energies.

[0004] Prior art literature

[0005] Patent Literature

[0006] Patent Document 1: Japanese Patent Application No. 2020-515899

[0007] Non-patent literature

[0008] Non-patent literature 1: Yi Ren, Chenxu Hu, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, Tie-Yan Liu, "FastSpeech2: Fast and High-Quality End-to-End Text to Speech," Proc.ICLR, 2021

[0009] Non-patent document 2: A van den Oord et al., “WAVENET: A GENERATIVE MODEL FOR RAWAUDIO”, arxiv preprint, 2016.

[0010] Non-patent literature 3: Z. Wu et al., "A study of speaker adaptation for DNN-based speech synthesis", Proc. Interspeech 2015, 2015.

[0011] Non-patent document 4: Y. Wang et al., “Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesis”, Proceedings of the 35th International Conference on Machine Learning, PMLR 80: 5180-5189, 2018 Summary of the invention

[0012] Problems to be solved by the invention

[0013] An object of the present invention is to provide a speech synthesis device, a speech synthesis method and a program, which can improve the response time until the waveform is generated and can specifically process the prosodic feature quantity based on the entire input before the waveform is generated.

[0014] Means for solving problems

[0015] The speech synthesis device of the embodiment includes a parsing unit, a first processing unit, and a second processing unit. The parsing unit parses the input text to generate a language feature sequence, wherein the language feature sequence includes one or more vectors representing language features. The first processing unit includes: an encoder that converts the language feature sequence into an intermediate expression sequence through a first neural network, wherein the intermediate expression sequence includes one or more vectors representing latent variables; and a prosodic feature decoder that generates a prosodic feature based on the intermediate expression sequence through a second neural network. The second processing unit includes a speech waveform decoder that generates speech waveforms sequentially through a third neural network based on the intermediate expression sequence and the prosodic feature. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 This is a diagram showing an example of the functional configuration of the speech synthesis device according to the first embodiment.

[0017] Figure 2 It is a diagram showing an example of vector expression of context information according to the first embodiment.

[0018] Figure 3 This is a flowchart showing an example of the speech synthesis method according to the first embodiment.

[0019] Figure 4 FIG. 1 is a diagram showing an example of the functional configuration of the prosody feature decoder according to the first embodiment.

[0020] Figure 5This is a flowchart showing an example of a method for generating a prosodic feature value according to the first embodiment.

[0021] Figure 6 It is a diagram showing an example of the functional configuration of the speech synthesis device according to the second embodiment.

[0022] Figure 7 This is a flowchart showing an example of the speech synthesis method according to the second embodiment.

[0023] Figure 8 It is a diagram for explaining a processing example of the processing unit according to the second embodiment.

[0024] Fig. 9 It is a diagram showing an example of the functional configuration of the speech synthesis device according to the third embodiment.

[0025] Fig.10 It is a diagram showing an example of the functional configuration of a continuous speech frame number generating unit according to the third embodiment.

[0026] Fig.11 It is a diagram showing an example of a tone waveform according to the third embodiment.

[0027] Fig.12 This is a flowchart showing an example of a speech synthesis method according to the third embodiment.

[0028] Fig.13 It is a diagram for explaining a processing example of the continuous speech frame number generating unit according to the third embodiment.

[0029] Fig.14 It is a diagram showing an example of the functional configuration of a speech synthesis device according to the fourth embodiment.

[0030] Fig.15 This is a flowchart showing an example of a speech synthesis method according to the fourth embodiment.

[0031] Fig.16 It is a diagram for explaining a processing example of the first processing unit according to the fourth embodiment.

[0032] Fig.17 It is a diagram showing an example of the hardware configuration of the speech synthesis device according to the first to fourth embodiments. DETAILED DESCRIPTION

[0033] In DNN speech synthesis based on the encoder / decoder structure, two neural networks, encoder and decoder, are used. The encoder converts the input sequence into latent variables. Latent variables are values ​​that cannot be directly observed from the outside, and the conversion results of each input, that is, the sequence of intermediate expressions, are used in speech synthesis. The decoder converts the obtained latent variables (i.e., the intermediate expression sequence) into acoustic features and speech waveforms, etc. In the case where the sequence lengths of the intermediate expression sequence and the acoustic features output by the decoder are different, this can be dealt with by using an attention mechanism as in Patent Document 1 or separately calculating the number of frames of the acoustic features corresponding to each intermediate expression as in Non-Patent Document 1.

[0034] However, in the prior art, since a decoder based on the attention mechanism is used, the entire input needs to be processed during synthesis, resulting in a long response time. In addition, as an improvement, it is also possible to consider outputting all the acoustic feature quantities and speech waveforms one by one, but this will result in the problem that the feature quantities related to rhythm (rhythmic feature quantities) such as the duration of the phoneme and the pitch / inflection of the sound cannot be specifically processed before the entire input is processed.

[0035] Hereinafter, embodiments of a speech synthesis device, a speech synthesis method, and a program for solving the above-mentioned problems will be described in detail with reference to the accompanying drawings.

[0036] (First Embodiment)

[0037] First, an example of the functional configuration of the speech synthesis device according to the first embodiment will be described.

[0038] [Example of functional configuration]

[0039] Figure 1 1 is a diagram showing an example of the functional configuration of the speech synthesis device 10 of the first embodiment. In the DNN speech synthesis based on the encoder / decoder structure, the speech synthesis device 10 outputs the intermediate expression sequence and the prosodic feature value in advance, and then outputs the speech waveform successively. As a result, the response time is improved compared with the conventional DNN speech synthesis processing based on the encoder / decoder structure.

[0040] The speech synthesis device 10 according to the first embodiment includes an analyzing unit 1 , a first processing unit 2 , and a second processing unit 3 .

[0041] The analysis unit 1 analyzes the input text and generates a language feature sequence 101. The language feature sequence 101 is information in which the utterance information (language feature) obtained by analyzing the input text is arranged in a time series. As the utterance information (language feature), for example, context information used as a unit for classifying speech, such as phonemes, semiphones, and syllables, is used.

[0042] Figure 2It is a diagram showing an example of vector expression of context information according to the first embodiment. Figure 2 This is an example of vector expression of context information when phonemes are used as speech units, and a sequence of the vector expressions is used as a language feature value sequence 101 .

[0043] Figure 2 The vector expression includes phonemes, phoneme category information, stress type, stress position in the sentence, word ending information and word class information. Phonemes are one-hot vectors that indicate which phoneme the phoneme is. Phoneme category information is flag information indicating the category of the phoneme. The category indicates the classification of the phoneme as voiced / unvoiced and further refined attributes of the phoneme category.

[0044] The stress type is a numerical value indicating the stress type of the phoneme. The stress position in the sentence is a numerical value indicating the stress position in the sentence of the phoneme. The word ending information is a one-hot vector indicating the word ending information of the phoneme. The word class information is a one-hot vector indicating the word class information of the phoneme.

[0045] In addition, as the language feature sequence 101, it is also possible to use Figure 2 For example, the input text may be converted into a symbol sequence such as the Japanese text-to-speech synthesis symbol specified in JEITA standard IT-4006, each symbol may be one-hot vectorized into utterance information, and the sequence in which the one-hot vectors are arranged in time series may be used as the language feature sequence 101.

[0046] return Figure 1 The first processing unit 2 includes an encoder 21 and a prosody feature decoder 22. The encoder 21 converts a language feature sequence 101 into an intermediate expression sequence 102.

[0047] As described above, the intermediate expression sequence 102 is a latent variable in the speech synthesis device 10, and includes information for obtaining the prosodic feature 103 and the speech waveform 104 in the subsequent prosodic feature decoder 22 and the second processing unit 3. Each vector included in the intermediate expression sequence 102 represents an intermediate expression. The sequence length of the intermediate expression sequence 102 is determined by the sequence length of the language feature sequence 101, but it does not need to be consistent with the sequence length of the language feature sequence 101. For example, multiple intermediate expressions may correspond to one language feature.

[0048] The prosody feature decoder 22 generates a prosody feature 103 based on the intermediate expression sequence 102 .

[0049] The prosody feature 103 is a feature related to prosody such as speech rate, pitch and intonation of the sound, and includes the number of continuous speech frames of each vector included in the intermediate expression sequence 102 and the pitch feature in each speech frame. Here, the speech frame refers to the waveform cut-out unit when the sound feature is obtained by analyzing the speech waveform, and the speech waveform 104 is synthesized according to the sound feature generated for each speech frame during synthesis. In the first embodiment, the interval between each speech frame is set to a fixed time length. In addition, the number of continuous speech frames represents the number of speech frames included in the speech interval corresponding to each vector included in the intermediate expression sequence 102. In addition, as the pitch feature, for example, the fundamental frequency and the logarithm of the fundamental frequency can be cited.

[0050] Furthermore, the prosodic feature 103 may include, in addition to the above examples, a gain in each speech frame and a duration of each vector included in the intermediate expression sequence 102 .

[0051] The second processing unit 3 includes a speech waveform decoder 31, which generates a speech waveform 104 based on the intermediate expression sequence 102 and the prosodic feature 103, and outputs the speech waveform 104 one by one. Here, the sequential generation / output process refers to a process of performing waveform generation processing only on each interval obtained by sequentially dividing the intermediate expression sequence 102 into a small number of segments from the beginning, thereby outputting the speech waveform 104 of the interval. For example, the sequential generation / output process is a process of generating / outputting the speech waveform 104 one by one according to a predetermined number of samples (predetermined data length) arbitrarily determined by the user. Through the sequential generation / output process, the calculation process related to waveform generation can be divided into intervals, and the output and reproduction of the speech of each interval can be performed without waiting for the generation process of the speech waveform 104 of the entire input text.

[0052] Specifically, the speech waveform decoder 31 includes a spectral feature quantity generating unit 311 and a waveform generating unit 312 . The spectral feature quantity generating unit 311 generates spectral feature quantities based on the intermediate expression sequence 102 and the prosodic feature quantity 103 .

[0053] The spectral feature refers to the feature of the spectral characteristics of the speech waveform of each speech frame. The acoustic feature required for speech synthesis is composed of the rhythmic feature 103 and the spectral feature. The spectral feature includes information related to the spectral envelope and the non-periodic index, etc. The spectral envelope represents the vocal tract characteristics such as the resonance peak structure of the speech, and the non-periodic index represents the mixing ratio of the noise component excited by the breathing sound and the overtone component excited by the vibration of the vocal cords. For example, as the spectral envelope information, Mel-cepstrum and Mel line spectral pair can be cited. As the non-periodic index, for example, the frequency band non-periodic index can be cited. In addition, the reproducibility of the waveform can also be improved by including the feature related to the phase spectrum in the spectral feature.

[0054] For example, the spectral feature value generating unit 311 generates spectral feature values ​​of speech frames corresponding to a predetermined number of samples in time series based on the intermediate expression sequence 102 and the prosodic feature value 103 .

[0055] The waveform generation unit 312 generates a synthesized waveform (speech waveform 104) by performing speech synthesis processing using the spectral feature quantity. For example, the waveform generation unit 312 generates the speech waveform 104 by using the spectral feature quantity in a time series with a predetermined number of samples each time, thereby successively generating the speech waveform 104. Thus, for example, the speech waveform 104 can be synthesized little by little in a time series with a predetermined number of speech waveform samples determined by a user, and the response time until the generation of the speech waveform 104 can be improved. In addition, the waveform generation unit 312 can also synthesize the speech waveform 104 using the prosody feature quantity 103 as needed.

[0056] [Example of speech synthesis method]

[0057] Figure 3 1 is a flowchart showing an example of a speech synthesis method according to the first embodiment. First, the analysis unit 1 analyzes the input text and outputs a language feature sequence 101, wherein the language feature sequence 101 includes one or more vectors representing language features (step S1). For example, the analysis unit 1 performs morpheme analysis on the input text to obtain language information required for speech synthesis, such as reading information and stress information, and outputs the language feature sequence 101 based on the obtained reading information and language information. For another example, the analysis unit 1 may also generate the language feature sequence 101 for the input text based on the corrected reading / stress information generated separately in advance.

[0058] Next, the first processing unit 2 performs the processing of steps S2 and S3 to output the intermediate expression sequence 102 and the prosodic feature 103. Specifically, first, the encoder 21 converts the language feature sequence 101 into the intermediate expression sequence 102 (step S2). Next, the prosodic feature decoder 22 generates the prosodic feature 103 based on the intermediate expression sequence 102 (step S3).

[0059] Next, the speech waveform decoder 31 of the second processing unit 3 performs the processing of steps S4 to S6. First, the spectral feature quantity generation unit 311 generates the necessary amount of spectral feature quantities based on the intermediate expression sequence 102 and the required prosodic feature quantity 103 such as the number of continuous speech frames of each vector contained in the intermediate expression sequence 102 of the processing object (step S4). Next, the waveform generation unit 312 uses the spectral feature quantities to generate the required amount of speech waveform 104 (step S5). The user and the second processing unit 3 play back and save the speech waveform 104 generated by the processing of step S5 asynchronously, thereby suppressing the delay caused by waveform generation until the start of playback.

[0060] If the synthesis of all speech waveforms 104 is not completed (step S6: No), the process returns to step S4. By repeatedly executing steps S4 and S5, the entire speech waveform 104 can be generated. If the synthesis of all speech waveforms 104 is completed (step S6: Yes), the process ends.

[0061] Next, details of each unit of the speech synthesis device 10 according to the first embodiment will be described.

[0062] [Details of each section]

[0063] exist Figure 1 In the speech synthesis device 10, the encoder 21 converts the language feature sequence 101 into the intermediate expression sequence 102 through the first neural network. As a neural network, by using structures such as a loop structure, a convolution structure, and a self-attention mechanism that can process time series, the intermediate expression sequence 102 can be given previous and next information.

[0064] Figure 4 2 is a diagram showing an example of the functional configuration of the prosody feature decoder 22 according to the first embodiment. The prosody feature decoder 22 according to the first embodiment includes a continuous speech frame number generating unit 221 and a pitch feature generating unit 222 .

[0065] The continuous speech frame number generating unit 221 generates the continuous speech frame number of each vector included in the intermediate expression sequence 102 .

[0066] The pitch feature value generation unit 222 generates a pitch feature value in each speech frame based on the number of continuous speech frames of each vector of the intermediate expression sequence 102. In addition, the prosody feature value decoder 22 may generate a gain in each speech frame, for example.

[0067] The neural network included in the second neural network is used in the processing of the continuous speech frame number generating unit 221 and the pitch feature generating unit 222. As the neural network used in the processing of the pitch feature decoder 222, a structure such as a loop structure, a convolution structure, and a self-attention mechanism that can process a time series is used. In this way, the pitch feature in each speech frame that takes into account the previous and next information can be obtained, and the smoothness of the synthesized speech is increased.

[0068] [Example of a method for generating a prosodic feature value]

[0069] Figure 5 1 is a flowchart showing an example of a method for generating the prosodic feature 103 according to the first embodiment. First, the continuous speech frame number generating unit 221 generates the continuous speech frame number of each vector included in the intermediate expression sequence 102 (step S11). Next, the pitch feature value generating unit 222 generates the pitch feature value in each speech frame (step S12).

[0070] In addition, Figure 1 In the speech synthesis device 10, the spectral feature quantity generating unit 311 of the speech waveform decoder 31 of the second processing unit 3 generates the spectral feature quantity required for the successive generation of the speech waveform 104 based on the intermediate expression sequence 102 and the prosodic feature quantity 103, using the neural network included in the third neural network. As the neural network, a neural network having at least one of a recurrent structure and a convolutional structure is used. Specifically, as a neural network, by using a recurrent structure with a one-way gate (GRU, Gated Recurrent Unit) and a causal convolutional structure, etc., a smooth spectral feature quantity can be generated without processing all speech frames. In addition, a spectral feature quantity reflecting the time series structure can be obtained, and a smooth synthetic speech can be synthesized.

[0071] The waveform generation unit 312 of the second processing unit 3 synthesizes the speech waveform 104 in the amount required for successive generation using signal processing or using a vocoder based on a neural network included in the third neural network. When a neural network is used, for example, a neural vocoder such as WaveNet proposed in Non-Patent Document 2 can be used to generate a waveform.

[0072] As described above, the speech synthesis device 10 of the first embodiment includes an analysis unit 1, a first processing unit 2, and a second processing unit 3. The analysis unit 1 analyzes the input text and generates a language feature sequence 101, which includes one or more vectors representing language features. In the first processing unit 2, the encoder 21 converts the language feature sequence 101 into an intermediate expression sequence 102 through a first neural network, and the intermediate expression sequence 102 includes one or more vectors representing latent variables. In addition, the prosody feature decoder 22 generates a prosody feature 103 based on the intermediate expression sequence 102. In the second processing unit 3, the speech waveform decoder 31 generates a speech waveform 104 based on the intermediate expression sequence 102 and the prosody feature 103.

[0073] Thus, according to the speech synthesis device 10 of the first embodiment, the response time until the waveform is generated can be improved. Specifically, in the speech synthesis device 10 of the first embodiment, the processing is divided into the first processing unit 2 and the second processing unit 3, the first processing unit 2 outputs the intermediate expression sequence 102 and the prosodic feature 103 in advance, and the second processing unit 3 outputs the speech waveform 104 one by one. Thus, the next speech waveform 104 can be output during the playback of a certain speech waveform 104. Therefore, according to the speech synthesis device 10 of the first embodiment, since the response time is the time until the beginning speech waveform 104 is played back, the response time is improved compared with the prior art in which all the acoustic feature quantities and speech waveforms 104 are obtained at one time.

[0074] (Second Embodiment)

[0075] Next, the second embodiment will be described. In the description of the second embodiment, the same description as the first embodiment will be omitted, and the differences from the first embodiment will be described.

[0076] [Example of functional configuration]

[0077] Figure 6 1 is a diagram showing an example of the functional configuration of the speech synthesis device 10-2 according to the second embodiment. In the speech synthesis device 10-2 according to the second embodiment, the first processing unit 2-2 further includes a processing unit 23. Thus, before the second processing unit 3 processes the speech waveform 104, the prosodic feature 103 of the entire input text can be specifically processed.

[0078] Upon receiving the processing instruction for the prosody feature 103, the processing unit 23 reflects the processing instruction on the prosody feature 103. The processing instruction is received by, for example, input from the user.

[0079] The processing instruction is an instruction to change the value of each prosodic feature 103. For example, the processing instruction is an instruction to change the value of the pitch feature in each speech frame of a certain interval. Specifically, the processing instruction is, for example, an instruction to change the pitch of the second frame to the tenth frame to 300 Hz. For another example, the processing instruction is an instruction to change the number of continuous speech frames of each vector included in the intermediate expression sequence 102. For another example, the processing instruction is an instruction to change the number of continuous speech frames of the 17th intermediate expression included in the intermediate expression sequence 102 to 30.

[0080] In addition to the above examples, the processing instruction may be an instruction to project the prosodic feature 103 of the uttered speech of the input text. Specifically, the processing unit 23 uses the uttered speech of the input text prepared in advance. In addition, the processing unit 23 receives an instruction to project the prosodic feature 103 generated from the input text by the analysis unit 1, the encoder 21, and the prosodic feature decoder 22 in a manner consistent with the prosodic feature of the uttered speech. In this case, the desired processing result can be obtained without directly manipulating the value of the prosodic feature 103 generated from the input text.

[0081] The second processing unit 3 receives the prosodic feature 103 generated by the prosodic feature decoder 22 or the prosodic feature 103 processed by the processing unit 23 .

[0082] [Example of speech synthesis method]

[0083] Figure 7 1 is a flowchart showing an example of a speech synthesis method according to the second embodiment. First, the analysis unit 1 analyzes the input text and outputs a language feature sequence 101, which includes one or more vectors representing language features (step S21). Next, the first processing unit 2-2 obtains an intermediate expression sequence 102 and a prosodic feature 103 based on the language feature sequence 101 (step S22).

[0084] Next, the processing unit 23 determines whether to process the prosodic feature 103 (step S23). The determination of whether to process the prosodic feature 103 is made, for example, based on whether there is an instruction to process the prosodic feature 103. For example, the values ​​of the pitch feature and the duration of each phoneme generated based on the prosodic feature 103 are displayed on a display device, and the user edits the values ​​by operating a mouse or the like, thereby giving an instruction to process.

[0085] When the prosodic feature amount 103 is not to be processed (step S23 : No), the process proceeds to step S25 .

[0086] When the prosodic feature 103 is processed (step S23: Yes), the processing unit 23 reflects the processing instruction on the prosodic feature 103 (step S24). When the prosodic feature 103 needs to be regenerated, such as when the number of continuous speech frames of each vector included in the intermediate representation sequence 102 is changed, the prosodic feature decoder 22 regenerates the prosodic feature 103. The processing of the prosodic feature 103 is repeatedly performed as long as the input of the processing instruction is received from the user.

[0087] Next, the second processing unit 3 (speech waveform decoder 31) sequentially outputs the speech waveform 104 (step S25). The details of the process of step S25 are the same as those of the first embodiment, and thus the description thereof is omitted.

[0088] Next, since the waveform generation unit 312 synthesizes the speech waveform 104 again, it is determined whether the prosodic feature 103 is to be reprocessed (step S26). If the prosodic feature 103 is to be reprocessed (step S26: Yes), the process returns to step S24. For example, if the desired speech waveform 104 is not obtained, a processing instruction is further received from the user, and the process returns to step S24.

[0089] When the prosody feature value 103 is not to be reprocessed (step S26 : No), the process ends.

[0090] [Details of processing]

[0091] The details of the processing when the processing is prosody projection are described. When the processing unit 23 receives a projection instruction for the prosody feature 103 of the uttered speech of the input text, the following processing is performed in step S24. First, the processing unit 23 analyzes the uttered speech to obtain the prosody feature 103. By performing phoneme alignment according to the utterance content of the uttered speech and extracting the phoneme boundary, the duration of each phoneme in the prosody feature 103 is obtained. In addition, by extracting the acoustic feature of the uttered speech, the pitch feature in each speech frame is obtained. Next, the processing unit 23 changes the number of continuous speech frames of each vector included in the intermediate expression sequence 102 based on the phoneme duration obtained from the uttered speech. Then, the processing unit 23 changes the pitch feature in each speech frame to be consistent with the pitch feature extracted from the uttered speech. The other feature quantities included in the prosody feature 103 are also changed to be consistent with the feature quantities obtained by analyzing the uttered speech.

[0092] Figure 8 It is a diagram for explaining a processing example of the processing unit 23 according to the second embodiment. Figure 8The example is a processing example in which the processing unit 23 receives a projection instruction of the pitch feature of the uttered speech of the input text. The pitch feature 105 represents the pitch feature generated by the prosody feature decoder 22. The pitch feature 106 represents the pitch feature of the uttered speech of the input text (for example, the uttered speech of the user). The pitch feature 107 represents the pitch feature generated by the processing unit 23. For example, the processing unit 23 generates the pitch feature 107 by performing processing to make the maximum value and minimum value (or mean and variance) of the pitch feature 106 consistent with the maximum value and minimum value (or mean and variance) of the pitch feature 105.

[0093] As described above, in the speech synthesis device 10-2 of the second embodiment, the first processing unit 2-2 outputs the prosodic feature 103, and the processing unit 23 reflects the processing instruction of the user. That is, the prosodic feature 103 for the entire input text is output before the speech waveform 104 is generated, so that the entire input text can be specifically processed before the waveform is generated. In the prior art, when all the acoustic feature quantities and the speech waveform 104 are outputted sequentially as a means of improving the response time, it is difficult to specifically process the prosodic feature 103 of the entire input text.

[0094] In the speech synthesis device 10-2 of the second embodiment, the pitch of the entire input text in speech frame units can be specifically processed before the second processing unit 3 processes the speech waveform 104. Thus, the second processing unit 3 can synthesize the speech waveform 104 reflecting the user's instruction on the specific processing of the prosodic feature 103.

[0095] (Third Embodiment)

[0096] Next, a third embodiment will be described. In the description of the third embodiment, the same description as the first embodiment will be omitted, and the differences from the first embodiment will be described.

[0097] [Example of functional configuration]

[0098] Fig. 9 1 is a diagram showing an example of the functional configuration of the speech synthesis device 10-3 of the third embodiment. In the speech synthesis device 10-3 of the third embodiment, the speech frame is determined based on the pitch. Specifically, the interval of the speech frame is changed to the pitch period. Thus, in the third embodiment, accurate speech analysis based on pitch synchronization analysis can be applied.

[0099] The speech synthesis device 10-3 of the third embodiment includes an analysis unit 1, a first processing unit 2-3, and a second processing unit 3. The first processing unit 2-3 includes an encoder 21 and a prosody feature decoder 22. The prosody feature decoder 22 includes a continuous speech frame number generator 221 and a pitch feature generator 222.

[0100] Fig.10 1 is a diagram showing an example of the functional configuration of the continuous speech frame number generator 221 according to the third embodiment. The continuous speech frame number generator 221 according to the third embodiment includes a rough pitch generator 2211 , a duration generator 2212 , and a calculator 2213 .

[0101] The rough pitch generation unit 2211 generates an average pitch feature of each vector included in the intermediate expression sequence 102. The duration generation unit 2212 generates the duration of each vector included in the intermediate expression sequence 102. The average pitch feature and duration represent the average of the pitch feature in each speech frame included in the speech interval corresponding to each vector and the duration of the speech interval.

[0102] The calculation unit 2213 calculates the pitch waveform number indicating the number of pitch waveforms based on the average pitch feature value and duration of each vector included in the intermediate expression sequence 102 .

[0103] The pitch waveform refers to a waveform cut-out unit of a speech frame in the pitch synchronization analysis method.

[0104] Fig.11 1 is a diagram showing an example of a pitch waveform of the third embodiment. The pitch waveform is obtained as follows. First, the waveform generation unit 312 generates pitch label information 108 representing the center time of each period of the periodic speech waveform 104 based on the pitch feature in each speech frame included in the prosody feature 103.

[0105] Next, the waveform generation unit 312 determines the position of the pitch marker information 108 as the center position and synthesizes the speech waveform 104 based on the pitch period. By performing synthesis with the position of the appropriately assigned pitch marker information 108 as the center moment, appropriate synthesis can be performed that can also cope with local changes in the speech waveform 104, thereby reducing sound quality degradation.

[0106] However, even for intervals of the same duration, the higher the pitch, the more pitch waveforms there are, and the lower the pitch, the fewer pitch waveforms there are. Therefore, the number of speech frames included in each interval may be different. Therefore, the calculation unit 2213 does not directly calculate the number of continuous speech frames (number of pitch waveforms) of each vector included in the intermediate expression sequence 102, but calculates it based on the duration of the vector and the average pitch feature.

[0107] [Example of speech synthesis method]

[0108] Fig.12 1 is a flowchart showing an example of a speech synthesis method according to the third embodiment. First, the analysis unit 1 analyzes the input text and outputs a language feature sequence 101, which includes one or more vectors representing language features (step S31). Next, the encoder 21 converts the language feature sequence 101 into an intermediate expression sequence 102 (step S32).

[0109] Next, the continuous speech frame number generator 221 generates the continuous speech frame number of each vector included in the intermediate expression sequence 102 (step S33). Next, the pitch feature value generator 222 generates the pitch feature value in each speech frame (step S34).

[0110] Next, the second processing unit 3 (speech waveform decoder 31) sequentially outputs the speech waveform 104 based on the intermediate expression sequence 102 and the prosodic feature 103 (step S35).

[0111] [Details of the continuous speech frame number generation process]

[0112] Fig.13 1 is a diagram for explaining a processing example of the continuous speech frame number generating unit 221 of the third embodiment. First, the coarse pitch generating unit 2211 generates an average pitch feature value of each vector included in the intermediate expression sequence 102 (step S41). Next, the duration generating unit 2212 generates the duration of each vector included in the intermediate expression sequence 102 (step S42). In addition, the execution order of steps S41 and S42 may be reversed.

[0113] Next, the calculation unit 2213 calculates the number of pitch waveforms of each vector based on the average pitch feature and duration of each vector included in the intermediate expression sequence 102 (step S43). The number of pitch waveforms obtained in step S43 is output as the number of continuous speech frames.

[0114] [Details of each section]

[0115] The coarse pitch generation unit 2211 and the duration generation unit 2212 use the neural network included in the second neural network to generate the average pitch feature and duration of each vector included in the intermediate expression sequence 102 based on the intermediate expression sequence 102. As the structure of the neural network, for example, a multilayer perceptron, a convolution structure, and a loop structure can be cited. In particular, by using the convolution structure and the loop structure, time series information can be reflected in the average pitch feature and the duration.

[0116] The calculation unit 2213 calculates the number of pitch waveforms of each vector based on the average pitch feature and duration of each vector included in the intermediate expression sequence 102. For example, if the average pitch feature of a vector (intermediate expression) in the intermediate expression sequence 102 is the mean value of the fundamental frequency f (Hz) and the duration is d (seconds), the number of pitch waveforms n of the vector (intermediate expression) is calculated by n=f×d.

[0117] In addition to the intermediate expression sequence 102, the pitch feature generating unit 222 can also calculate the pitch in each speech frame using the average pitch feature of each vector included in the intermediate expression sequence 102. In this way, the difference between the average pitch feature generated by the rough pitch generating unit 2211 and the pitch actually generated becomes smaller, and it can be expected that a synthesized speech (speech waveform 104) close to the duration generated by the duration generating unit 2212 can be obtained.

[0118] As described above, in the speech synthesis device 10-3 of the third embodiment, the processing is divided into the first processing unit 2-3 that generates the prosodic feature 103 and the second processing unit 3 that generates the spectrum feature and the speech waveform 104. In addition, the speech frame is determined based on the pitch. Therefore, according to the speech synthesis device 10-3 of the third embodiment, it is possible to improve the quality of the synthesized speech (speech waveform 104) by using accurate speech analysis based on pitch synchronization analysis.

[0119] (Fourth Embodiment)

[0120] Next, a fourth embodiment will be described. In the description of the fourth embodiment, the same description as the first embodiment will be omitted, and the differences from the first embodiment will be described.

[0121] [Example of functional configuration]

[0122] Fig.14 1 is a diagram showing an example of the functional configuration of a speech synthesis device 10-4 according to the fourth embodiment. The speech synthesis device 10-4 according to the fourth embodiment includes an analysis unit 1, a first processing unit 2-4, a second processing unit 3, a speaker identification information conversion unit 4, and a style identification information conversion unit 5. The first processing unit 2-4 includes an encoder 21, a prosody feature value decoder 22, and an assignment unit 24.

[0123] In the speech synthesis device 10-4 of the fourth embodiment, the speaker identification information conversion unit 4, the style identification information conversion unit 5 and the assignment unit 24 reflect the speaker identification information and the style identification information to the synthesized speech (speech waveform 104). Thus, the speech synthesis device 10-4 of the fourth embodiment can obtain synthesized speech of multiple speakers and multiple styles.

[0124] The speaker identification information identifies the input speaker. For example, the speaker identification information is represented by "the second speaker (the speaker identified by the number)" and "the speaker of the speech (the speaker indicated by the uttered speech)".

[0125] The style identification information identifies the style of speech (e.g., emotion, etc.) For example, the style identification information is represented by "the first style (the style identified by the number)" and "the style of the speech (the style indicated by the uttered speech)".

[0126] The speaker identification information conversion unit 4 converts the speaker identification information into a speaker vector representing the characteristic information of the speaker. The speaker vector is a vector used by the speech synthesis device 10-4 to use the speaker identification information. For example, when the speaker identification information includes the designation of a speaker that can be synthesized by the speech synthesis device 10-4, the speaker vector is a vector of an embedded expression corresponding to the speaker. In addition, when the speaker identification information is a separately prepared uttered speech of a certain speaker, the speaker vector is a vector obtained from the acoustic feature quantity of the uttered speech and a statistical model for identifying the speaker, such as an i-vector, as proposed in non-patent document 3.

[0127] The style determination information conversion unit 5 converts the style determination information that determines the style of the speech into a style vector that represents the feature information of the style. Like the speaker vector, the style vector is a vector used by the speech synthesis device 10-4 to use the style determination information. For example, in the case where the style determination information includes a designation of a style that can be synthesized by the speech synthesis device 10-4, the style vector is a vector of an embedded expression corresponding to the style. In addition, in the case where the style determination information is a voiced speech of a certain style prepared separately, the style vector is a vector obtained by converting the acoustic feature quantity of the voiced speech through a neural network, etc., such as the Global Style Tokens (GST) proposed in non-patent document 4.

[0128] The adding unit 24 adds feature information represented by a speaker vector, a style vector, etc. to the intermediate expression sequence 102 obtained by the encoder 21 .

[0129] [Example of speech synthesis method]

[0130] Fig.15: is a flowchart showing an example of a speech synthesis method according to the fourth embodiment. First, the parsing unit 1 parses the input text and outputs a language feature sequence 101, wherein the language feature sequence 101 includes one or more vectors representing language features (step S51). Next, the speaker determination information conversion unit 4 converts the speaker determination information into a speaker vector by the above method (step S52). Next, the style determination information conversion unit 5 converts the style determination information into a style vector by the above method (step S53). In addition, the execution order of steps S52 and S53 may also be reversed.

[0131] Next, the assigning unit 24 assigns information such as the speaker vector and the style vector to the intermediate expression sequence 102, and the prosodic feature decoder 22 generates the prosodic feature 103 based on the intermediate expression sequence 102 (step S54). Then, the second processing unit 3 (speech waveform decoder 31) sequentially outputs the speech waveform 104 based on the intermediate expression sequence 102 and the prosodic feature 103 (step S55).

[0132] [Details of the processing in the first processing unit]

[0133] Fig.16 This is a diagram for explaining a processing example of the first processing unit 2 - 4 of the fourth embodiment. First, the encoder 21 converts the language feature value sequence 101 into the intermediate expression sequence 102 (step S61 ).

[0134] Next, the adding unit 24 adds information such as the speaker vector and the style vector to the intermediate expression sequence 102 (step S62 ).

[0135] Several methods are conceivable regarding the method of adding information in step S62 . For example, information may be added to the intermediate expression sequence 102 by adding a speaker vector and a style vector to each vector (intermediate expression) included in the intermediate expression sequence 102 .

[0136] For another example, information may be added to the intermediate expression sequence 102 by combining the speaker vector and the style vector with each vector (intermediate expression) included in the intermediate expression sequence 102. Specifically, information may be added to the intermediate expression sequence 102 by combining the components of the m1-dimensional speaker vector and the m2-dimensional style vector with the components of the n-dimensional vector (intermediate expression) to form an n+m1+m2-dimensional vector.

[0137] For another example, the intermediate expression sequence 102 combining the speaker vector and the style vector may be further linearly transformed to convert the intermediate expression sequence 102 combining the speaker vector and the style vector into a more appropriate vector expression.

[0138] Next, the prosody feature decoder 22 generates the prosody feature 103 based on the intermediate expression sequence 102 obtained in step S62 (step S63).

[0139] Since the intermediate expression sequence 102 obtained in step S62 and the prosodic feature 103 generated in step S63 reflect the speaker / style information, the speech waveform 104 obtained by the second processing unit 3 has the characteristics of the speaker and the characteristics of the style.

[0140] Furthermore, when the waveform generation unit 312 of the speech waveform decoder 31 of the second processing unit 3 generates a waveform using the neural network included in the third neural network, the neural network may use the speaker vector and the style vector. In this way, it is expected that the reproducibility of the speaker and style of the synthesized speech (speech waveform 104) can be improved.

[0141] As described above, the speech synthesis device 10 - 4 of the fourth embodiment receives speaker identification information and style identification information and reflects them on the speech waveform 104 , thereby obtaining synthesized speech (speech waveform 104 ) of multiple speakers and multiple styles.

[0142] (Variation Example)

[0143] The parsing unit 1 of the speech synthesis device 10 (10-2, 10-3, 10-4) of the first to fourth embodiments may also divide the input text into a plurality of partial texts, and output a language feature sequence 101 for each partial text. For example, in the case where the input text is composed of a plurality of sentences, it may be divided into partial texts based on the sentences, and a language feature sequence 101 may be obtained for each partial text. In the case where a plurality of language feature sequences 101 are output, the subsequent processing is performed on each language feature sequence 101. For example, each language feature sequence 101 may be processed sequentially in a time series. For another example, a plurality of language feature sequences 101 may also be processed in parallel.

[0144] In addition, the neural networks used in the speech synthesis devices 10 (10-2, 10-3, 10-4) of the first to fourth embodiments are all learned by statistical methods. In this case, by learning several neural networks at the same time, the overall optimal parameters can be obtained.

[0145] For example, in the speech synthesis device 10 of the first embodiment, the neural network used in the first processing unit 2 and the neural network used in the spectral feature value generation unit 311 may be optimized simultaneously. In this way, the speech synthesis device 10 can use the optimal neural network for generating both the prosodic feature value 103 and the spectral feature value.

[0146] Finally, an example of the hardware configuration of the speech synthesis apparatus 10 (10-2, 10-3, 10-4) of the first to fourth embodiments is described. The speech synthesis apparatus 10 (10-2, 10-3, 10-4) of the first to fourth embodiments can be realized by using any computer device as basic hardware, for example.

[0147] [Example of hardware configuration]

[0148] Fig.17 The figure shows an example of the hardware structure of the speech synthesis device 10 (10-2, 10-3, 10-4) of the first to fourth embodiments. The speech synthesis device 10 (10-2, 10-3, 10-4) of the first to fourth embodiments includes a processor 201, a main storage device 202, an auxiliary storage device 203, a display device 204, an input device 205, and a communication device 206. The processor 201, the main storage device 202, the auxiliary storage device 203, the display device 204, the input device 205, and the communication device 206 are connected via a bus 210.

[0149] In addition, the speech synthesis device 10 (10-2, 10-3, 10-4) may not include a part of the above-mentioned structure. For example, when the speech synthesis device 10 (10-2, 10-3, 10-4) can use the input function and display function of the external device, the speech synthesis device 10 (10-2, 10-3, 10-4) may not include the display device 204 and the input device 205.

[0150] The processor 201 executes a program read from the auxiliary storage device 203 to the main storage device 202. The main storage device 202 is a memory such as a ROM and a RAM. The auxiliary storage device 203 is a HDD (Hard Disk Drive) or a memory card.

[0151] The display device 204 is, for example, a liquid crystal display. The input device 205 is an interface for operating the information processing device 100. In addition, the display device 204 and the input device 205 may also be implemented by a touch panel having a display function and an input function. The communication device 206 is an interface for communicating with other devices.

[0152] For example, the program executed by the speech synthesis device 10 (10-2, 10-3, 10-4) is recorded in an installable format or an executable format file on a computer-readable storage medium such as a memory card, hard disk, CD-RW, CD-ROM, CD-R, DVD-RAM and DVD-R, and provided as a computer program product.

[0153] For another example, the program executed by the speech synthesis device 10 (10-2, 10-3, 10-4) may be stored on a computer connected to a network such as the Internet, and provided by being downloaded via the network.

[0154] For another example, the program executed by the speech synthesis device 10 (10-2, 10-3, 10-4) may be provided via a network such as the Internet without downloading. Specifically, the following structure may be adopted: the speech synthesis process is performed by a so-called ASP (Application Service Provider) type service that implements the processing function only through its execution instruction and result acquisition, without transmitting the program from the server computer.

[0155] For another example, the program of the speech synthesis device 10 (10-2, 10-3, 10-4) may be configured to be pre-installed in a ROM or the like and provided.

[0156] The program executed by the speech synthesis device 10 (10-2, 10-3, 10-4) is a module structure including functions that can also be realized by the program in the above-mentioned functional structure. Each of these functions is actual hardware, and the processor 201 reads the program from the storage medium and executes it, so that each of the above-mentioned functional blocks is loaded into the main storage device 202. That is, each of the above-mentioned functional blocks is generated on the main storage device 202.

[0157] Furthermore, part or all of the above-described functions may be realized by hardware such as IC instead of software.

[0158] Furthermore, each function may be implemented using a plurality of processors 201 . In this case, each processor 201 may implement one of the functions or two or more of the functions.

[0159] Some embodiments of the present invention have been described, but these embodiments are presented as examples and are not intended to limit the scope of the invention. These new embodiments can be implemented in various other ways, and various omissions, substitutions, and changes can be made without departing from the scope of the subject matter of the invention. These embodiments and their variations are included in the scope and subject matter of the invention, and are included in the invention described in the claims and the scope of their equivalents.

Claims

1. A speech synthesis device, characterized in that: have: An analyzing unit that analyzes an input text to generate a language feature sequence, wherein the language feature sequence includes one or more vectors representing language features; a first processing unit; and Second processing unit; The first processing unit comprises: an encoder that converts the language feature sequence into an intermediate expression sequence through a first neural network, wherein the intermediate expression sequence includes one or more vectors representing latent variables; and A prosody feature decoder, which generates a prosody feature through a second neural network according to the intermediate expression sequence; The second processing unit includes a speech waveform decoder that sequentially generates speech waveforms through a third neural network based on the intermediate expression sequence and the prosodic feature value.

2. The speech synthesis device according to claim 1, characterized in that: The speech waveform decoder of the second processing unit comprises: a spectral feature value generating unit, which generates spectral feature values ​​of speech frames corresponding to a predetermined number of samples in time series based on the intermediate expression sequence and the prosodic feature value; as well as The waveform generating unit generates the speech waveform by a predetermined number of samples each in a time series based on the spectral feature value, thereby sequentially generating the speech waveform.

3. The speech synthesis device according to claim 2, characterized in that: The spectral feature value generation unit generates the spectral feature value in time series based on the intermediate expression sequence and the prosody feature value by using a neural network having at least one of a recurrent structure and a convolutional structure included in a third neural network.

4. The speech synthesis device according to any one of claims 1 to 3, characterized in that: The prosody feature decoder comprises: a continuous speech frame number generating unit, which generates the continuous speech frame number of each vector included in the intermediate expression sequence; and A pitch feature value generating unit generates a pitch feature value in each speech frame by a neural network included in the second neural network based on the number of continuous speech frames.

5. The speech synthesis device according to claim 4, characterized in that: Speech frames are determined based on pitch, The continuous speech frame number generating unit comprises: a coarse pitch generating unit for generating an average pitch feature of each vector included in the intermediate expression sequence; A duration generating unit, which generates the duration of each vector contained in the intermediate expression sequence; as well as A calculation unit calculates the pitch waveform number based on the average pitch feature and the duration.

6. The speech synthesis device according to any one of claims 1 to 5, characterized in that: The first processing unit further includes a processing unit for processing the prosodic feature value. The second processing unit receives the prosodic feature generated by the prosodic feature decoder or the prosodic feature processed by the processing unit.

7. The speech synthesis device according to claim 6, characterized in that: The processing unit receives a user's instruction to process the prosody feature value, and processes the prosody feature value based on the user's instruction to process the prosody feature value. The processing instruction from the user includes an instruction to change the value of the prosodic feature value or an instruction to project the value onto a prosodic feature value obtained by speech analysis of the uttered speech of the input text.

8. The speech synthesis device according to any one of claims 1 to 7, characterized in that: The speech synthesis device further comprises a speaker identification information conversion unit for converting speaker identification information that identifies a speaker into a speaker vector that represents feature information of the speaker. The first processing unit further includes an assigning unit that assigns feature information of the speaker vector to the intermediate utterance sequence.

9. The speech synthesis device according to any one of claims 1 to 8, characterized in that: The speech synthesis device further includes a style identification information conversion unit for converting style identification information that identifies a style of speech into a style vector that represents feature information of the style. The first processing unit further includes an assigning unit that assigns feature information of the style vector to the intermediate expression sequence.

10. A speech synthesis method, characterized in that: include: A step of analyzing the input text by an analyzing unit to generate a language feature sequence, wherein the language feature sequence includes one or more vectors representing the language feature; The first processing unit converts the language feature sequence into an intermediate expression sequence through a first neural network, wherein the intermediate expression sequence includes one or more vectors representing latent variables; The first processing unit generates a prosody feature value through a second neural network according to the intermediate expression sequence; as well as The second processing unit sequentially generates speech waveforms through a third neural network based on the intermediate expression sequence and the prosodic feature value.

11. A program, characterized in that The program causes the computer to function as an analyzing unit, a first processing unit, and a second processing unit. The analyzing unit analyzes the input text to generate a language feature sequence, wherein the language feature sequence includes one or more vectors representing language features. The first processing unit has the functions of an encoder and a prosody feature decoder. The encoder converts the language feature sequence into an intermediate expression sequence through a first neural network, wherein the intermediate expression sequence includes one or more vectors representing latent variables. The prosody feature decoder generates a prosody feature through a second neural network according to the intermediate expression sequence, The second processing unit has a function of a speech waveform decoder that sequentially generates speech waveforms through a third neural network based on the intermediate expression sequence and the prosodic feature value.