Speech waveform generating system, speech waveform generating method, and speech waveform generating program

Abstract

To provide a model capable of voice-extending a word or phrase with high quality.SOLUTION: A speech waveform generation system includes an embedding layer that sequentially arranges phoneme sequences in a vector space to generate phoneme embeddings, an acoustic model that generates acoustic features from the phoneme embeddings, and a waveform generation model that generates speech waveforms from the acoustic features. The acoustic model includes an encoder to convert the phoneme embeddings into a continuous representation, a variance adapter to predict a duration of each phoneme from the continuous representation, and a decoder to predict an acoustic feature from an output of the variance adapter. The speech waveform generation system further includes a skip connection to add features based on the phoneme embedding to an output of the encoder.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to a voice waveform generation system, a voice waveform generation method, and a voice waveform generation program. [Background technology]

[0002] Recent advances in text-to-speech synthesis technology using neural networks have made it possible to synthesize high-quality speech that sounds close to natural speech.

[0003] Conventional text-to-speech synthesis technologies employ a cascade architecture that uses individual acoustic models. For example, models such as FastSpeech 2 (Non-Patent Document 1) and Conformer-FastSpeech 2 (Non-Patent Documents 2 and 3) achieve high-efficiency and high-quality speech synthesis.

[0004] In recent years, end-to-end models such as JETS (Non-Patent Document 4) and VITS (Non-Patent Document 5) have emerged, which integrate acoustic modeling and vocoding to simplify the synthesis process and improve the naturalness of speech. [Prior art documents] [Non-patent literature]

[0005] [Non-Patent Document 1] Y. Ren, C. Hu, X. Tan, T. Qin, S. Zhao, Z. Zhao, and T.-Y. Liu, "FastSpeech 2: Fast and high-quality end-to-end text to speech," in Proc. ICLR, May 2021. [Non-patent document 2] P. Guo, F. Boyer, X. Chang, T. Hayashi, Y. Higuchi, H. Inaguma, N. Kamo, C. Li, D. Garcia-Romero, J. Shi, J. Shi, S. Watanabe, K. Wei, W. Zhang, and Y. Zhang, "Recent developments on ESPnet toolkit boosted by Conformer," in Proc. ICASSP, June 2021, pp. 5874-5878. [Non-patent document 3] T. Hayashi, R. Yamamoto, T. Yoshimura, P. Wu, J. Shi, T. Saeki, Y. Ju, Y. Yasuda, S. Takamichi, and S. Watanabe, "ESPnet2-TTS: Extending the edge of TTS research," arXiv:2110.07840, 2021. [Non-patent document 4] D. Lim, S. Jung, and E. Kim, "JETS: Jointly training FastSpeech2 and HiFi-GAN for end to end text to speech," in Proc. Interspeech, Sept. 2022, pp. 21-25. [Non-patent document 5] J. Kim, J. Kong, and J. Son, "Conditional variational autoencoder with adversarial learning for end-to-end text-to-speech," in Proc. ICML, July 2021, pp. 5530-5540. [Non-patent document 6] Haruo Yamashita et al., "MS-FC-HiFiGAN: A fast neural waveform generation model using learnable lightweight upsampling," Research Report Spoken Language Processing (SLP) 2023.2 (2023): 1-6. [Non-Patent Document 7] Takuma Okamoto et al., "Neural TTS for Speech Rate Conversion Using Maximum Likelihood Estimation Duration Model," Proceedings of the Acoustical Society of Japan, Spring 2022, pp. 1021-1024, Mar. 2022. Summary of the Invention [Problem to be solved by the invention]

[0006] In human speech, speech rates of certain words or phrases may be adjusted to convey importance or emotion. In applications such as simultaneous interpretation, text-to-speech systems must reflect the original speaker's changes in emphasis and pace to accurately convey the speaker's intent.

[0007] The models described above share the common feature of controlling the duration of phonemes after passing through the encoder. However, because the features after passing through the encoder contain complex contextual information, extending the duration of words or phrases based on the information contained in those features can result in unnatural speech elongation. More specifically, because the Transformer's structure includes skip connections, phoneme-level information is incorporated, but because it is repeated in multiple layers, the phoneme-level information becomes increasingly diluted.

[0008] As described above, known text-to-speech synthesis techniques have had the problem of difficulty in extending the speech of words and phrases with high quality.

[0009] The present invention aims to provide a model that allows high-quality speech stretching of words and phrases. [Means for solving the problem]

[0010] A speech waveform generation system according to one embodiment includes an embedding layer that sequentially arranges a phoneme sequence in a vector space to generate phoneme embeddings, an acoustic model that generates acoustic features from the phoneme embeddings, and a waveform generation model that generates a speech waveform from the acoustic features. The acoustic model includes an encoder that converts the phoneme embeddings into a continuous representation, a variance adapter that predicts the duration of each phoneme from the continuous representation, and a decoder that predicts the acoustic features from the output of the variance adapter. The speech waveform generation system further includes a skip connection for adding features based on the phoneme embeddings at a stage subsequent to the encoder.

[0011] The audio waveform generation system may further include an adder disposed between the encoder and the variance adapter.

[0012] A skip connection may connect the embedded layer to the adder.

[0013] The speech waveform generation system may further include an adder disposed between the acoustic model and the waveform generation model. A skip connection may connect the embedding layer and the adder.

[0014] The skip connection may include a Gaussian upsampling layer that upsamples the phoneme embedding.

[0015] The variance adapter may adjust the predicted duration of each phoneme according to the information for controlling speech stretching.

[0016] The audio waveform generation system may further include means for providing a user interface screen that accepts a specification for generating information for controlling audio stretching.

[0017] The speech waveform generation system may further include a text analysis unit that analyzes input text and outputs a phoneme sequence.

[0018] The skip connection may include a normalization layer.

[0019] Another embodiment of a speech waveform generation method includes generating phoneme embeddings by sequentially arranging a phoneme sequence in a vector space using an embedding layer, generating acoustic features from the phoneme embeddings using an acoustic model, and generating a speech waveform from the acoustic features using a waveform generation model. The generating acoustic features includes converting the phoneme embeddings to a continuous representation using an encoder, adding features based on the phoneme embeddings to the continuous representation, predicting the duration of each phoneme from the continuous representation to which the features based on the phoneme embeddings have been added using a variance adapter, and predicting the acoustic features from the output of the variance adapter using a decoder.

[0020] A speech waveform generation program according to yet another embodiment causes a computer to execute the steps of: generating phoneme embeddings by sequentially arranging a phoneme sequence in a vector space using an embedding layer; generating acoustic features from the phoneme embeddings using an acoustic model; and generating a speech waveform from the acoustic features using a waveform generation model. The step of generating acoustic features includes: converting the phoneme embeddings into a continuous representation using an encoder; adding features based on the phoneme embeddings to the continuous representation; predicting the duration of each phoneme from the continuous representation to which the features based on the phoneme embeddings have been added using a variance adapter; and predicting the acoustic features from the output of the variance adapter using a decoder. [Effects of the Invention]

[0021] The present invention provides a model that allows for high quality speech stretching of words and phrases. [Brief explanation of the drawings]

[0022] [Figure 1] 1 shows an example of the configuration of a model according to this embodiment. [Figure 2]FIG. 10 is a diagram showing an example of the results of subjective evaluation under the condition that audio stretching is not performed. [Figure 3] FIG. 10 is a diagram showing an example of the results of subjective evaluation under partial enlargement conditions. [Figure 4] FIG. 1 is a schematic diagram showing an example of implementation of a speech waveform generation system including a PESC model according to an embodiment of the present invention. [Figure 5] FIG. 2 is a schematic diagram showing an example of a user interface screen of the speech waveform generation system including the PESC model according to the present embodiment. [Figure 6] 1 is a schematic diagram showing an example of a hardware configuration for realizing a voice waveform generation system according to an embodiment of the present invention; [Figure 7] 10 is a flowchart showing an example of processing in the voice waveform generation system according to the present embodiment. [Figure 8] FIG. 10 is a schematic diagram showing a modified example in which Gaussian duration modeling is added to the speech waveform generation system according to the present embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0023] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The present invention will be described in detail with reference to the accompanying drawings, in which the same or corresponding parts are designated by the same reference numerals and will not be described repeatedly.

[0024] [A. Model example] Fig. 1 shows an example of the configuration of a model according to this embodiment. Several models, including a baseline model, will be described with reference to Fig. 1.

[0025] Each model shown in Fig. 1 is an end-to-end model for a text-to-speech synthesis (TTS) task that generates (or predicts) a speech waveform from input text 100. More specifically, each model shown in Fig. 1 includes a text analyzer 6, an embedding layer 8, an acoustic model, and a vocoder (neural waveform generation model).

[0026] The text analysis unit 6 analyzes input text 100 and outputs a phoneme sequence 102. The phoneme sequence 102 may include accent information.

[0027] The embedding layer 8 sequentially places the phoneme sequence into a vector space to generate a phoneme embedding.

[0028] The acoustic model generates (or predicts) acoustic features from the phoneme embeddings output by the embedding layer 8.

[0029] The vocoder generates (or predicts) a speech waveform 110 from acoustic features output by the acoustic model. Each model shown in FIG. 1 includes MS-FC-HiFi-GAN (Non-Patent Document 6) as an example of the vocoder 60.

[0030] Figure 1(a) shows an example of the configuration of the baseline JETS model 1. JETS model 1 is a combination of JETS and MS-FC-HiFi-GAN, both of which are disclosed in Non-Patent Document 4. JETS model 1 includes a text analysis unit 6, an embedding layer 8, an acoustic model 10, and a vocoder 60.

[0031] The acoustic model 10 includes an encoder 12 , a variance adapter 14 , and a decoder 16 .

[0032] The encoder 12 converts the input phoneme embedding (discrete representation) into a continuous representation. The encoder 12 may be Transformer-based.

[0033] The variance adapter 14 predicts the duration of each phoneme from the continuous representation. The variance adapter 14 may also predict the energy and pitch from the continuous representation. The predicted results are added to the output of the decoder 16.

[0034] The decoder 16 predicts acoustic features from the output of the variance adapter 14. The decoder 16 may be Transformer-based, similar to the encoder 12.

[0035] The encoder 12 and decoder 16 of the JETS model 1 each have four layers.

[0036] Note that an alignment training framework may be employed during training of each model shown in FIG. 1 (indicated by dashed lines in the figure). By employing the alignment training framework, intermediate mel spectrograms and external aligners are not required during inference. More specifically, a monotonic alignment search (MAS) 70 is employed. The monotonic alignment search 70 optimizes a variance adapter using the output of the encoder 52 and the target mel spectrogram 120. In addition to the variance adapter, the variance adapter and the vocoder 60 are also optimized. Through training, an alignment between the target mel spectrogram 120 and the phoneme embedding (features calculated from the input text 100) is gradually obtained.

[0037] The vocoder 60 may be trained by adversarial training 72, such as using a HiFi-GAN discriminator.

[0038] Figure 1(b) shows a configuration example of an encoderless model 2 as a first comparative example. Compared to acoustic model 10 of JETS model 1 shown in Figure 1(a), acoustic model 20 of encoderless model 2 omits the encoder and has a multi-layered decoder. More specifically, acoustic model 20 includes a variance adapter 24 and a decoder 26. Decoder 26 has eight layers, which corresponds to the total number of layers of encoder 12 and decoder 16 included in acoustic model 10 of JETS model 1.

[0039] If the encoder of the acoustic model 10 improves the accuracy of phoneme alignment (i.e., has a negative effect when stretching words or phrases), then the encoderless model 2, which does not have an encoder, may improve the performance of stretching words or phrases.

[0040] 1(c) shows a configuration example of the embedded MAS model 3 as a second comparative example. The acoustic model 30 of the embedded MAS model 3 includes an encoder 32, a variance adapter 34, and a decoder 36. The network structure of the embedded MAS model 3 during inference is the same as that of the JETS model 1.

[0041] However, during training, the output of the embedding layer 8, which precedes the encoder 32, is used instead of the output of the encoder 32. That is, the monotonic alignment search 70 optimizes the variance adapter 34 using the output of the embedding layer 8 and the target mel spectrogram 120. By adopting this approach, it is expected that more accurate word-level segmentation can be learned. Learning more accurate word-level segmentation is expected to directly improve the mechanism for speech-lengthening words and phrases.

[0042] 1(d) and 1(e) show configuration examples of PESC (Phone Embedding Skip Connection) models 4 and 5 according to this embodiment. Each of the PESC models 4 and 5 has a skip connection.

[0043] In this embodiment, the PESC is a skip connection for adding features based on phoneme embedding to a stage subsequent to the encoder. The PESC is a detour path that connects the phoneme embedding (or features generated based on phoneme embedding), which is phoneme-level information generated by the embedding layer 8, to the stage subsequent to the encoder.

[0044] The PESC model 4 shown in Figure 1(d) includes an acoustic model 40. The acoustic model 40 includes an encoder 42, a variance adapter 44, and a decoder 46. An adder 48 is disposed between the encoder 42 and the variance adapter 44. A skip connection 80 is connected to the adder 48 for adding the phoneme embedding generated by the embedding layer 8. The skip connection 80 connects the embedding layer 8 and the adder 48.

[0045] By using the skip connection 80 to add the phoneme embedding obtained in the previous stage of processing to the previous stage of the variance adapter 44, the phoneme embedding can be directly influenced on the generation of acoustic features. This is expected to improve the control performance of speech nuances, etc.

[0046] The PESC model 4 shown in Figure 1(e) includes an acoustic model 50. The acoustic model 50 includes an encoder 52, a variance adapter 54, and a decoder 56. An adder 58 is disposed between the acoustic model 50 and a vocoder 60. A skip connection 82 is connected to the adder 58 for adding the phoneme embedding generated by the embedding layer 8. The skip connection 82 connects the embedding layer 8 and the adder 58.

[0047] The skip connections 82 are joined by a Gaussian upsampling layer 84, which upsamples the phoneme embeddings to match the output dimensions of the decoder 56.

[0048] Using skip connections 82, phoneme embeddings obtained in the previous stage of processing are added to the acoustic features generated by the acoustic model 50. By injecting detailed phoneme feature information into the synthetic speech, it is expected that the clarity and naturalness of the output synthetic speech will be improved.

[0049] [B. Performance Evaluation] Next, an example of performance evaluation of each model shown in FIG. 1 will be described.

[0050] (1) Dataset The datasets used were American English and Japanese speech data (male and female) from the HiFi-CAPTAIN corpus, published by the National Institute of Information and Communications Technology (NICT). For English, we used parallel data of 12,988 sentences (approximately 14 hours) recorded in a conversational style, and for Japanese, we used parallel data of 18,855 sentences (approximately 22 hours) recorded in a conversational style.

[0051] A model for females was trained using 18,655 parallel utterances and 201 non-parallel utterances, and a model for males was trained using 18,655 parallel utterances and 203 non-parallel utterances.

[0052] The target mel-spectrogram 120 (see Figure 1) input during training was an 80-dimensional feature with the band limited to 7600 Hz. The STFT length and shift length were 1024 samples and 256 samples, respectively.

[0053] (2) Model setting All models shown in Figure 1 were trained and implemented by modifying the JETS-based E2E TTS model implemented in ESPnet2-TTS. The Harvest algorithm was introduced for fundamental frequency analysis. For the Japanese TTS model, a G2P function based on pyopenjtalk enhanced with prosodic symbols was used. Various acoustic models were employed under cascade fine-tuning and end-to-end training based on established methodologies. More specifically, we evaluated models obtained by cascade fine-tuning Conformer-FastSpeech 2 (hereinafter abbreviated as "CFS2") and ConvNeXt (hereinafter abbreviated as "CN"), and models obtained by end-to-end training of Transformer (FastSpeech 2) (hereinafter abbreviated as "TF") and CN.

[0054] We evaluated four types of acoustic models: CFS2 obtained by cascade fine tuning (Cascade:CFS2), CN2 obtained by cascade fine tuning (Cascade:CN), CFS2 obtained by end-to-end training (End-to-End:TF), and CN obtained by end-to-end training (End-to-End:CN). These models were used to evaluate the accuracy improvement of speech stretching and speech synthesis using different approaches.

[0055] (3) Preliminary evaluation We evaluated the accuracy of phoneme alignment for each model shown in Figure 1. More specifically, we trained each model using a MAS-based phoneme alignment model using pairs of phoneme embeddings and mel-spectrograms. Optimization was performed based on forward-sum loss and bin loss. After training each model shown in Figure 1, we evaluated the mean squared error (MSE) between the duration predicted by each model and the duration of the speech waveform synthesized by each model using the MAS-based phoneme alignment model. The evaluation results are shown in Table 1.

[0056] [Table 1]

[0057] Table 1 shows that (e) PESC Model 4 and (f) PESC Model 5 both maintain accurate phoneme alignment. On the other hand, the baseline JETS Model 1 suggests that there is a significant alignment error between the duration predicted from the hidden features and the duration of the phonemes. In other words, the PESC model according to this embodiment is expected to provide a significant improvement in speech lengthening.

[0058] (4) Evaluation criteria For objective quantitative evaluation, we used the character error rate (CER) calculated using automatic speech recognition (ASR). For English, we used ESPnet2-TTS trained on LibriSpeech as the ASR, and for Japanese, we used a Transformer-based model trained on the CSJ corpus as the ASR. For English text-to-speech synthesis, we also evaluated UTMOS.

[0059] For Japanese text-to-speech synthesis, we conducted a Mean Opinion Score (MOS) test, an example of subjective evaluation. Twenty native Japanese speakers rated a total of 90 samples on a 5-point scale.

[0060] (5) Evaluation of audio stretching The evaluation focused on phrases from the test set of the HiFi-CAPTAIN corpus, where deliberately stressed and slowed speech segments were identified. For the evaluation, each phrase, consisting of approximately five words, had to be selected to represent stressed and prolonged points in speech, allowing for a focused evaluation of how speech stretching affects linguistically important sections. A pre-prepared test dataset was used for each model's inference, where the start and end positions of slowed speech segments were identified.

[0061] In the evaluation, the speech was selectively stretched by a factor of 2 for specific frames. The Gaussian upsampling layer 84 uses the modified duration data to precisely lengthen the speech output for specific phrases, ensuring that only the identified phrases are slowed and intelligible within the utterance, thereby achieving the desired speech stretching.

[0062] To analyze the effectiveness of speech stretching, we compared two experimental conditions: partial stretching and global stretching. Partial stretching stretches only selected frames. Global stretching stretches the entire speech. We evaluated the effect of these speech stretching techniques on the speech waveform.

[0063] (6) Evaluation results We transformed a total of four types of cascade and end-to-end models into the network structures (a) to (e) shown in Figure 1 to obtain 20 models, and evaluated three types of models: no voice duration expansion, partial duration expansion, and whole duration expansion.

[0064] Table 2 shows the evaluation results of the character error rate for speech recognition on English test data, and Table 3 shows the evaluation results of the character error rate for speech recognition on Japanese test data. Note that the smaller the CER value, the more accurate the system is evaluated. The highest-rated results in each category are shown in bold.

[0065] [Table 2]

[0066] [Table 3]

[0067] Tables 2 and 3 show that the models corresponding to (c) to (e) in Figure 1 show superior results compared to the baseline. Comparing Tables 2 and 3 also shows that the improvement is more pronounced for Japanese than for English. It also shows that the difference in performance becomes more pronounced the more speech is stretched.

[0068] Next, the evaluation results of UTMOS, an index of speech clarity and naturalness, are shown in Table 4. Note that a higher UTMOS value indicates a higher evaluation. The highest evaluation results in each category are shown in bold.

[0069] [Table 4]

[0070] Table 4 shows that the models corresponding to (d) and (e) in Figure 1 (PESC models 4 and 5) are superior to other models in both partial and full stretching.

[0071] Next, the results of the subjective evaluation are shown in Figures 2 and 3. Figure 2 is a diagram showing an example of the results of the subjective evaluation under the condition of no audio stretching. In Figure 2, "Original" means the audio data contained in the test data as is. Figure 3 is a diagram showing an example of the results of the subjective evaluation under the condition of partial stretching.

[0072] Referring to FIG. 2, even when no voice stretching is performed, the PESC model 4 according to this embodiment is evaluated to improve the clarity and naturalness of the output synthesized voice.

[0073] Referring to FIG. 3, in the case of partial stretching, the PESC model 4 according to this embodiment is evaluated as having significantly improved clarity and naturalness of the output synthetic speech compared to the baseline.

[0074] As is clear from the above evaluation results, the PESC model according to this embodiment is capable of extending the speech of words and phrases with higher quality.

[0075] [C. Implementation Example] Next, an example of implementation of a speech waveform generation system including a PESC model according to this embodiment will be described.

[0076] FIG. 4 is a schematic diagram showing an example of implementation of a speech waveform generation system including a PESC model according to this embodiment.

[0077] FIG. 4(a) shows an example implementation of a speech waveform generation system including the PESC model 4 shown in FIG. 1(d). As shown in FIG. 4(a), a variance adapter 44 adjusts the predicted duration of each phoneme according to information for controlling speech stretching. As an example of the information for controlling speech stretching, a duration ratio 150 may be provided to the variance adapter 44 for each word or phrase. The ratio 150 may be a sequence of numbers associated with the sequence of words or phrases. The variance adapter 44 predicts the duration of each phoneme. Speech stretching is achieved by multiplying the duration of each phoneme predicted by the variance adapter 44 by the ratio 150.

[0078] FIG. 4(b) shows an example implementation of a speech waveform generation system including the PESC model 5 shown in FIG. 1(e). As shown in FIG. 4(b), information for controlling speech stretching, such as a duration ratio 150 for each word or phrase, is provided to the variance adapter 44 and the Gaussian upsampling layer 84. The decoder 56 outputs acoustic features indicating the stretched words or phrases according to the ratio 150, which is an example of information for controlling speech stretching. The Gaussian upsampling layer 84 outputs stretched phoneme embeddings according to the ratio 150, so as to correspond to the acoustic features output by the decoder 56.

[0079] It should be noted that information required to extend the audio of a particular word or phrase may be given in any format other than the ratio 150.

[0080] FIG. 5 is a schematic diagram showing an example of a user interface screen of a speech waveform generation system including a PESC model according to the present embodiment.

[0081] Referring to Fig. 5, a text area 350 is displayed on the display of the information processing device 300. The text to be input to the model is displayed in the text area 350. The user selects a portion of the text that the user wishes to intentionally emphasize and speak slowly (emphasized portion 360). The information processing device 300 calculates the ratio 150 shown in Fig. 4 corresponding to the emphasized portion 360. Then, when a speech icon 370 is selected, a speech waveform is generated and speech is output.

[0082] As shown in FIG. 5, the information processing device 300 may provide a user interface screen that accepts input text and a designation for generating information for controlling audio stretching.

[0083] The highlighted portion 360 may be designated by a method such as dragging, or by other methods. Audio stretching attribute information may be added to the input text.

[0084] [D. Hardware configuration example] Next, an example of a hardware configuration for realizing the voice waveform generation system according to the present embodiment will be described. The voice waveform generation system according to the present embodiment may be realized using the same computing resource or different computing resources. The computing resource is provided, for example, using a general-purpose computer.

[0085] Fig. 6 is a schematic diagram showing an example of a hardware configuration for realizing a voice waveform generation system according to the present embodiment. Fig. 6 shows an example of the hardware configuration of a portable information processing device such as a smartphone.

[0086] 6, the information processing device 300 includes, as main hardware components, a CPU (central processing unit) 302, a main memory 304, an input device 306, a communication unit 308, a storage 310, an input interface 322, an output interface 324, and a display 326. These components are connected to each other via an internal bus 330.

[0087] The CPU 302 is a processor that executes processes necessary to implement the system. Multiple CPUs 302 may be provided, and the CPUs 302 may have multiple cores. A graphics processing unit (GPU) may be used instead of or in addition to the CPU 302.

[0088] The main memory 304 is a storage area that temporarily stores (or caches) program code, work data, etc. when the CPU 302 executes processing, and is composed of volatile memory such as DRAM (dynamic random access memory) or SRAM (static random access memory).

[0089] The input device 306 is a device that accepts instructions and operations from the user, and is configured by, for example, a keyboard, a mouse, a touch panel, a pen, and the like.

[0090] The communication unit 308 exchanges data with any information processing device on the Internet or an intranet, etc. The communication unit 308 can employ any communication method, such as Ethernet (registered trademark), wireless LAN (local area network), or Bluetooth (registered trademark).

[0091] The input interface 322 receives, for example, an audio signal from a microphone 332 .

[0092] The output interface 324 outputs an audio signal to, for example, a speaker 334 .

[0093] The display 326 may be a liquid crystal display, an organic EL display, or the like.

[0094] The storage 310 stores programs and data necessary to implement the system. The storage 310 is configured with a non-volatile storage device such as a hard disk or a solid state drive (SSD).

[0095] The storage 310 includes, for example, a user interface program 312 and an audio waveform generation program 314 in addition to an operating system (OS) (not shown).

[0096] The user interface program 312 provides a user interface screen for inputting text and outputting synthesized speech, as shown in FIG.

[0097] The speech waveform generation program 314 realizes the speech waveform generation system shown in Fig. 4. The speech waveform generation program 314 includes a trained model of the speech waveform generation system shown in Fig. 4. The trained model included in the speech waveform generation program 314 may include a set of optimized model parameters.

[0098] The storage 310 may include a training program 316. The training program 316 may include instructions for training the audio waveform generation system, such as using a monotonic alignment search as shown in FIG.

[0099] Some of the libraries and functional modules required for a processor (such as CPU 302) to execute a program may be replaced with libraries or functional modules provided as standard by the OS. In this case, the program itself does not include all of the program modules required to realize the corresponding functions, but the program can achieve the desired processing by being installed in the OS execution environment. Furthermore, general-purpose libraries or functional modules licensed for use under a specific license may be used. Even a program that does not include some of these libraries or functional modules is within the technical scope of the present invention.

[0100] Furthermore, programs and data stored on a non-transitory recording medium (for example, an optical disk) may be installed in storage 310. These programs may not only be stored on any of the above-mentioned recording media and distributed, but may also be distributed by downloading them from a server or the like via the Internet or an intranet.

[0101] FIG. 6 shows an example of a configuration using a single computer, but the present invention is not limited to this. Multiple computers connected via a computer network may work together explicitly or implicitly to execute the processing required to realize the system.

[0102] All or part of the functions realized by a processor (such as the CPU 302) executing a program may be realized using a hard-wired circuit such as an integrated circuit, for example, an application specific integrated circuit (ASIC) or a field-programmable gate array (FPGA).

[0103] Those skilled in the art will be able to implement the information processing device 300 according to this embodiment by appropriately using technology appropriate for the era in which the present invention is implemented.

[0104] [E. Processing Procedure] Next, an example of a processing procedure of the system according to the present embodiment will be described.

[0105] Fig. 7 is a flowchart showing an example of processing in the audio waveform generation system according to the present embodiment. Each step shown in Fig. 7 may be realized by CPU 302 of information processing device 300 executing audio waveform generation program 314. The example of processing shown in Fig. 7 is based on the audio waveform generation system shown in Fig. 4(a).

[0106] 7, the information processing device 300 acquires the text 100 and the duration ratio 150 (step S100), analyzes the acquired text 100, and generates the phoneme sequence 102 (step S102). The information processing device 300 sequentially arranges the phoneme sequence 102 in a vector space to generate phoneme embeddings (step S104).

[0107] The information processing device 300 converts the phoneme embedding, which is a discrete representation, into a continuous representation (step S106). The information processing device 300 adds the phoneme embedding to the continuous representation (step S108). The information processing device 300 predicts the duration of each phoneme from the continuous representation to which the phoneme embedding has been added (step S110). The information processing device 300 determines the duration of each phoneme by multiplying the predicted duration of each phoneme by a ratio of 150 (step S112).

[0108] The information processing device 300 generates acoustic features based on the determined duration of each phoneme (step S114). The information processing device 300 generates a speech waveform 110 from the acoustic features (step S116). Then, the processes from step S100 onwards are repeated.

[0109] When based on the speech waveform generation system shown in FIG. 4(b), instead of step S108, a process of upsampling the phoneme embedding and a process of adding the upsampling result to the acoustic feature are adopted.

[0110] [F. Gaussian Duration Modeling] To further improve the quality of synthesized speech under extended duration conditions, Gaussian duration modeling (GMD) can also be applied (see "Neural TTS for speech rate conversion using maximum likelihood estimation type duration model" (Non-Patent Document 7) and Patent Publication No. 2022-105505 (Patent Document 1) in addition to the following explanation).

[0111] 8 is a schematic diagram showing a modified example of the speech waveform generation system according to the present embodiment, in which Gaussian duration modeling is added. Referring to FIG. 8, the trained neural TTS model includes an encoder 62, a variance adapter 64, a decoder 66, and a vocoder 60.

[0112] In the Hidden Markov Model TTS, Gaussian duration modeling (GDM) has been introduced to predict the mean and variance of duration. k is the predicted mean ξ k and variance σ2 k and the target length T, and is given by the maximum likelihood estimation shown in the following equation (1).

[0113]

number

[0114] where variance σ 2 k represents the elasticity of each phoneme. Therefore, GDM is expected to be more robust than simple uniform stretching under fast and slow speaking rate conditions. In conventional neural TTS models, the Gaussian maximum likelihood duration predictor predicts the forced aligned phoneme duration d k In the Gaussian maximum likelihood estimation problem, the negative log-likelihood shown in the following equation (2) is minimized as the loss function.

[0115]

number

[0116] However, a model with this loss function cannot be trained well. This is because the prediction accuracy of the neural TTS acoustic model encoder is high and most of the prediction variance values ​​are close to zero. To solve this problem, we introduce two modifications for stable training. First, when calculating the loss function, d k Gaussian white noise n~N(0,1) is added to the variance. Furthermore, to prevent only the resting phonemes from expanding or contracting due to larger variance values, we introduce a variance limit based on the sigmoid function shown in equation (3) below instead of simple truncation.

[0117]

number

[0118] where hv is the output of the duration variance predictor 90 (see Figure 8 for a conceptual diagram of the configuration and processing flow). With these modifications, the predicted variance value is constrained to the range of 1 to 10 without discontinuous truncation. The duration loss of the proposed method is then defined as follows:

[0119]

number

[0120] When using forcedly aligned phoneme durations, the Gaussian maximum likelihood duration predictor can be trained successfully, but when using monotonic alignment search, it cannot be trained well. This is because phoneme alignment is gradually obtained during the training process, making it difficult to predict variances in the early stages of training. To introduce GDM into a neural TTS model using monotonic alignment search, after training the neural TTS model, only the duration variance predictor 90 is additionally trained using the loss function in Equation (4). In training the duration variance predictor 90, the gradient of the phoneme durations predicted by the pre-trained neural TTS acoustic model is stopped, and only the weights of the duration variance predictor are trained. The additional duration variance predictor enables the application of GDM to a neural TTS model using MAS, achieving duration control that takes into account the elasticity of each phoneme.

[0121] [G. Variation] In the above model, a configuration in which the phoneme embedding is directly added to the encoder output or decoder output using skip connections has been exemplified, but weighting or normalization layers (LN: Layer Normalization) may also be added. That is, weighted phoneme embedding may be added to the encoder output or decoder output. Alternatively, the phoneme embedding may be input to a normalization layer, and the result obtained may be added to the encoder output or decoder output. In this way, the skip connection may include a weighting layer or a normalization layer.

[0122] In the above description, a text-to-speech (TTS) system and process for generating a speech waveform from input text have been exemplified, but the text may also be generated by speech recognition. It may also be combined with neural translation. For example, the speech waveform generation system according to this embodiment can also be used in a simultaneous translation system, a picture book reading system, and the like.

[0123] [H.Advantages] PESC according to this embodiment can be easily introduced into, for example, an already implemented text-to-speech synthesis system. Introducing PESC does not require changing the amount of calculation or parameters in the already implemented text-to-speech synthesis system. Therefore, high-quality speech stretching can be easily achieved.

[0124] The embodiments disclosed herein should be considered to be illustrative in all respects and not restrictive. The scope of the present invention is defined by the claims, not by the description of the above embodiments, and is intended to include all modifications within the meaning and scope of the claims. [Explanation of symbols]

[0125] 1 JETS model, 2 Encoderless model, 3 Embedding MAS model, 4, 5 PESC model, 6 Text analysis unit, 8 Embedding layer, 10, 20, 30, 40, 50 Acoustic model, 12, 32, 42, 52, 62 Encoder, 14, 24, 34, 44, 54 Variance adapter, 16, 26, 36, 46, 56, 66 Decoder, 48, 58 Adder, 60 Vocoder, 70 Monotonic alignment search, 80, 82 Skip connection, 84 Gaussian upsampling layer, 90 Duration variance predictor, 100 Text, 102 Phoneme sequence, 110 Speech waveform, 120 Target mel spectrogram, 150 Ratio, 300 Information processing unit, 302 CPU, 304 Main memory, 306 Input device, 308 Communication unit, 310 Storage, 312 user interface program, 314 speech waveform generation program, 316 learning program, 322 input interface, 324 output interface, 326 display, 330 internal bus, 332 microphone, 334 speaker, 350 text area, 360 highlight, 370 speech icon.

Claims

1. an embedding layer that sequentially places the phoneme sequence into a vector space to generate phoneme embeddings; an acoustic model that generates acoustic features from the phoneme embedding; a waveform generation model that generates a speech waveform from the acoustic feature; The acoustic model includes an encoder that converts the phoneme embeddings into a continuous representation, a variance adapter that predicts the duration of each phoneme from the continuous representation, and a decoder that predicts the acoustic features from an output of the variance adapter. The speech waveform generation system further comprises a skip connection for adding the feature based on the phoneme embedding to a stage subsequent to the encoder.

2. The speech waveform generation system according to claim 1 , wherein the variance adaptor adjusts the predicted duration of each phoneme according to information for controlling speech stretching.

3. The speech waveform generation system according to claim 1 , further comprising a text analysis unit that analyzes input text and outputs the phoneme sequence.

4. The speech waveform generation system according to claim 1 or 2, wherein the skip connections include a normalization layer.

5. generating phoneme embeddings by sequentially placing the phoneme sequence in a vector space using an embedding layer; generating acoustic features from the phoneme embeddings using an acoustic model; generating a speech waveform from the acoustic features using a waveform generation model; The step of generating acoustic features includes: converting the phoneme embeddings into a continuous representation using an encoder; adding the phoneme-embedding-based features to the continuous representation; predicting the duration of each phoneme from the continuous representation to which the phoneme-embedding-based features have been added using a variance adapter; and predicting the acoustic features from the output of the variance adapter using a decoder.

6. A voice waveform generation program, comprising: generating phoneme embeddings by sequentially placing the phoneme sequence in a vector space using an embedding layer; generating acoustic features from the phoneme embeddings using an acoustic model; generating a speech waveform from the acoustic features using a waveform generation model; The step of generating acoustic features includes: converting the phoneme embeddings into a continuous representation using an encoder; adding the phoneme-embedding-based features to the continuous representation; predicting the duration of each phoneme from the continuous representation to which the phoneme-embedding-based features have been added using a variance adapter; and predicting the acoustic feature from the output of the variance adapter using a decoder.

Images