Speech waveform generating system, speech waveform generating method, and speech waveform generating program

Abstract

To provide a new method for achieving singing synthesis using only text speech synthesis corpuses.SOLUTION: The speech waveform generation system includes a text analysis unit that analyzes a text and outputs a phoneme sequence, an acoustic model that estimates an acoustic feature from the phoneme sequence, and a neural vocoder that generates a speech waveform from the acoustic feature. The variance adapter includes a phoneme duration adjustment unit, an energy estimation unit, a fundamental frequency estimation unit, and a phoneme duration estimation unit. In the singing voice synthesis, lyrics included in an input musical score are input to a text analysis unit, a phoneme duration of each phoneme is determined based on note information included in the input musical score, a frequency obtained by subtracting a predetermined shift frequency from a fundamental frequency of each phoneme determined based on the note information included in the input musical score is used for processing in an acoustic model, and a frequency obtained by adding the predetermined shift frequency to a fundamental frequency included in an acoustic feature amount estimated by the acoustic model is output to a neural vocoder.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to a voice waveform generation system, a voice waveform generation method, and a voice waveform generation program. [Background technology]

[0002] Neural network-based text-to-speech synthesis, voice conversion, and singing voice synthesis have made great progress in recent years. Depending on the experimental conditions, it is now possible to synthesize high-quality speech that is almost indistinguishable from natural speech.

[0003] To realize singing voice synthesis, it is necessary to record a singing voice corpus in advance, but the recording cost is higher than that of a text-to-speech corpus. The following two methods have been proposed as singing voice synthesis methods that take into account the recording cost of a singing voice corpus.

[0004] The first method uses a model that realizes both text-to-speech synthesis and singing voice synthesis using a single neural network (see, for example, Non-Patent Document 1). With this model, if there is singing voice data for speaker A and text-to-speech synthesis data for speaker B, it is possible to realize singing voice synthesis for speaker B and text-to-speech synthesis for speaker A, even if there is no text-to-speech synthesis data for speaker A or singing voice data for speaker B.

[0005] The second method uses a model that synthesizes singing voices only from text-to-speech synthesis data (see, for example, Non-Patent Document 2). In this model, during training, the fundamental frequency analyzed from the speech waveform and text are input, and the duration of each phoneme is estimated, and then acoustic features are estimated. During singing voice synthesis, the acoustic features are estimated by inputting text from lyric information and the fundamental frequency and duration of each phoneme from note information. [Prior art documents] [Non-patent literature]

[0006] [Non-Patent Document 1] https: / / doi.org / 10.1103 / PhysRevLett.198.111302 , Google Scholar Crossref , CAS 40. Y. Lei, S. Yang, X. Wang, Q. Xie, J. Yao, L. Xie, and D. Su, “UniSyn: An end-to-end unified model for text-to-speech and singing voice synthesis,” in Proc. AAAI, Feb. 2023, pp. 101-1 13025-1

Outdoor Tool2

Outdoor Tools3

Outdoor Tools 4

Direct Environment 5

[0007] The first method mentioned above still requires a singing voice corpus, which is an issue. The model in the second method is specifically designed for singing voice models and cannot be used for text-to-speech synthesis. The second method also has the issue of not being able to externalize the fundamental frequency of the acoustic features or the subsequent neural vocoder, resulting in a stepped input fundamental frequency sequence.

[0008] An object of the present invention is to provide a configuration capable of performing text-to-speech synthesis and singing voice synthesis using a text-to-speech synthesis corpus. [Means for solving the problem]

[0009] According to one embodiment, there is provided a speech waveform generation system capable of text-to-speech synthesis and singing voice synthesis. The speech waveform generation system includes a text analysis unit that analyzes text and outputs a phoneme sequence, an acoustic model that estimates acoustic features from the phoneme sequence, and a neural vocoder that generates a speech waveform from the acoustic features. The acoustic model includes an encoder, a variance adapter, and a decoder. The variance adapter includes a phoneme duration adjustment unit, an energy estimation unit, a fundamental frequency estimation unit, and a phoneme duration estimation unit. In singing voice synthesis, lyrics included in an input musical score are input to the text analysis unit, the phoneme duration of each phoneme is determined based on note information included in the input musical score, the fundamental frequency of each phoneme determined based on the note information included in the input musical score is subtracted by a predetermined shift frequency, and the resulting frequency is used for processing in the acoustic model. A frequency obtained by adding the predetermined shift frequency to the fundamental frequency included in the acoustic features estimated by the acoustic model is output to the neural vocoder.

[0010] The predetermined shift frequency may be calculated based on the difference between the average value of the fundamental frequency of the note information included in the input musical score and the average value of the fundamental frequency of the text-to-speech synthesis corpus.

[0011] The acoustic model may further include an embedding layer that generates phoneme embeddings from the output of the text analyzer, and skip connections to connect the phoneme embeddings to something after the encoder.

[0012] The variance adapter may further include a first convolutional layer to which the output of the energy estimator is input, a second convolutional layer to which the output of the fundamental frequency estimator is input, and an adder for adding the output of the first convolutional layer and the output of the second convolutional layer to the output of the encoder.

[0013] The acoustic model may be trained solely on a text-to-speech corpus as training data.

[0014] The acoustic features may include a fundamental frequency, a mel-cepstrum, and an aperiodic component.

[0015] The neural vocoder may be a pulse excitation source input type neural vocoder.

[0016] A speech waveform generation method according to another embodiment includes constructing a trained model including a text analysis unit that analyzes text and outputs a phoneme sequence, an acoustic model that estimates acoustic features from the phoneme sequence, and a neural vocoder that generates a speech waveform from the acoustic features. The acoustic model includes an encoder, a variance adapter, and a decoder, and the variance adapter includes an energy estimation unit, a fundamental frequency estimation unit, and a phoneme duration estimation unit. The speech waveform generation method is executed in response to input of a musical score for singing synthesis, and includes the steps of inputting lyrics included in the input musical score to the text analysis unit, determining the phoneme duration of each phoneme based on note information included in the input musical score, calculating frequencies obtained by subtracting a predetermined shift frequency from the fundamental frequency of each phoneme based on the note information included in the input musical score, and outputting the frequency obtained by adding the predetermined shift frequency to the fundamental frequency included in the acoustic features estimated by the acoustic model to the neural vocoder.

[0017] A speech waveform generation program according to yet another embodiment causes a computer to execute steps for constructing a trained model capable of text-to-speech synthesis and singing voice synthesis. The trained model includes a text analysis unit that analyzes text and outputs a phoneme sequence, an acoustic model that estimates acoustic features from the phoneme sequence, and a neural vocoder that generates a speech waveform from the acoustic features. The acoustic model includes an encoder, a variance adapter, and a decoder. The variance adapter includes a phoneme duration adjustment unit, an energy estimation unit, a fundamental frequency estimation unit, and a phoneme duration estimation unit. In singing voice synthesis, lyrics included in an input musical score are input to the text analysis unit. The phoneme duration of each phoneme is determined based on note information included in the input musical score. The fundamental frequency of each phoneme determined based on the note information included in the input musical score is subtracted by a predetermined shift frequency, and the resulting frequency is used for processing in the acoustic model. A frequency obtained by adding the predetermined shift frequency to the fundamental frequency included in the acoustic features estimated by the acoustic model is output to the neural vocoder. [Effects of the Invention]

[0018] According to the present invention, a configuration capable of text-to-speech synthesis and singing voice synthesis can be realized using a text-to-speech synthesis corpus. [Brief explanation of the drawings]

[0019] [Figure 1] FIG. 10 is a diagram illustrating differences in fundamental frequency and phoneme duration between a text-to-speech synthesis corpus and a singing corpus. [Figure 2] FIG. 2 is a schematic diagram showing an example of the configuration of a voice waveform generation model according to the present embodiment. [Figure 3] FIG. 2 is a schematic diagram showing an example of the configuration of a voice waveform generation model according to the present embodiment. [Figure 4] 4 is a schematic diagram showing an example of the configuration of the variance adapter of FIG. 2 and FIG. 3. [Figure 5] FIG. 10 is a diagram for explaining a frequency shift in the voice waveform generation model according to the present embodiment. [Figure 6]1 is a schematic diagram showing an example of a hardware configuration for realizing a voice waveform generation system according to an embodiment of the present invention; [Figure 7] 10 is a flowchart showing an example of a text-to-speech synthesis process in the speech waveform generation system according to the present embodiment. [Figure 8] 10 is a flowchart showing an example of a singing voice synthesis process in the voice waveform generation system according to the present embodiment. [Figure 9] FIG. 10 is a diagram showing an example of a result of a subjective evaluation of a speech waveform generation model according to the present embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0020] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The present invention will be described in detail with reference to the accompanying drawings, in which the same or corresponding parts are designated by the same reference numerals and will not be described repeatedly.

[0021] [A. Background technology] The first model (see Non-Patent Document 1) mentioned above realizes both text-to-speech synthesis and singing voice synthesis, but requires a singing voice corpus. The second model (see Non-Patent Document 2) mentioned above does not require a speech corpus, but has problems such as being unable to synthesize text-to-speech.

[0022] The voice waveform generation model according to this embodiment does not require a singing voice corpus as training data, but instead realizes text-to-speech synthesis and singing voice synthesis using a text-to-speech corpus.

[0023] Figure 1 illustrates the differences in fundamental frequency and phoneme duration between a text-to-speech corpus and a singing corpus. The results shown in Figure 1 show an example in which the JSUT (Basic5000) corpus was used as the text-to-speech corpus and the JSUT-Song corpus was used as the singing corpus.

[0024] Figure 1(a) shows an example of the distribution of fundamental frequencies. Referring to Figure 1(a), the frequencies of the text-to-speech corpus are distributed between approximately 100 Hz and 600 Hz, whereas the frequencies of the singing corpus are distributed between approximately 100 Hz and 760 Hz. The shapes of the distributions are also different.

[0025] Figure 1(b) shows an example of the phoneme duration distribution. As can be seen from Figure 1(b), the phoneme durations in the text-to-speech corpus are distributed within a range of 0.7 seconds, whereas the phoneme durations in the singing corpus are distributed within a range of up to 2 seconds. The distribution shapes are also different.

[0026] Thus, synthesizing singing voices that are trained using a singing voice corpus using a text-to-speech corpus with different frequencies and phoneme durations is a challenging endeavor.

[0027] [B. Speech waveform generation model] Next, an example of a voice waveform generation model employed by the voice waveform generation system according to the present embodiment will be described.

[0028] The speech waveform generation model (hereinafter simply referred to as "model") according to this embodiment realizes both text-to-speech synthesis (TTS) and singing voice synthesis (SVS) using, for example, only a text-to-speech corpus. In other words, the speech waveform generation system is capable of text-to-speech synthesis and singing voice synthesis.

[0029] As an example, the voice waveform generation model has a configuration in which a Conformer-FastSpeech 2 (hereinafter abbreviated as "CFS2") type acoustic model (see Non-Patent Document 3, etc.) and a neural vocoder are combined.

[0030] Fig. 2 is a schematic diagram showing an example of the configuration of a speech waveform generation model 1 according to this embodiment. Fig. 2(a) shows an example of the configuration when model 1 is trained, and Fig. 2(b) shows an example of the configuration when speech waveforms are synthesized using model 1.

[0031] Referring to FIG. 2, model 1 is a trained model and includes a text analysis unit 10, an acoustic model 20, and a neural vocoder 30. Model 1 is trained using a text-to-speech corpus as training data. Note that neural vocoder 30 may use other training data. At least acoustic model 20 may be trained based only on the text-to-speech corpus as training data.

[0032] The text analysis unit 10 analyzes an input text 50 and outputs a phoneme sequence 51. The phoneme sequence 51 may include accent information.

[0033] The acoustic model 20 estimates acoustic features 52 from a phoneme sequence 51. The acoustic model 20 includes an embedding layer 21, an encoder 22, a variance adapter 23, and a decoder 24.

[0034] The embedding layer 21 generates phoneme embeddings from the output of the text analysis unit 10. More specifically, the embedding layer 21 generates phoneme embeddings by sequentially arranging the phoneme sequence 51 in a vector space.

[0035] The encoder 22 converts the input phoneme embedding (discrete representation) into a continuous representation. The encoder 22 may be conformer-based or transformer-based.

[0036] The variance adapter 23 extends the phoneme-level phoneme embedding (continuous representation) to the frame level. The result extended to the frame level is output to the decoder 24.

[0037] The decoder 24 estimates acoustic features 52 from the output of the variance adapter 23. Like the encoder 22, the decoder 24 may be conformer-based or transformer-based.

[0038] The acoustic feature 52 is, for example, a WORLD feature (fundamental frequency (f0), mel-cepstrum, aperiodic component). The WORLD feature may include a voiced / unvoiced (V / UV) flag. The fundamental frequency may be a logarithmic fundamental frequency. The acoustic feature 52 may be a mel-spectrogram.

[0039] The neural vocoder 30 generates (or estimates) a speech waveform 53 from the acoustic features estimated by the acoustic model 20. The neural vocoder 30 may be, for example, a HiFi-GAN vocoder or a FIRNet neural vocoder.

[0040] Since the FIRNet neural vocoder is a pulse excitation source input type neural vocoder, when used as neural vocoder 30, acoustic model 20 is configured to output WORLD features including the fundamental frequency. In other words, a pulse waveform corresponding to the fundamental frequency is directly input to the FIRNet neural vocoder.

[0041] The FIRNet neural vocoder can control the fundamental frequency by directly inputting a pulse excitation signal corresponding to the fundamental frequency into the neural network. This allows for externalization of the fundamental frequency, which was previously impossible with conventional mel-spectrogram input-type neural vocoders (see Non-Patent Document 4, etc.). Using such a pulse excitation-type neural vocoder also makes it possible to achieve frequency shifting, which will be described later.

[0042] Furthermore, the pulse excitation source input type neural vocoder can synthesize high-quality speech waveforms even when controlling the speech rate (see Non-Patent Document 5, etc.).

[0043] Referring to Fig. 2(a), an alignment training framework may be adopted when training Model 1. By adopting the alignment training framework, an intermediate mel-spectrogram and an external aligner are not required when synthesizing speech waveforms.

[0044] More specifically, a monotonic alignment search (MAS) 28 is used. The monotonic alignment search 28 uses the output of the encoder 22 and a mel spectrogram 54 or mel cepstrum 55 to optimize the parameters of the variance adapter 23. Through training, an alignment between the mel spectrogram 54 or mel cepstrum 55 and the phoneme embedding is gradually obtained.

[0045] Neural vocoder 30 may be trained by adversarial training 72, such as using a HiFi-GAN discriminator. Acoustic model 20 and neural vocoder 30 may be trained and fine-tuned independently.

[0046] 2(b), during speech waveform synthesis, neural vocoder 30 generates a speech waveform. During text-to-speech synthesis (TTS), a phoneme sequence 51 corresponding to text 56 output by text analysis unit 10 is input to acoustic model 20. Acoustic features 52 estimated from phoneme sequence 51 by acoustic model 20 are input to neural vocoder 30. Then, neural vocoder 30 generates a speech waveform 53.

[0047] Furthermore, during singing voice synthesis (SVS), lyrics 58 included in an input musical score 57 are input to the text analysis unit 10. A phoneme sequence 51 corresponding to the lyrics 58 output by the text analysis unit 10 is input to the acoustic model 20. Furthermore, a fundamental frequency 60 of each phoneme and a phoneme duration 61 of each note are determined based on note information 59 included in the input musical score 57. The phoneme sequence 51 is referenced in determining the fundamental frequency 60 and the phoneme duration 61. The fundamental frequency 60 and the phoneme duration 61 are provided to the variance adapter 23.

[0048] The acoustic model 20 estimates acoustic features 52 based on input of a phoneme sequence 51, a fundamental frequency 60, and a phoneme duration 61. The estimated acoustic features 52 are input to the neural vocoder 30. The neural vocoder 30 then generates a speech waveform 53.

[0049] Fig. 3 is a schematic diagram showing an example of the configuration of speech waveform generation model 2 according to this embodiment. Speech waveform generation model 2 is obtained by adding a skip connection to speech waveform generation model 1 shown in Fig. 2. Fig. 3(a) shows an example of the configuration of model 2 during training, and Fig. 3(b) shows an example of the configuration during speech waveform synthesis using model 2.

[0050] Referring to FIG. 3, model 2 is a trained model and includes text analysis unit 10, acoustic model 20A, and neural vocoder 30. Model 2 is trained using a text-to-speech corpus as training data. Note that neural vocoder 30 may use other training data. At least acoustic model 20A may be trained based only on the text-to-speech corpus as training data.

[0051] Compared to the acoustic model 20 shown in FIG. 2, the acoustic model 20A further includes an adder 25 and a skip connection 26. The skip connection 26 adds the phoneme embedding generated by the embedding layer 21 to a stage subsequent to the encoder 22 via Gaussian upsampling 29. Note that the skip connection 26 is not limited to the output of the decoder 24, and the phoneme embedding may be connected to any stage subsequent to the encoder 22. A configuration including such a skip connection 26 is hereinafter also referred to as a "Phoneme Embedding Skip Connection (PESC)." The PESC is a detour path that connects the phoneme embedding (or features generated based on the phoneme embedding), which is phoneme-level information, after the encoder processing.

[0052] Using skip connections 26, phoneme embedding is added to the acoustic features generated by the acoustic model 20A. This phoneme embedding adds robustness to phoneme stretching. It is also expected to improve the clarity and naturalness of the generated speech waveform 53.

[0053] Gaussian upsampling 29 is provided to match the dimensionality of the output of the decoder 24 with the dimensionality of the phoneme embeddings produced by the embedding layer 21.

[0054] Furthermore, a weighting or normalization layer (LN: Layer Normalization) may be added to the skip connection 26. That is, weighted phoneme embeddings may be added to the encoder output or the decoder output. Alternatively, the phoneme embeddings may be input to a normalization layer, and the result obtained may be added to the encoder output or the decoder output. In this way, the skip connection 26 may include a weighting layer or a normalization layer.

[0055] The acoustic features 62 estimated by the acoustic model 20A may be WORLD features (fundamental frequency (f0), mel-cepstrum, aperiodic components) instead of mel-spectrograms. The WORLD features may include a voiced / unvoiced (V / UV) flag. The fundamental frequency may be a logarithmic fundamental frequency.

[0056] [C. Variance Adapter] Next, the variance adapter 23 of the voice waveform generation models 1 and 2 will be described.

[0057] Fig. 4 is a schematic diagram showing an example of the configuration of variance adapter 23 of Fig. 2 and Fig. 3. Referring to Fig. 4, variance adapter 23 includes a phoneme duration adjustment section 231, an energy estimation section 232, an energy convolution layer 233, a fundamental frequency estimation section 234, a fundamental frequency convolution layer 235, an adder 236, a phoneme duration estimation section 237, and a compensation section 238.

[0058] The phoneme duration adjustment unit 231 extends the phoneme-level phoneme embedding to the frame level by upsampling the phoneme embedding (continuous representation) according to the phoneme duration. The phoneme duration adjustment unit 231 may include a Gaussian upsampling layer.

[0059] The energy estimation unit 232 estimates energy from phoneme embedding (continuous expression) during text-to-speech synthesis. The output (estimated output) of the energy estimation unit 232 is input to the energy convolution layer 233. The energy convolution layer 233 performs a convolution operation on the estimated energy.

[0060] The result of the convolution operation is added to the phoneme embedding (continuous representation) by adder 236. Adder 236 adds the output of energy convolution layer 233 and the output of fundamental frequency convolution layer 235 to the output of encoder 22.

[0061] During learning, parameters are optimized so that the mean square error (MSE) of the energy estimated by the energy estimation unit 232 relative to the target energy 65 is minimized.

[0062] During text-to-speech synthesis, the fundamental frequency estimation unit 234 estimates the fundamental frequency from the phoneme embedding (continuous representation). The output (estimated output) of the fundamental frequency estimation unit 234 is input to the fundamental frequency convolution layer 235. The fundamental frequency convolution layer 235 performs a convolution operation on the estimated fundamental frequency. The result of the convolution operation is added to the phoneme embedding (continuous representation) by the adder 236.

[0063] During learning, the parameters are optimized so that the mean square error of the fundamental frequency estimated by the fundamental frequency estimation unit 234 relative to the target fundamental frequency 64 is minimized.

[0064] The phoneme duration estimation unit 237 estimates the duration of each phoneme from the phoneme embedding (continuous representation). During speech waveform synthesis, the estimated phoneme duration is input to the phoneme duration adjustment unit 231 (TTS) or the compensation unit 238 (SVS).

[0065] During training, parameters are optimized so that the mean square error (logarithmic domain) of the phoneme duration estimated by the phoneme duration estimation unit 237 relative to the target phoneme duration 63 is minimized.

[0066] The compensation unit 238 determines the phoneme duration based on the duration of each phoneme estimated by the phoneme duration estimation unit 237 and the phoneme duration 61 of each note in the note information 59. The determined phoneme duration is input to the phoneme duration adjustment unit 231.

[0067] In this way, when training the variance adapter 23 of the acoustic model 20, 20A, the phoneme duration, fundamental frequency, and energy are estimated from the input text 50.

[0068] [D. Frequency Shift] In order to improve the accuracy of estimating the fundamental frequency by the acoustic model, when synthesizing singing voice, the fundamental frequency 60 determined based on the note information 59 may be frequency-shifted before estimating the acoustic feature 62, and the estimated acoustic feature 62 may then be frequency-shifted to return to its original value.

[0069] A pulse excitation input neural vocoder can estimate fundamental frequencies outside the range of the training data (text-to-speech corpus) by applying external filtering. In contrast, an acoustic model cannot directly estimate fundamental frequencies outside the range of the training data. This can lead to a decrease in the accuracy of fundamental frequency estimation. Therefore, a range within which the acoustic model can maintain estimation accuracy is determined based on the training data, and fundamental frequencies within that range are input to the acoustic model.

[0070] 5 is a diagram for explaining frequency shifting in a voice waveform generation model according to the present embodiment. Referring to FIG. 5, a fundamental frequency 60 (f0) is determined based on note information 59 included in an input musical score 57. The determined fundamental frequency is shifted by a predetermined shift frequency f 0,shift For example, the frequency is shifted from the fundamental frequency f0 to the shift frequency f 0,shift The shifted fundamental frequency 66(log(f0-f 0,shift )) is given to the acoustic model 20A. In this way, the fundamental frequency 60 of each phoneme determined based on the note information 59 included in the input musical score 57 is shifted by a predetermined shift frequency f 0,shift The frequencies reduced by this amount are used for processing in the acoustic model.

[0071] Shift frequency f 0,shift is calculated based on the difference between the average value of the fundamental frequency of the note information 59 included in the target input musical score 57 and the average value of the fundamental frequency of the text-to-speech synthesis corpus, which is the training data. For example, the shift frequency f 0,shift = (average value of fundamental frequency of musical note information 59) - (average value of fundamental frequency of text-to-speech synthesis corpus). For convenience of processing, rounding may be performed.

[0072] The average value of the fundamental frequency of note information 59 may be calculated each time input musical score 57 is provided. The average value of the fundamental frequency of the text-to-speech synthesis corpus may be calculated in advance, such as during training.

[0073] The estimated fundamental frequency (f0) included in the acoustic feature 62 estimated by the acoustic model 20A is shifted to the shift frequency f 0,shift For example, by shifting the fundamental frequency sequence (f0) included in the acoustic feature 62 by a shift frequency f 0,shift The result of adding (log(f0+f 0,shift )) is provided to the neural vocoder 30. In this way, the predetermined shift frequency f 0,shift The resulting frequency is output to the neural vocoder 30.

[0074] The accuracy of fundamental frequency estimation by the acoustic model can be improved by frequency-shifting (subtracting) the fundamental frequency sequence so that it matches the average fundamental frequency of the text-to-speech corpus, then estimating it using an acoustic model, and then frequency-shifting (adding) the estimated fundamental frequency sequence again.

[0075] For example, the note information 59 of the input musical score 57 may be lowered by one octave, and then acoustic features may be estimated, and the estimated acoustic features may be raised by one octave and provided to the neural vocoder.

[0076] Although FIG. 5 illustrates the model 2 shown in FIG. 4, the same processing can be applied to the model 1 shown in FIG.

[0077] [E. Hardware configuration example] Next, an example of a hardware configuration for realizing the voice waveform generation system according to the present embodiment will be described. The voice waveform generation system according to the present embodiment may be realized using the same computing resource or different computing resources. The computing resource may be provided using, for example, a general-purpose computer or a mobile device.

[0078] FIG. 6 is a schematic diagram showing an example of a hardware configuration for realizing the voice waveform generation system according to the present embodiment.

[0079] 6, the information processing device 300 includes, as main hardware components, a central processing unit (CPU) 302, a graphics processing unit (GPU) 304, a main memory 306, an input device 308, a network interface (I / F) 310, a storage 312, an input interface 322, an output interface 324, and an optical drive 326. These components are connected to each other via an internal bus 330.

[0080] The CPU 302 and / or the GPU 304 are processors that execute processes necessary to implement the system. A plurality of CPUs 302 and GPUs 304 may be provided, and each may have a plurality of cores.

[0081] The voice waveform generation model according to this embodiment has a low processing load, so the GPU 304 may not be necessary.

[0082] The main memory 306 is a storage area that temporarily stores (or caches) program code, work data, etc. when the processor (CPU 302 and / or GPU 304) executes processing, and is composed of volatile memory such as DRAM (dynamic random access memory) or SRAM (static random access memory).

[0083] The input device 308 is a device that accepts instructions and operations from the user, and is configured by, for example, a keyboard, a mouse, a touch panel, a pen, and the like.

[0084] The network interface 310 exchanges data with any information processing device on the Internet or an intranet. Any communication method such as Ethernet (registered trademark), wireless LAN (local area network), or Bluetooth (registered trademark) can be used as the network interface 310.

[0085] The input interface 322 receives an audio signal from a microphone 332 .

[0086] The output interface 324 outputs an audio signal to a speaker 334 .

[0087] The optical drive 326 reads information stored on an optical disk 328, such as a CD-ROM (compact disc read only memory) or a DVD (digital versatile disc), and outputs the information to other components via the internal bus 330. The optical disk 328 is an example of a non-transitory recording medium, and is distributed with any program stored therein in a non-volatile manner. The optical drive 326 reads the program from the optical disk 328 and installs it in the storage 312 or the like, causing the computer to function as the information processing device 300.

[0088] The storage 312 stores programs and data necessary to implement the system. The storage 312 is configured with a non-volatile storage device such as a hard disk or a solid state drive (SSD).

[0089] More specifically, the storage 312 stores an operating system (OS) (not shown), as well as a text analysis program 314, an acoustic model 316, and a neural vocoder 318, which are examples of speech waveform generation programs. The text analysis program 314, the acoustic model 316, and the neural vocoder 318 may be integrated into one program (speech waveform generation program).

[0090] The storage 312 may also store a training program 320 and a text-to-speech corpus 350 .

[0091] The text analysis program 314 includes computer readable instructions for implementing the text analysis portion 10 of Model 1 or Model 2.

[0092] Acoustic model 316 includes computer-readable instructions for implementing Model 1 acoustic model 20 or Model 2 acoustic model 20A. Acoustic model 316 may include a network structure definition, a trained parameter set, and hyperparameters.

[0093] Neural vocoder 318 includes computer-readable instructions for implementing Model 1 or Model 2 neural vocoder 30. Neural vocoder 318 may include a definition of the network structure, a trained parameter set, and hyperparameters.

[0094] The training program 320 includes computer-readable instructions for training an acoustic model and / or a neural vocoder. The training program 320 may include algorithms for performing training, such as monotonic alignment search 28.

[0095] The text-to-speech corpus 350 may be downloaded from an external server as needed.

[0096] Some of the libraries and functional modules required when the processor (CPU 302 and / or GPU 304) executes a program may be replaced with libraries or functional modules provided as standard by the OS. In this case, the program itself does not include all of the program modules required to realize the corresponding functions, but the program can achieve the desired processing by being installed in the OS execution environment. Furthermore, general-purpose libraries or functional modules licensed for use under a specific license may be used. Even a program that does not include some of these libraries or functional modules is within the technical scope of the present invention.

[0097] Furthermore, these programs may not only be distributed by being stored on any of the above-mentioned recording media, but may also be distributed by being downloaded from a server or the like via the Internet or an intranet.

[0098] All or part of the functions realized by the processor (CPU 302 and / or GPU 304) executing the programs may be realized using a hard-wired circuit such as an integrated circuit, for example, an application specific integrated circuit (ASIC) or a field-programmable gate array (FPGA).

[0099] 6 shows an example of a configuration using a single computer, but the present invention is not limited to this. Multiple computers connected via a computer network may cooperate explicitly or implicitly to execute the processes required to realize the system. For example, the voice waveform generation system according to this embodiment may be realized using computing resources on the cloud.

[0100] Those skilled in the art will be able to implement the information processing device 300 according to this embodiment by appropriately using technology suited to the era in which the present invention is implemented.

[0101] [F. Processing Procedure] Next, an example of the processing procedure of the speech waveform generation system according to this embodiment will be described. Model 1 shown in Fig. 2 or Model 2 shown in Fig. 3 may be configured by the processor of the information processing device 300 executing the text analysis program 314, the acoustic model 316, and the neural vocoder 318. Furthermore, each step shown in Fig. 7 and Fig. 8 may be realized by the processor of the information processing device 300 executing the text analysis program 314, the acoustic model 316, and the neural vocoder 318.

[0102] (f1: Text-to-speech synthesis) Fig. 7 is a flowchart showing an example of the text-to-speech synthesis process in the speech waveform generation system according to the present embodiment. Information processing device 300 executes the following process in response to text 56 for text-to-speech synthesis. The process example shown in Fig. 7 is based on model 1 shown in Fig. 2, but is similar to that for model 2 shown in Fig. 3, except for the process of skip connection 26.

[0103] 4 and 7, information processing device 300 acquires text 56 (step S100), and analyzes acquired text 56 to generate phoneme sequence 51 (step S102). That is, text 56 is input to text analysis unit 10.

[0104] The information processing device 300 sequentially arranges the phoneme sequence 51 in a vector space to generate phoneme embeddings (step S104).

[0105] The information processing device 300 converts the phoneme embedding, which is a discrete representation, into a continuous representation (step S106).

[0106] The information processing device 300 extends the phoneme-level phoneme embedding (continuous representation) to the frame level (step S108). The process of extending the phoneme-level phoneme embedding (continuous representation) to the frame level includes processes of estimating the phoneme duration, fundamental frequency, and energy from the phoneme embedding (continuous representation). The estimated fundamental frequency and energy are added to the phoneme embedding (continuous representation). In this way, the feature based on the energy estimated by the energy estimation unit 232 (output of the energy convolution layer 233) and the feature based on the fundamental frequency estimated by the fundamental frequency estimation unit 234 (output of the fundamental frequency convolution layer 235) are added to the output of the encoder 22.

[0107] The information processing device 300 extends the phoneme-level phoneme embedding to the frame level by repeating each phoneme based on the estimated phoneme duration. More specifically, the phoneme duration adjustment unit 231 upsamples the encoder output after addition according to the duration of each phoneme estimated by the phoneme duration estimation unit 237.

[0108] The information processing device 300 estimates the acoustic features 52 from the phoneme embedding extended to the frame level (step S110).

[0109] The information processing device 300 generates a speech waveform 53 from the estimated acoustic feature quantity 52 (step S112).

[0110] The processes of steps S102 to S112 are repeated until all of the text 56 has been processed (NO in step S114).

[0111] When all the texts 56 have been processed (YES in step S114), the process ends.

[0112] (f2: singing voice synthesis) Fig. 8 is a flowchart showing an example of a singing voice synthesis process in the voice waveform generation system according to the present embodiment. The information processing device 300 executes the following process in response to an input musical score 57 for singing voice synthesis. The process example shown in Fig. 8 is based on Model 1 shown in Fig. 2, but it is similar in the case of Model 2 shown in Fig. 3, except for the process of skip connection 26.

[0113] 4 and 8, the information processing device 300 acquires the input musical score 57 (step S200), and analyzes the lyrics 58 included in the input musical score 57 to generate the phoneme sequence 51 (step S202). That is, the lyrics 58 included in the input musical score 57 are input to the text analysis unit 10.

[0114] The information processing device 300 sequentially arranges the phoneme sequence 51 in a vector space to generate phoneme embeddings (step S204). The information processing device 300 converts the phoneme embeddings, which are discrete representations, into continuous representations (step S206).

[0115] In parallel, the information processing device 300 determines the fundamental frequency 60 of each phoneme and the phoneme duration 61 of each note based on the note information 59 included in the input musical score 57 and the phoneme sequence 51 (step S208).

[0116] The information processing device 300 extends the phoneme-level phoneme embedding (continuous representation) to the frame level (step S210). The process of extending the phoneme-level phoneme embedding (continuous representation) to the frame level includes the process of adding the convolution calculation result of the fundamental frequency 60 of each phoneme to the phoneme embedding (continuous representation), the process of compensating the duration of each phoneme estimated from the addition result with the phoneme duration 61 for each note, and the process of extending the phoneme-level phoneme embedding to the frame level by repeating the process based on the compensated result for each phoneme. In this way, the feature based on the fundamental frequency 60 of each phoneme (output of the fundamental frequency convolution layer 235) is added to the output of the encoder 22.

[0117] At this time, the fundamental frequency estimation unit 234 is disabled. That is, the outputs of the energy estimation unit 232 and the fundamental frequency estimation unit 234 are not added to the output of the encoder 22.

[0118] The information processing device 300 estimates acoustic features 52 from the phoneme embedding extended to the frame level (step S212). More specifically, the phoneme duration adjustment unit 231 upsamples the output of the encoder after addition in accordance with the duration of each phoneme determined based on the duration of each phoneme estimated by the phoneme duration estimation unit 237 and the phoneme duration 61 of each note.

[0119] The information processing device 300 generates a speech waveform 53 from the estimated acoustic feature quantity 52 (step S214).

[0120] The processing of steps S202 to S214 is repeated until all of the input musical score 57 has been processed (NO in step S216).

[0121] When all of the input musical score 57 has been processed (YES in step S112), the process ends.

[0122] When frequency shifting is performed, a process may be added in which a shift frequency is calculated based on the acquired note information 59, the shift frequency is subtracted from the fundamental frequency 60 of each phoneme in step S208, and the shift frequency is added to the fundamental frequency included in the estimated acoustic feature 52 in step S212.

[0123] [G. Performance Evaluation] Next, an example of performance evaluation of a model according to this embodiment will be described.

[0124] (1) Dataset and Prerequisites The JSUT (Basic5000) corpus was used for training the model. The JSUT-Song corpus was used only for evaluating the performance of singing voice synthesis.

[0125] Similar to the ESPnet2-TTS recipe, for text-to-speech synthesis, we used 4500 utterances, 250 utterances, and 250 utterances as the training, validation, and test datasets, respectively. For Japanese text-to-speech synthesis, we used a G2P function based on pyopenjtalk enhanced with prosodic symbols.

[0126] For the subjective evaluation of text-to-speech synthesis, 10 randomly selected utterances were used. For the objective and subjective evaluation of singing synthesis, the first phrases of 10 songs included in JSUT-Song were used. An example of input sheet music is shown below.

[0127] [Table 1]

[0128] In Table 1, "Phoneme" is a phoneme obtained from the lyrics, and "Note" represents a pitch obtained from a note in the musical score. For example, G4 is the so-called "G" note, with a frequency of 391.995 Hz. "Number of frames" represents the tempo specified in the musical score (in the case of Table 1, 120 BPM of quarter notes) and the duration of the sound obtained from the note, expressed in frames. Based on the lyrics analyzed by the text analysis unit 10, the phoneme duration estimation unit 237 estimates the duration of each phoneme, but as mentioned above, the phoneme duration is compensated based on information obtained from the musical note information 59.

[0129] To give a specific example, the lyrics of the song in Table 1 (the elementary school song "Den-den Mushi Mushi Katatsumuri") begin with "de," which corresponds to the phonemes "d" and "e." If the phoneme duration estimation unit 237 estimates 6 frames for "d" and 10 frames for "e," the 37.5 frames in Table 1 are proportionally allocated in this 10:6 ratio ("d" is (37.5 x 6) / (10 + 6), and "e" is (37.5 x 10) / (10 + 6)). The compensated result is provided to the phoneme duration adjustment unit 231. The Gaussian upsampling in the phoneme duration adjustment unit 231 is configured to be able to process real numbers.

[0130] Regarding frequency shift, the average fundamental frequency of the training data set was approximately 206 Hz. The minimum and maximum frequencies in the note information were C4 (261.6 Hz) and C5 (523.3 Hz), respectively. 0,shift The calculation was =261.6+(523.3-261.6) / 2-206=186.5Hz.

[0131] The acoustic features input to HiFi-GAN were an 80-dimensional mel-spectrogram band-limited to 7600 Hz. The WORLD features were 50-dimensional mel-cepstral coefficients with a warping factor α = 0.455, a 3-dimensional aperiodic component, and a logarithmic scale continuous f0. The STFT length and shift length were 1024 and 240 samples, respectively.

[0132] (2) Objective evaluation For objective evaluation, we measured Mel-Spectrogram Distortion (MCD) and Root Mean Square Error (RMSE) of logarithmic scale continuous f0. Table 1 shows examples of evaluation results for different types of acoustic model, neural vocoder, and acoustic features. The highest evaluation results in each category are shown in bold.

[0133] [Table 2]

[0134] In the acoustic model (AM) column of Table 1, CFS2 refers to the Conformer-FastSpeech 2 acoustic model shown in FIG. 2, and PESC refers to the CFS2 acoustic model that employs PESC shown in FIG.

[0135] The Neural Vocoder (NV) column in Table 1 indicates whether the neural vocoder used was the HiFi-GAN vocoder or the FIRNet neural vocoder.

[0136] Experimental results with a check mark in the "Input f0shift" column in Table 1 indicate that frequency shift was performed.

[0137] The column for acoustic feature in Table 1 indicates whether the mel spectrogram or WORLD was used as the acoustic feature.

[0138] Table 1 shows the MCD and RMSE values ​​for text-to-speech synthesis (TTS) and singing voice synthesis (SVS).

[0139] For text-to-speech synthesis (TTS), all models can generate high-quality voice waveforms. For singing voice synthesis (SVS), we used the FIRNet neural vocoder as the neural vocoder, and we found that frequency shifting, combined with PESC as shown in Figure 3, can improve the quality of the voice waveforms.

[0140] (3) Subjective evaluation For subjective evaluation, we used the Mean Opinion Score (MOS) test. Twenty native Japanese speakers rated each of the 200 samples (10 utterances x 20 models) on a 5-point scale.

[0141] FIG. 9 shows an example of the results of a subjective evaluation of a speech waveform generation model according to the present embodiment. In FIG. 9, "CFS2" refers to the Conformer-FastSpeech 2 acoustic model (acoustic model 20) shown in FIG. 2, and "PESC" refers to the CFS2 acoustic model (acoustic model 20A) that employs PESC shown in FIG. 3. "HFG" refers to the HiFi-GAN vocoder, and "FIRN" refers to the FIRNet neural vocoder. "IFS" refers to the case where frequency shifting is employed. "Melspc" refers to the case where mel spectrograms are employed as acoustic features, and "WORLD" refers to the case where WORLD features are employed as acoustic features. "ORG" refers to the samples themselves included in the training data.

[0142] Figure 9(a) shows the evaluation results for text-to-speech (TTS). For text-to-speech synthesis, the model using the HiFi-GAN vocoder as the neural vocoder can be said to achieve higher quality speech waveform synthesis, but it can also be seen that adopting PESC can improve performance regardless of the neural vocoder.

[0143] Figure 9(b) shows the evaluation results for singing voice synthesis (SVS). For singing voice synthesis, we can see that the quality can be significantly improved by adopting frequency shift in Model 2 (acoustic model 20A with PESC and FIRNet neural vocoder) shown in Figure 3.

[0144] Figure 9(c) shows the evaluation results for singing voice synthesis (SVS) when the phoneme duration was halved (played back at double speed). The tempo was 240 BPM (Beat Per Minute). Compared to the HiFi-GAN vocoder, the FIRNet neural vocoder can achieve robust processing against phoneme stretching.

[0145] Figure 9(d) shows the evaluation results for singing voice synthesis (SVS) when the fundamental frequency is halved (lowered by one octave). Figure 9(e) shows the evaluation results for singing voice synthesis (SVS) when the fundamental frequency is doubled (upper by one octave). These evaluation results show that, compared to the HiFi-GAN vocoder, adopting the FIRNet neural vocoder enables processing that is robust to changes in fundamental frequency.

[0146] [H. Variations] In the above explanation, models for text-to-speech synthesis and singing voice synthesis are exemplified, but the present invention can also be applied to neural voice models for voice conversion, singing voice enhancement, and the like.

[0147] [I. Advantages] The speech waveform generation model according to this embodiment is a single model that can realize both text-to-speech synthesis and singing voice synthesis using only a text-to-speech corpus, and can robustly handle phoneme elongation, changes in fundamental frequency, etc.

[0148] The embodiments disclosed herein should be considered to be illustrative in all respects and not restrictive. The scope of the present invention is defined by the claims, not by the description of the above embodiments, and is intended to include all modifications within the meaning and scope of the claims. [Explanation of symbols]

[0149] 1,2 Speech waveform generation model (model), 10 Text analysis unit, 20,20A,316 Acoustic model, 21 Embedding layer, 22 Encoder, 23 Variance adapter, 24 Decoder, 25,236 Adder, 26 Skip connection, 28 Monotonic alignment search, 29 Gaussian upsampling, 30,318 Neural vocoder, 50 Input text, 51 Phoneme sequence, 52,62 Acoustic features, 53 Speech waveform, 54 Mel spectrogram, 55 Mel cepstrum, 56 Text, 57 Input music score, 58 Lyrics, 59 Note information, 60 Fundamental frequency, 61 Phoneme duration, 63 Target phoneme duration, 64 Target fundamental frequency, 65 Target energy, 66 Shifted fundamental frequency, 67 Corrected acoustic features, 231 Phoneme duration adjustment unit, 232 Energy estimation unit, 233 energy convolution layer, 234 fundamental frequency estimation unit, 235 fundamental frequency convolution layer, 237 phoneme duration estimation unit, 238 compensation unit, 300 information processing device, 302 CPU, 304 GPU, 306 main memory, 308 input device, 310 network interface, 312 storage, 314 text analysis program, 320 learning program, 322 input interface, 324 output interface, 326 optical drive, 328 optical disk, 330 internal bus, 332 microphone, 334 speaker, 350 text-to-speech synthesis corpus.

Claims

1. A voice waveform generation system capable of text-to-speech synthesis and singing voice synthesis, a text analysis unit that analyzes text and outputs a phoneme sequence; an acoustic model for estimating acoustic features from the phoneme sequence; a neural vocoder that generates a speech waveform from the acoustic features; The acoustic model comprises an encoder, a variance adapter, and a decoder; the variance adapter comprises a phoneme duration adjustment unit, an energy estimation unit, a fundamental frequency estimation unit, and a phoneme duration estimation unit; In the singing voice synthesis, The lyrics included in the input musical score are input to the text analysis unit, A phoneme duration of each phoneme is determined based on note information included in the input musical score; a frequency obtained by subtracting a predetermined shift frequency from the fundamental frequency of each phoneme determined based on note information included in the input musical score is used for processing in the acoustic model; a frequency obtained by adding the predetermined shift frequency to a fundamental frequency included in the acoustic feature estimated by the acoustic model, and outputting the result to the neural vocoder.

2. The speech waveform generation system according to claim 1 , wherein the predetermined shift frequency is calculated based on the difference between the average value of the fundamental frequency of the note information included in the input musical score and the average value of the fundamental frequency of a text-to-speech synthesis corpus.

3. The acoustic model is an embedding layer that generates phoneme embeddings from the output of the text analysis unit; 2. The speech waveform generation system of claim 1, further comprising: a skip connection for connecting the phoneme embedding somewhere after the encoder.

4. The variance adapter is a first convolutional layer to which the output of the energy estimation unit is input; a second convolutional layer to which the output of the fundamental frequency estimation unit is input; 4. The audio waveform generation system according to claim 1, further comprising an adder for adding an output of the first convolutional layer and an output of the second convolutional layer to an output of the encoder.

5. a step of constructing a trained model including a text analysis unit that analyzes text and outputs a phoneme sequence, an acoustic model that estimates acoustic features from the phoneme sequence, and a neural vocoder that generates a speech waveform from the acoustic features, wherein the acoustic model includes an encoder, a variance adapter, and a decoder, and the variance adapter includes an energy estimation unit, a fundamental frequency estimation unit, and a phoneme duration estimation unit; Executed in response to input of musical score for singing voice synthesis, inputting lyrics included in an input musical score into the text analysis unit; determining the phoneme duration of each phoneme based on note information included in the input musical score; calculating a frequency obtained by subtracting a predetermined shift frequency from the fundamental frequency of each phoneme based on note information included in the input musical score; and outputting to the neural vocoder a frequency obtained by adding the predetermined shift frequency to a fundamental frequency included in the acoustic feature estimated by the acoustic model.

6. A voice waveform generation program, comprising: Executing a step of constructing a trained model capable of text-to-speech synthesis and singing voice synthesis; the trained model includes a text analysis unit that analyzes text and outputs a phoneme sequence, an acoustic model that estimates acoustic features from the phoneme sequence, and a neural vocoder that generates a speech waveform from the acoustic features; The acoustic model comprises an encoder, a variance adapter, and a decoder; the variance adapter comprises a phoneme duration adjustment unit, an energy estimation unit, a fundamental frequency estimation unit, and a phoneme duration estimation unit; In the singing voice synthesis, The lyrics included in the input musical score are input to the text analysis unit, A phoneme duration of each phoneme is determined based on note information included in the input musical score; a frequency obtained by subtracting a predetermined shift frequency from the fundamental frequency of each phoneme determined based on note information included in the input musical score is used for processing in the acoustic model; a frequency obtained by adding the predetermined shift frequency to a fundamental frequency included in the acoustic feature estimated by the acoustic model, and outputting the frequency to the neural vocoder.

Images