An end-to-end text-to-speech synthesis method and system combining autoregression
By combining the end-to-end text synthesis method of autoregression, the audio discrete encoding and time information is obtained using semantics and continuous language modules, the problem of poor data dependence and generation stability in the prior art is solved, and fast and high-quality voice is achieved generate.
Patent Information
- Application Number
- CN202410543863.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-05-06
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2044-05-06
AI Technical Summary
The existing end-to-end TTS method relies on a large amount of data to generate poor voice emotional information; the Zero Shot TTS method generates poor sound stability, slow inference speed, and short prompt audio information is not enough to learn personal speaking style.
Using an end-to-end text synthesis method combining autoregression, audio discrete encoding and duration information is obtained through semantic language modules and continuous language modules, information encoding modules are passed to obtain prior distributions, hidden variables are sampled and audio is generated, and the use of One-dimensional convolution and multi-layer RVQ residual modules reduce frame rates, introduce continuous projection layer aligned phoneme-level time information, and use adversarial network training targets to maximize the variational lower bound ELBO.
It improves the speed and stability of speech generation, reduces data dependence, and enhances voice emotional expression. Only a small amount of data is required when cloning sounds, and the inference speed is fast and the generation quality is high.
Smart Images

Figure CN118135990B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of speech synthesis, and in particular to an end-to-end text-to-speech synthesis method and system combined with autoregression. Background Art
[0002] In recent years, Text to Speech (TTS) has made significant progress. The most advanced TTS methods can be divided into end-to-end and zero-shot methods.
[0003] End-to-end TTS technology is a revolutionary approach that allows natural and fluent speech to be generated directly from text without complex intermediate steps. The core idea of the end-to-end TTS model is to map text directly to speech waveforms, usually through neural networks. The advantage of this model is that it simplifies the system architecture, making the entire speech synthesis process more efficient and direct. Through an end-to-end approach, the TTS system can better capture the association between text and speech, thereby generating more natural and fluent speech output, and is able to synthesize high-quality speech for single or multiple speakers. Related works include Tacotron2, FastSpeech, VITS, etc.
[0004] Zero-shot TTS is a speech synthesis technology that can generate speech without speech samples from a specific speaker or with very limited training data. The core idea of this technology is to enable the system to generate natural and fluent speech directly from text without training for a specific speaker by leveraging advanced generative models and representation learning techniques. Generally speaking, large-scale generative models trained on tens of thousands of hours of data sets are required to perform zero-shot TTS, such as VALL-E, XTTS, yourTTS, etc. These powerful models can effectively synthesize speech with just a single voice prompt, eliminating the need for large amounts of data preparation.
[0005] Although the end-to-end TTS method simplifies the system architecture and improves the training efficiency, it also has some defects. First, to clone a voice, a large amount of data of the voice is required for training, which is highly dependent on data. Second, the generated voice has poor emotional information and the voice intonation is relatively flat. The Zero Shot method relies on a data set of tens of thousands of hours, and can clone the voice without targeted training, only about 5 seconds of target audio. However, the current Zero ShotTTS method mainly discretizes the audio, represents the audio as a discrete code, trains the relevant model using the autoregressive training method of the large language model, and finally uses the relevant decoding components according to the output of the model. This method brings some defects. First, the sound generated by the model is unstable, and it is easy to swallow words, change the tone, and make strange sounds. Second, it can only use short prompt audio of 3-10 seconds, and the amount of information in the short prompt audio is not enough to guide the model to learn the individual's speaking style, such as the rhythm and pronunciation habits of personal speech. 3. The speed of generating audio by autoregression is relatively slow. Assuming that 1 second of audio is encoded as 75 tokens, generating 1 second of audio requires 75 inferences! In addition, in order to obtain better audio effects, multiple inferences are generally performed to obtain better inference results, which is even slower. According to the practice of the vall-e model, it takes 1 to 2 minutes of waiting time to generate 1 minute of audio. Summary of the invention
[0006] The present invention provides an end-to-end text-to-speech synthesis method and system combined with autoregression, which can at least partially improve the above-mentioned problems.
[0007] To achieve the above object, the present invention adopts the following technical solutions:
[0008] An end-to-end text-to-speech synthesis method combined with autoregression, comprising:
[0009] Input the current reference audio and the text to be inferred into the semantic language module of the speech model, and infer the audio discrete encoding;
[0010] Input the current reference audio and the text to be inferred into the continuous language module of the speech model to infer the duration information of the audio;
[0011] The current reference audio, the text to be inferred, the audio discrete code and the duration information are passed into the information encoding module of the speech model to obtain the mean of the prior distribution and variance ;
[0012] Based on the mean and variance , sampling to obtain the hidden variables of the prior distribution ;
[0013] The hidden variable and the current reference audio are passed to the generator module to generate audio.
[0014] Preferably, when the audio discrete encoding is obtained by inference:
[0015] The HuBert module based on the speech model encodes the current reference audio, and the encoding result Expressed as ,in Indicates the time length of the current reference audio, 50 is the frame rate of the current reference audio, represents the hidden dimension, Indicates that the value is in the real number domain and the dimension is multi-dimensional;
[0016] Using one-dimensional convolution and multi-layer RVQ residual modules, the frame rate of the current reference audio is reduced from 50 frames to 25 frames, and then compressed to obtain audio discrete coding.
[0017] Preferably, the duration information is information at the phoneme level, and the duration information is aligned with the text to be inferred at the phoneme level through a continuous projection layer, so that the speech model knows the time when each phoneme should be pronounced.
[0018] Preferably, the speech model is trained using an adversarial network training method, and its training goal is to maximize the variational lower bound ELBO. The formula for maximizing the likelihood is:
[0019]
[0020] in, Represents a given text Hidden variables The prior distribution of is a data sample The likelihood function of is the approximate posterior distribution;
[0021] The training loss is -ELBO, where the training loss is the reconstruction loss and KL divergence The sum of is: .
[0022] Preferably, the single forward and backward training process is specifically as follows:
[0023] Get the linear spectrum of the audio used as training samples , the linear spectrum Through the semantic quantization module, the audio discrete code is obtained , Indicates the length of the current audio. represents hidden dimension;
[0024] Discrete audio encoding , duration information, text phonemes , and the linear spectrum Input to the information encoding module to obtain the mean of the posterior distribution of the current training sample and variance ;
[0025] Linear spectrum Input the posterior encoder to obtain the mean of the posterior distribution of the current training sample ,variance , and hidden variables ;
[0026] The latent variable of the posterior distribution and linear spectrum Perform affine transformation through the flow module to obtain the affine result ;
[0027] The latent variable of the posterior distribution and linear spectrum , through the generator module, sampling generates audio ;
[0028] The audio and linear spectrum Convert to Mel spectrum and , and calculate the reconstruction loss ;
[0029] According to the affine result , the mean of the prior distribution , the mean of the posterior distribution , the variance of the posterior distribution Calculate KL divergence;
[0030] The discriminator generates a loss according to the KL divergence calculation, wherein the discriminator uses a multi-cycle discriminator of HiFi-GAN;
[0031] Computational audio discrete coding MSE loss of decoding and encoding results;
[0032] According to the loss of each module, the parameters of the speech model are adjusted.
[0033] Preferably, the flow module is reversible, and improves the distribution complexity of the Gaussian distribution through multiple affine coupling layers, thereby improving the expressiveness of the speech model.
[0034] The embodiment of the present invention further provides an end-to-end text-to-speech system combined with autoregression, which includes:
[0035] A semantic language module, which is used to input the current reference audio and the text to be inferred into the semantic language module of the speech model, and infer the audio discrete code;
[0036] A continuous language module is used to input the current reference audio and the text to be inferred into the continuous language module of the speech model to infer the duration information of the audio;
[0037] An information encoding module is used to pass the current reference audio, the text to be inferred, the audio discrete code and the duration information into the information encoding module of the speech model to obtain the mean of the prior distribution and variance ;
[0038] A priori encoding module for and variance , sampling to obtain the hidden variables of the prior distribution ;
[0039] The generator module is used to transform the latent variables and the current reference audio are passed to the generator module to generate audio.
[0040] The present invention provides an end-to-end text-to-speech method and system combined with autoregression. The core of the method is that two autoregressive models SLM and DLM are introduced to first encode the audio based on the speech model, reduce the frame rate of the audio, and then compress it into a discrete codebook; this means that to generate 3 seconds of audio, SLM theoretically only needs to infer 75 tokens, which is more than 3 times faster than the pure autoregressive text-to-speech model, such as Vall-e. At the same time, in view of the problem that the autoregressive text-to-speech model is prone to swallowing words, the duration information at the phoneme level is introduced, and the duration information is aligned with the text at the phoneme level through a continuous projection layer (Duration Project), so that the model knows the time when each phoneme should be pronounced. Finally, based on the reference audio, the text to be inferred, the audio discrete encoding and the duration information, the synthesized audio is obtained; thereby solving the problems of the pure autoregressive text-to-speech model having a slow reasoning speed, easy swallowing of words, and the need for the end-to-end model to rely on a large amount of audio data. BRIEF DESCRIPTION OF THE DRAWINGS
[0041] Figure 1 It is a flowchart of an end-to-end text-to-speech synthesis method combined with autoregression provided by the first embodiment of the present invention;
[0042] Figure 2It is a general framework diagram of the CMM-TTS model provided by the first embodiment of the present invention;
[0043] Figure 3 is a structural diagram of an information encoding module provided by the first embodiment of the present invention;
[0044] Figure 4 A schematic diagram of the structure of an end-to-end text-to-speech synthesis device combined with autoregression provided in the second embodiment of the present invention. DETAILED DESCRIPTION
[0045] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.
[0046] refer to Figures 1 to 3 , as shown, the first embodiment of the present invention discloses an end-to-end text-to-speech method combined with autoregression, which can be executed by a text-to-speech device (hereinafter referred to as a speech device), and in particular, by one or more processors in the speech device to implement the following method:
[0047] S101, inputting the current reference audio and the text to be inferred into the semantic language module of the speech model, and inferring to obtain the audio discrete code;
[0048] Specifically, step S101 includes: encoding the current reference audio based on the HuBert module of the speech model, and the encoding result Expressed as ,in Indicates the time length of the current reference audio, 50 is the frame rate of the current reference audio, represents the hidden dimension, Indicates that the value is in the real number domain and the dimension is multidimensional. The specific dimension is ;
[0049] Using one-dimensional convolution and multi-layer RVQ residual modules, the frame rate of the current reference audio is reduced from 50 frames to 25 frames, and then compressed to obtain audio discrete coding.
[0050] In this embodiment, the end-to-end text-to-speech method combined with autoregression proposes an end-to-end text-to-speech model CMM-TTS combined with autoregression. The CMM-TTS model has the ability to clone voices with zero samples, and its architecture is roughly as follows: Figure 2As shown in the figure, the dotted arrows are the direction of inference, the solid arrows are the direction of training, and the arrows ①②③ are required for both training and inference. The CMM-TTS model mainly consists of 6 modules, namely the information encoding module (InfoEncoder), the semantic quantization module, the duration module, the first verification encoding module, the flow module (Flow) and the generator module (Generator). It also includes the text sequence after text-to-audio conversion, the pronunciation time of each phoneme, and the linear spectrum of the audio; CMM refers to Conditional Mel-spectrogram synthesis with a Tacotron model, which is a conditional Mel spectrum synthesis model based on the Tacotron model; it is an end-to-end model for text-to-speech synthesis (TTS), which converts the input text into a Mel spectrum and then generates the final speech through a vocoder.
[0051] In this embodiment, the encoding result It can be understood as a matrix with a shape of (l*50, h); first encode the audio based on HuBert, which generates high-quality speech representation at a frame rate of 20ms. The encoding result can be expressed as ,in Indicates the time length of the current audio. Then, using one-dimensional convolution and multi-layer RVQ residual modules, the frame rate of the audio is reduced from 50 frames to 25 frames, and then compressed into a discrete codebook. This means that when inferring, to generate 3 seconds of audio, SLM theoretically only needs to infer 75 tokens. Compared with pure autoregressive text-to-speech models, such as Vall-e, it is more than 3 times faster!
[0052] S102, inputting the current reference audio and the text to be inferred into the continuous language module of the speech model, and inferring the duration information of the audio;
[0053] Preferably, the duration information is information at the phoneme level, and the duration information is aligned with the text to be inferred at the phoneme level through a continuous projection layer, so that the speech model knows the time when each phoneme should be pronounced.
[0054] In this embodiment, the problem that the autoregressive text-generated speech model is prone to word swallowing is addressed. Figure 3 As shown, the end-to-end text-to-speech method combined with autoregression introduces duration information at the phoneme level, and aligns the duration with the text at the phoneme level through a continuous projection layer (Duration Project), so that the model knows the time when each phoneme should be pronounced, solving the problem of easy-to-swallow words.
[0055] S103, the current reference audio, the text to be inferred, the audio discrete code and the duration information are passed into the information encoding module of the speech model to obtain the mean of the prior distribution and variance ;
[0056] S104, based on the mean and variance , sampling to obtain the hidden variables of the prior distribution ;
[0057] Preferably, the flow module is reversible, and improves the distribution complexity of the Gaussian distribution through multiple affine coupling layers, thereby improving the expressiveness of the speech model.
[0058] S105, the hidden variable and the current reference audio are passed to the generator module to generate audio.
[0059] In this embodiment, the CMM-TTS model proposed by the end-to-end text-to-speech method combined with autoregression integrates duration information and audio semantic information through the information encoder, which takes on a large part of the timbre guidance function, greatly reducing the pressure on the generator to reconstruct the audio, making 5-second audio cloning feasible. In addition, in order to maintain a sufficiently fast generation speed and high enough expression ability, CMM-TTS, like VITS, combines variational inference and normalizing flows to generate audio.
[0060] VAE assumes the distribution to be a standard Gaussian distribution, which limits the expressive power of the model. The flow model, through a series of affine coupling layers, can transform the original simple Gaussian distribution into a more complex distribution, thereby improving the expressive power of the model. The flow model is reversible, and the addition of this module has almost negligible impact on the speed of audio generation. It is worth noting that in order to keep reasoning and training consistent, we introduced two autoregressive models, the Semantic Language Model (SLM) and the Duration Language Model (DLM).
[0061] In short, the core time consumption of the CMM-TTS model lies in the two autoregressive models SLM and DLM introduced during inference. Among them, SLM takes up the largest proportion of time. This is because the phoneme length of the inference text is generally much shorter than the representation length of the audio. Therefore, in order to reduce the time consumption without losing the sound generation effect, two autoregressive models SLM and DLM are introduced.
[0062] Preferably, the speech model is trained using an adversarial network training method, and its training goal is to maximize the variational lower bound ELBO. The formula for maximizing the likelihood is:
[0063]
[0064] in, Represents a given text Hidden variables The prior distribution of is a data sample The likelihood function of is the approximate posterior distribution;
[0065] The training loss is -ELBO, where the training loss is the reconstruction loss and KL divergence The sum of is: .
[0066] Preferably, the single forward and backward training process is specifically as follows:
[0067] Get the linear spectrum of the audio used as training samples , the linear spectrum Through the semantic quantization module, the audio discrete code is obtained , Indicates the length of the current audio. represents hidden dimension;
[0068] Discrete audio encoding , duration information, text phonemes , and the linear spectrum Input to the information encoding module to obtain the mean of the posterior distribution of the current training sample and variance ;
[0069] Linear spectrum Input the posterior encoder to obtain the mean of the posterior distribution of the current training sample ,variance , and hidden variables ;
[0070] The latent variable of the posterior distribution and linear spectrum Perform affine transformation through the flow module to obtain the affine result ;
[0071] The latent variable of the posterior distribution and linear spectrum , through the generator module, sampling generates audio ;
[0072] The audio and linear spectrum Convert to Mel spectrum and , and calculate the reconstruction loss ;
[0073] According to the affine result , the mean of the prior distribution , the mean of the posterior distribution , the variance of the posterior distribution Calculate KL divergence;
[0074] The discriminator generates a loss according to the KL divergence calculation, wherein the discriminator uses a multi-cycle discriminator of HiFi-GAN;
[0075] Computational audio discrete coding MSE loss of decoding and encoding results;
[0076] According to the loss of each module, the parameters of the speech model are adjusted.
[0077] In this embodiment, like VITS, the CMM-TTS model uses adversarial network training to improve the quality of speech synthesis. The discriminator uses the multi-cycle discriminator of HiFi-GAN. The training goal of CMM-TTS is to maximize the variational lower bound ELBO, that is, to maximize the likelihood .in, Represents a given text Hidden variables The prior distribution of is a data sample The likelihood function of is the approximate posterior distribution. So the training loss is -ELBO, which can be regarded as the reconstruction loss and KL divergence The sum of .
[0078] In summary, the end-to-end text-to-speech method combined with autoregression realizes an end-to-end TTS model combined with autoregression, and incorporates external duration information to solve the problem of word swallowing. It solves the current Zero Shot TTS model's slow reasoning and word swallowing problems, as well as the data dependency problem of the current end-to-end TTS model.
[0079] See also Figure 4 The second embodiment of the present invention further provides an end-to-end text-to-speech system combined with autoregression, which includes:
[0080] The semantic language module 201 is used to input the current reference audio and the text to be inferred into the semantic language module of the speech model to infer the audio discrete code;
[0081] A continuous language module 202, used to input the current reference audio and the text to be inferred into the continuous language module of the speech model, and infer the duration information of the audio;
[0082] The information encoding module 203 is used to pass the current reference audio, the text to be inferred, the audio discrete code and the duration information into the information encoding module of the speech model to obtain the mean of the prior distribution. and variance ;
[0083] A priori encoding module 204 is used to and variance , sampling to obtain the hidden variables of the prior distribution ;
[0084] Generator module 205, used to convert the hidden variable and the current reference audio are passed to the generator module to generate audio.
[0085] The above is a preferred embodiment of the present invention. It should be pointed out that a person skilled in the art can make several improvements and modifications without departing from the principle of the present invention. These improvements and modifications are also considered to be within the scope of protection of the present invention.
Claims
1. An end-to-end text-to-speech method combined with autoregression, characterized in that: include: Inputting the current reference audio and the text to be inferred into the semantic language module of the speech model to obtain the audio discrete code by inference, wherein the speech model is a CMM-TTS model; When the audio discrete code is obtained by inference: the HuBert module based on the speech model encodes the current reference audio, and the encoding result Expressed as ,in Indicates the time length of the current reference audio, 50 is the frame rate of the current reference audio, represents the hidden dimension, Indicates that the value is in the real number domain and the dimension is multi-dimensional; Using one-dimensional convolution and multi-layer RVQ residual modules, the frame rate of the current reference audio is reduced from 50 frames to 25 frames, and then compressed to obtain audio discrete coding; Input the current reference audio and the text to be inferred into the continuous language module of the speech model to infer the duration information of the audio, wherein the semantic language module and the continuous language module are both autoregressive models; The current reference audio, the text to be inferred, the audio discrete code and the duration information are passed into the information encoding module of the speech model to obtain the mean of the prior distribution and variance ; Based on the mean and variance , sampling to obtain the hidden variables of the prior distribution ; The hidden variable and the current reference audio are passed to the generator module to generate audio; The duration information is information at the phoneme level, and the duration information is aligned with the text to be inferred at the phoneme level through a continuous projection layer, so that the speech model knows the time when each phoneme should be pronounced; The CMM-TTS model has the ability to clone voice with zero samples. The CMM-TTS model consists of six modules, namely, information encoding module, semantic quantization module, duration module, pre-verification encoding module, stream module and generator module. It also includes the text sequence after text-to-audio conversion, the pronunciation time of each phoneme, and the linear spectrum of the audio. Among them, CMM is a conditional Mel spectrum synthesis model based on the Tacotron model. It is an end-to-end model for text-to-speech synthesis, which converts the input text into Mel spectrum and then generates the final speech through a vocoder.
2. The end-to-end text-to-speech synthesis method combined with autoregression according to claim 1, characterized in that: The speech model is trained using an adversarial network, and its training goal is to maximize the variational lower bound ELBO. The formula for maximizing the likelihood is: in, Represents a given text Hidden variables The prior distribution of It is a data sample The likelihood function of is the approximate posterior distribution; The training loss is -ELBO, where the training loss is the reconstruction loss and KL divergence The sum of is: .
3. The end-to-end text-to-speech synthesis method combined with autoregression according to claim 2, characterized in that: The specific training process of a single forward and backward pass is as follows: Get the linear spectrum of the audio used as training samples , the linear spectrum Through the semantic quantization module, the audio discrete code is obtained , Indicates the length of the current audio. represents hidden dimension; Discrete audio encoding , duration information, text phonemes , and the linear spectrum Input to the information encoding module to obtain the mean of the posterior distribution of the current training sample and variance ; Linear spectrum Input the posterior encoder to obtain the mean of the posterior distribution of the current training sample ,variance , and hidden variables ; The latent variable of the posterior distribution and linear spectrum Perform affine transformation through the flow module to obtain the affine result ; The latent variable of the posterior distribution and linear spectrum , through the generator module, sampling generates audio ; The audio and linear spectrum Convert to Mel spectrum and , and calculate the reconstruction loss ; According to the affine result , the mean of the prior distribution , the mean of the posterior distribution , the variance of the posterior distribution Calculate KL divergence; The discriminator generates a loss according to the KL divergence calculation, wherein the discriminator uses a multi-cycle discriminator of HiFi-GAN; Computational audio discrete coding MSE loss of decoding and encoding results; According to the loss of each module, the parameters of the speech model are adjusted.
4. The end-to-end text-to-speech synthesis method combined with autoregression according to claim 3, characterized in that: The flow module is reversible, and improves the distribution complexity of the Gaussian distribution through multiple affine coupling layers, thereby improving the expressiveness of the speech model.
5. An end-to-end text-to-speech system combined with autoregression, used to implement the end-to-end text-to-speech method combined with autoregression according to claim 1, characterized in that: include: A semantic language module, which is used to input the current reference audio and the text to be inferred into the semantic language module of the speech model, and infer the audio discrete code; A continuous language module is used to input the current reference audio and the text to be inferred into the continuous language module of the speech model to infer the duration information of the audio; An information encoding module is used to pass the current reference audio, the text to be inferred, the audio discrete code and the duration information into the information encoding module of the speech model to obtain the mean of the prior distribution and variance ; A priori encoding module for and variance , sampling to obtain the hidden variables of the prior distribution ; The generator module is used to transform the latent variables and the current reference audio are passed to the generator module to generate audio.
