End-to-end speech synthesis method and device

Abstract

The present invention relates to an end-to-end speech synthesis method, which specifically includes: constructing a hierarchical conditional variational autoencoder model including HAE, HCE, and HAD; training the model with maximizing the lower bound of evidence as the training objective; and synthesizing a speech waveform. It also relates to a device including HAE, HCE, HAD, and a training module; wherein the HAE includes: a five-level speech encoder at the frame level, phoneme level, subword level, word level, and sentence level, and a first affine module; the HCE includes: a linguistic representation extraction module, a three-level text encoder at the subword level, word level, and sentence level; and the HAD includes: a five-level decoder at the sentence level, word level, subword level, phoneme level, and frame level, and a second affine module. The method and device of the present invention improve the overall quality, naturalness, and rhythmic expressiveness of the synthesized speech.

Description

Technical Field

[0001] The present invention relates to the fields of neural networks and speech synthesis, and in particular to an end-to-end speech synthesis method and device. Background Art

[0002] Text-to-speech (TTS) is a technical means of converting text into natural and fluent speech. It is an important component of voice interaction and is widely used in scenarios such as audio novels, virtual anchors, voice broadcasts, and intelligent voice assistants.

[0003] With the development of deep learning, speech synthesis technology has gradually evolved from a two-stage model to a fully end-to-end model.

[0004] The two-stage model primarily consists of an acoustic model and a vocoder, which are connected by manually designed acoustic features. Because the acoustic model and vocoder for two-stage speech synthesis are independently optimized and trained, there are problems with mismatch and error accumulation, which reduces overall quality and naturalness.

[0005] To avoid the problems of two-stage speech synthesis, fully end-to-end models are becoming increasingly popular. These models can directly generate speech waveforms from character or phoneme sequences, offering the following advantages: 1) requiring less prior knowledge of pronunciation; 2) avoiding error propagation in cascaded models; and 3) reducing training, development, and deployment costs. However, training TTS models in an end-to-end manner presents significant challenges, primarily due to the different modalities between text and speech waveforms, as well as the significant length mismatch between character / phoneme sequences and waveform sequences.

[0006] Speech is a hierarchical system consisting of syllables, phonemes, words, phrases, and sentences. Syllables and phonemes are defined by phonetic features, while words, phrases, and sentences are primarily defined by linguistic knowledge. Due to the inherent hierarchical nature of speech, extensive research has focused on leveraging multi-scale information to enrich the expressiveness of speech synthesis. Style vectors at different scales describe variations in speech style, with coarse-grained vectors capturing longer-term changes, while fine-grained vectors provide more detailed information. However, in previous research, style extraction and prediction modules at different scales were relatively independent, potentially leading to mismatches in prosodic information across different scales. Therefore, in the style extraction process, fine-grained style features should be extracted first, followed by the abstraction of coarse-grained information from the fine-grained information. Style prediction, on the other hand, takes the opposite approach: first generating coarse-grained information to establish the overall tone. Then, based on the coarse-grained style, more refined style information is predicted. In this way, styles at different scales can be harmonized and unified.

[0007] In addition to linguistic and phonetic feature scales, frames, as units of speech acoustic features, are also an important scale in the speech generation process. They connect phonetic information to the generation process of speech waveform signals. Due to the significant difference in temporal precision between waveforms and text, current speech synthesis systems generally consist of two components: an acoustic model and a vocoder. The acoustic model generates acoustic features from text, while the vocoder converts these acoustic features into a speech waveform. In recent years, methods such as VITS and NaturalSpeech have been used to construct frame-level latent variables to achieve end-to-end training of acoustic models and vocoders. However, the single-layer variational autoencoder (VAE) used in these models limits their expressiveness.

[0008] VAE is a neural network generation model. Given a dataset X = {x1, x2..., x N In VAEs, a prior distribution p(z) is defined and the joint distribution p(x, z) is modeled using p(z)p(x|z). VAEs consist of three parts: a generator p(x|z), a prior distribution p(z), and a posterior estimator q(z|x). Typically, the prior and posterior distributions in VAEs are assumed to be normally distributed with diagonal covariance, which allows the network to be trained using a technique called Gaussian resampling. VAE training maximizes the evidence lower bound (ELBO) through the resampling technique of backpropagation.

[0009]

[0010] However, the assumption of a normal distribution with diagonal covariance makes it difficult to model a more complex distribution. When the latent variables need to model a more complex distribution, VAE will lead to over-smoothing generation effects. Summary of the Invention

[0011] In view of this, the main purpose of the present invention is to provide an end-to-end speech synthesis method and device, which closely combines the hierarchical attributes of speech with hierarchical VAE by modeling latent variables at five scales: frame level, phoneme level, subword level, word level, and sentence level, thereby improving the overall quality, naturalness, and rhythmic expressiveness of the synthesized speech.

[0012] To achieve the above objectives, the present invention provides an end-to-end speech synthesis method, comprising:

[0013] Construct a hierarchical conditional variational autoencoder model including HAE, HCE, and HAD;

[0014] The model is trained with the goal of maximizing the lower bound of evidence, including:

[0015] HCE extracts phoneme-level and character-level linguistic representations from text samples, and downsamples the extracted linguistic representations to obtain subword-level, word-level, and sentence-level linguistic representations. The phoneme-level, subword-level, and word-level linguistic representations are then fed into HAE and HAD as supervisory information for obtaining their respective next-level representations.

[0016] HAE extracts frame-level representations from the linear spectrum corresponding to the text sample, and then downsamples each level of representation from the frame-level representation to obtain phoneme-level, subword-level, word-level, and sentence-level representations. It uses affine transformation to obtain corresponding latent variables from each level of representation; among them, phoneme duration is used when downsampling to obtain subword-level representations.

[0017] HAD upsamples the feature combination of sentence-level linguistic representation and sentence-level latent variables input by HAE to obtain word-level representation, and then upsamples the feature combination of each level of representation and the corresponding level of latent variables input by HAE from the word level to obtain subword, phoneme and frame-level representation, and uses affine transformation to obtain corresponding level of latent variables from the sentence-level linguistic representation and the upsampled level of representation; reconstructs the speech waveform based on the feature combination of frame-level representation and frame-level latent variables input by HAE; wherein, the phoneme duration is reconstructed when upsampling to obtain phoneme-level representation;

[0018] Synthesized speech waveforms, including:

[0019] HCE extracts phoneme-level linguistic representations and character-level linguistic representations from the text to be synthesized, and downsamples the extracted linguistic representations step by step to obtain subword-level, word-level, and sentence-level linguistic representations. The phoneme-level, subword-level, and word-level linguistic representations are then input into HAD as supervision information for obtaining the next-level representations.

[0020] HAD uses affine transformation to obtain sentence-level latent variables from sentence-level linguistic representations; upsamples the feature combination of sentence-level latent variables and sentence-level linguistic representations to obtain word-level representations, and then upsamples the feature combination of each level of representation and the corresponding level of latent variables from the word level to obtain subword, phoneme and frame-level representations; uses affine transformation to obtain corresponding level of latent variables from each level of representation; synthesizes speech waveforms based on the feature combination of frame-level latent variables and frame-level representations; and reconstructs the phoneme duration when upsampling to obtain phoneme-level representations.

[0021] In one possible implementation, the weights of KL at each level are gradually increased during training, and the weight of sentence-level KL is maintained to be < the weight of word-level KL < the weight of subword-level KL < the weight of phoneme-level KL < the weight of frame-level KL, until the weight of frame-level KL increases to 1.

[0022] On the other hand, the present invention provides an end-to-end speech synthesis device, comprising: an HAE, an HCE, an HAD, and a training module; wherein the HAE comprises: a five-level speech encoder at the frame level, the phoneme level, the subword level, the word level, and the sentence level, and a first affine module; the HCE comprises: a linguistic representation extraction module, a three-level text encoder at the subword level, the word level, and the sentence level; the HAD comprises: a five-level decoder at the sentence level, the word level, the subword level, the phoneme level, and the frame level, and a second affine module;

[0023] A training module is used to set the training goal to maximize the lower bound of evidence and terminate the training when the training goal is reached;

[0024] The linguistic representation extraction module is used to extract phoneme-level linguistic representations and character-level linguistic representations from text samples / text to be synthesized. During model training, the phoneme-level linguistic representations are input to HAE and HAD as supervision information for obtaining subword representations. After training is completed, the phoneme-level linguistic representations are input to HAD as supervision information for obtaining subword representations during speech synthesis.

[0025] Subword-level, word-level, and sentence-level text encoders are used to downsample the extracted linguistic representations step by step to obtain subword-level, word-level, and sentence-level linguistic representations; and during model training, they are used to input the subword-level and word-level linguistic representations into HAE and HAD as supervision information for obtaining their respective next-level representations. After training is completed, they are used to input the subword-level and word-level linguistic representations into HAD as supervision information for obtaining their respective next-level representations when performing speech synthesis;

[0026] A frame-level speech encoder is used to extract frame-level representations from the linear spectra corresponding to text samples;

[0027] Phoneme-level, subword-level, word-level, and sentence-level speech encoders for downsampling each level of representation from the frame-level representation to obtain phoneme-level, subword-level, word-level, and sentence-level representations; wherein the phoneme duration is used when downsampling to obtain the subword-level representation;

[0028] A first affine module is used to obtain corresponding latent variables at each level from the representations output by the speech encoders at each level using affine transformation;

[0029] The sentence-level decoder is used to upsample the sentence-level linguistic representation and the sentence-level latent variable input by the HAE to obtain the word-level representation during model training. When performing speech synthesis after training, it is used to upsample the sentence-level linguistic representation and the sentence-level latent variable output by the second affine module to obtain the word-level representation.

[0030] The word-level, subword-level, and phoneme-level decoders are used to upsample the feature combinations of the corresponding level representations and the corresponding level latent variables of the HAE input from the word level to obtain subword, phoneme, and frame-level representations during model training. When performing speech synthesis after training, they are used to upsample the feature combinations of the corresponding level representations and latent variables from the word level to obtain subword, phoneme, and frame-level representations. The subword-level decoder is also used to reconstruct the phoneme duration.

[0031] The second affine module is used to obtain sentence-level latent variables from the sentence-level linguistic representation using affine transformation; and obtain corresponding latent variables at each level from the representation output by each level of decoder using affine transformation;

[0032] The frame-level decoder is used to reconstruct the speech waveform based on the feature combination of the frame-level representation and the frame-level latent variables of the HAE input during model training; when performing speech synthesis after training is completed, it is used to synthesize the speech waveform based on the feature combination of the frame-level latent variables and the frame-level representation.

[0033] In one possible implementation, the training module is also used to gradually increase the weights of KL at each level during training, and maintain the weight of sentence-level KL < the weight of word-level KL < the weight of subword-level KL < the weight of phoneme-level KL < the weight of frame-level KL, until the weight of frame-level KL increases to 1.

[0034] Based on the above, the present invention provides an end-to-end speech synthesis method and apparatus with the following advantages and features:

[0035] A hierarchical text encoder is used to extract linguistic information at different levels as supervisory information for the encoder and decoder, and hierarchical KL weight annealing is used to avoid hierarchical posteriori failure, further improving the quality and rhythmic expressiveness of the synthesized speech. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 A schematic diagram of the structure of any one of a phoneme-level to sentence-level speech encoder and a subword-level to sentence-level text encoder according to an embodiment of the present invention;

[0037] Figure 2 A schematic diagram of the structure of any one of the phoneme-level to sentence-level speech decoders according to an embodiment of the present invention;

[0038] Figure 3 A schematic diagram of the model training process according to an embodiment of the present invention;

[0039] Figure 4 It is the KL weight change curve;

[0040] Figure 5 The figure is a flow chart of speech synthesis according to an embodiment of the present invention. DETAILED DESCRIPTION

[0041] Hierarchical Variational Autoencoder (HVAE) improves the complexity of prior distribution and posterior estimation by modeling hierarchical dependent latent variables. An HVAE can be seen as a series of VAEs stacked together. In addition to the observed variable x, it contains hierarchical latent variables {z1, z2, ..., z L}, where L is the number of levels. According to the chain rule, the joint distribution p(x, z1, ..., z L ) can be decomposed into:

[0042]

[0043] Then the prior and estimated posterior distributions are expressed as and The conditional distribution of each level of prior and estimated posterior adopts a standard Gaussian distribution with independent components. HVAE extracts the posterior distribution from the data based on a bottom-up path, and generates the prior distribution and reconstructs the sample along a top-down path. This architecture helps the model effectively learn the hierarchical structure between latent variables. The formula of HAVE's ELBO is:

[0044]

[0045] in, and They are Approximate posterior and predictive prior for the layer.

[0046] Specifically, an embodiment of the present invention provides an end-to-end speech synthesis device, including a hierarchical audio encoder (HAE), a hierarchical audio decoder (HAD), a hierarchical context encoder (HCE), and a training module; wherein the HAE includes five levels of speech encoders: frame level, phoneme level, subword level, word level, and sentence level, and a first affine module; the HCE includes a linguistic representation extraction module, three levels of text encoders: subword level, word level, and sentence level; the HAD includes five levels of decoders: sentence level, word level, subword level, phoneme level, and frame level, and a second affine module.

[0047] In one possible implementation, the frame-level speech encoder is implemented using a non-causal WaveNet residual module, which consists of dilated convolutional layers with gated activation units and skip connections.

[0048] The structures of phoneme-level, subword-level, word-level and sentence-level speech encoders are the same Figure 1As shown, it includes a bidirectional gated recurrent unit (Bidirectional-GRU) and an attentive pooling module (AP). The Bidirectional-GRU extracts temporal contextual information from the fine-grained representation, while the attention module combines alignment information to obtain a coarse-grained representation. The frame-phoneme, phoneme-subword, subword-word, and word-sentence alignment information are obtained using forced alignment, a pronunciation dictionary, a word segmentation tool, and full sentence information, respectively.

[0049] Subword-level, word-level, and sentence-level text encoder structures also adopt Figure 1 The structure shown in the figure downsamples the phoneme-level linguistic representation to subword-level, word-level, and sentence-level linguistic representations in sequence.

[0050] In another possible implementation, the frame-level speech decoder adopts the Univnet vocoder architecture, which is a MelGAN generator enhanced with location-variable convolution (LVC).

[0051] The structures of phoneme-level, subword-level, word-level and sentence-level speech decoders are the same Figure 2 As shown in the figure, the fully connected layer attention mechanism of Feed-Forward Transformer blocks is used as the basic structure. Rotary Position Embedding (RoPE) is added to each self-attention layer in each FFT block to enable the model to process sequences of arbitrary length. The Length Regulator (Length Regulator) uses alignment information to upsample the coarse-grained latent variable (Coarser Latent Variable) and coarser representation features to a lower scale by replicating and expanding. The expanded features are combined with linguistic input and fed into the FFT module to obtain the final Finer Representation.

[0052] like Figure 3 The model training process of an end-to-end speech synthesis device according to an embodiment of the present invention is as follows:

[0053] The linguistic representation extraction module extracts phoneme-level linguistic representations (Phoneme Embedding) and character-level linguistic representations (Character Embedding) from text samples, and inputs the Phoneme Embedding into the subword-level speech encoder and subword-level decoder to obtain supervision information for the subword-level representation; the subword-level text encoder downsamples the Phoneme Embedding to obtain a subword-level linguistic representation, and inputs the subword-level linguistic representation into the word-level speech encoder and word-level decoder to obtain supervision information for the phoneme-word representation; the word-level text encoder downsamples the feature combination of the subword-level linguistic representation and the Character Embedding to obtain a word-level linguistic representation, and inputs the word-level linguistic representation into the sentence-level speech encoder and sentence decoder to obtain supervision information for the sentence representation; the sentence-level text encoder downsamples the word-level linguistic representation to obtain a sentence-level linguistic representation, and inputs the sentence-level linguistic representation into the second affine module.

[0054] The frame-level speech encoder extracts frame-level representation from the linear spectrum corresponding to the text sample; the phoneme-level, subword-level, word-level, and sentence-level speech encoders downsample each level of representation from the frame-level representation to obtain phoneme-level, subword-level, word-level, and sentence-level representations; among them, the phoneme duration is used when downsampling to obtain the subword-level representation.

[0055] The first affine module uses affine transformation to obtain the corresponding frame-level latent variable Z5, phoneme-level latent variable Z4, subword-level latent variable Z3, word-level latent variable Z2 and sentence-level latent variable Z1 from the representations output by the frame-level, phoneme-level, subword-level, word-level and sentence-level speech encoders respectively.

[0056] The sentence-level decoder upsamples the feature combination of the sentence-level linguistic representation and the sentence-level latent variables of the HAE input to obtain the word-level representation; the word-level, subword-level, and phoneme-level decoders upsample the feature combination of their respective corresponding-level representations and the corresponding-level latent variables of the HAE input, starting from the word level, to obtain subword, phoneme, and frame-level representations; among them, the subword-level decoder is also used to reconstruct the phoneme duration.

[0057] The second affine module uses affine transformation to obtain sentence-level latent variables Z1 from sentence-level linguistic representations; and uses affine transformation to obtain corresponding latent variables Z2 to Z5 at each level from the representations output by each level of decoders.

[0058] The frame-level decoder reconstructs the speech waveform based on the feature combination of the frame-level representation and the frame-level latent variables input by the HAE.

[0059] The training module sets the training goal to maximize the lower bound of evidence. When the training goal is reached, the training is terminated. The calculation formula of the lower bound of evidence is as follows:

[0060]

[0061] in, and Respectively indicate the Hidden variables and linguistic features of the hierarchy; It is The approximate posterior of the layer, is the phoneme duration; No. The prediction prior of the layer; the first term is the reconstruction loss; the second and third terms represent the KL divergence; From 1 to 5, they correspond to sentence level to frame level.

[0062] The reconstruction loss includes two parts: waveform reconstruction and duration reconstruction, as follows:

[0063] Waveform reconstruction loss, here we use multi-precision STFT loss, which is derived from the spectral convergence loss L sc and log STFT amplitude loss L mag Composition, defined as follows:

[0064]

[0065]

[0066] Among them, ||·|| F and ||·||1 represent the Forbenius and L1 norms, respectively. S represents the number of elements in the spectrum. M sets of STFT parameters are selected. Here, only speech segments are generated to calculate the waveform reconstruction loss.

[0067] Duration reconstruction, here we use the phoneme duration obtained with the help of external alignment tools as the target, and then use the log-scale L2 loss to optimize the duration reconstruction, as follows:

[0068]

[0069] here, The prediction formula of the KL divergence of the level is:

[0070]

[0071] in and Belong to The mean and variance of the estimated posterior and predictive prior of the layer latent variables.

[0072] Another possible implementation is Figure 4 As shown, the training module is also used to gradually increase the weights of KL at each level during training, and maintain the weight of sentence-level KL < the weight of word-level KL < the weight of subword-level KL < the weight of phoneme-level KL < the weight of frame-level KL, until the weight of frame-level KL increases to 1.

[0073] In this implementation, after combining the HVAE loss and the GAN loss, the total loss during the training phase is:

[0074]

[0075] in, express Penalty coefficient for the level KL term.

[0076] To further improve speech waveform reconstruction, adversarial training is introduced in both the time and frequency domains during model training. The Multi-Precision Spectral Discriminator (MRSD) is used to discriminate speech at different temporal and spectral precisions, while the Multi-Periodic Waveform Discriminator (MPWD) is used to enhance detailed adversarial modeling in the time domain.

[0077] like Figure 5 The process of speech synthesis of an end-to-end speech synthesis device model according to an embodiment of the present invention is as follows:

[0078] The linguistic representation extraction module extracts Phoneme Embedding and Character Embedding from the text to be synthesized, and inputs Phoneme Embedding into the subword-level decoder to obtain supervision information for subword-level representation; the subword-level text encoder downsamples Phoneme Embedding to obtain subword-level linguistic representation, and inputs the subword-level linguistic representation into the word-level decoder to obtain supervision information for phonetic-word representation; the word-level text encoder downsamples the feature combination of subword-level linguistic representation and Character Embedding to obtain word-level linguistic representation, and inputs the word-level linguistic representation into the sentence decoder to obtain supervision information for sentence representation; the sentence-level text encoder downsamples the word-level linguistic representation to obtain sentence-level linguistic representation, and inputs the sentence-level linguistic representation into the second affine module.

[0079] The sentence-level decoder upsamples the feature combination of sentence-level linguistic representation and sentence-level latent variables to obtain word-level representation; the word-level, subword-level, and phoneme-level decoders upsample the feature combination of their respective corresponding-level representations and corresponding-level latent variables, starting from the word level, to obtain subword, phoneme, and frame-level representations; among them, the subword-level decoder is also used to reconstruct the phoneme duration.

[0080] The second affine module uses affine transformation to obtain sentence-level latent variables Z1 from sentence-level linguistic representations; and uses affine transformation to obtain corresponding latent variables Z2 to Z5 at each level from the representations output by each level of decoders.

[0081] The frame-level decoder synthesizes the speech waveform based on the feature combination of frame-level representation and frame-level latent variables.

[0082] This invention has been validated using Chinese data. This data consists of 15 hours of Mandarin Chinese speech from a male voice actor, including 5 hours of emotional data. The mean opinion score (MOS) was used to evaluate the naturalness of the speech synthesis system. The validation results are shown in the table below.

[0083] Table 1: MOS score results

[0084]

[0085] In Table 1, GroundTruth represents real speech. FastSpeech2+HiFiGAN, MultiGST+HiFiGAN, PortaSpeech+HiFiGAN, and VITS are currently mainstream system frameworks. HierTTS is our proposed highly expressive end-to-end speech synthesis framework using a multi-scale hierarchical variational autoencoder. During the testing phase, 100 test samples were generated for each system, and each test sample was evaluated by 20 testers. Table 1 shows that PortaSpeech has the lowest MOS. This is because our dataset is a mixture of emotional speech and reading, and PortaSpeech may have difficulty fitting training data with such rich prosodic variation. FastSpeech2 also has a low MOS, and its prosody is relatively flat, which may be due to FastSpeech2's excessive smoothing. MultiGST improves the naturalness of FastSpeech2, indicating that leveraging semantic information helps learn variable prosody. However, because MultGST only models prosody at the subword level and sentence level, its rhythmic representation deviates significantly from the recordings. VITS also generates speech with rhythmic and accent errors. As a result, our model achieves higher naturalness, with pauses and accents that are very natural and close to the rhythmic representation of the recordings.

[0086] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention.

Claims

1. An end-to-end speech synthesis method, characterized in that: include: Construct a hierarchical conditional variational autoencoder model including HAE, HCE, and HAD; The model is trained with the goal of maximizing the lower bound of evidence, including: HCE extracts phoneme-level and character-level linguistic representations from text samples, and downsamples the extracted linguistic representations to obtain subword-level, word-level, and sentence-level linguistic representations. The phoneme-level, subword-level, and word-level linguistic representations are then fed into HAE and HAD as supervisory information for obtaining their respective next-level representations. HAE extracts frame-level representations from the linear spectrum corresponding to the text sample, and then downsamples each level of representation from the frame-level representation to obtain phoneme-level, subword-level, word-level, and sentence-level representations. It uses affine transformation to obtain corresponding latent variables from each level of representation; among them, phoneme duration is used when downsampling to obtain subword-level representations. HAD upsamples the feature combination of sentence-level linguistic representation and sentence-level latent variables input by HAE to obtain word-level representation, and then upsamples the feature combination of each level of representation and the corresponding level of latent variables input by HAE from the word level to obtain subword, phoneme and frame-level representation, and uses affine transformation to obtain corresponding level of latent variables from the sentence-level linguistic representation and the upsampled level of representation; reconstructs the speech waveform based on the feature combination of frame-level representation and frame-level latent variables input by HAE; wherein, when upsampling to obtain phoneme-level representation, the phoneme duration is reconstructed; Synthesized speech waveforms, including: HCE extracts phoneme-level linguistic representations and character-level linguistic representations from the text to be synthesized, and downsamples the extracted linguistic representations step by step to obtain subword-level, word-level, and sentence-level linguistic representations. The phoneme-level, subword-level, and word-level linguistic representations are then input into HAD as supervision information for obtaining the next-level representations. HAD uses affine transformation to obtain sentence-level latent variables from sentence-level linguistic representations; upsamples the feature combination of sentence-level latent variables and sentence-level linguistic representations to obtain word-level representations, and then upsamples the feature combination of each level of representation and the corresponding level of latent variables from the word level to obtain subword, phoneme and frame-level representations; uses affine transformation to obtain corresponding level of latent variables from each level of representation; synthesizes speech waveforms based on the feature combination of frame-level latent variables and frame-level representations; and reconstructs the phoneme duration when upsampling to obtain phoneme-level representations.

2. The method according to claim 1, characterized in that During training, the weights of KL at each level are gradually increased, and the weight of sentence-level KL is maintained to be less than the weight of word-level KL, less than the weight of subword-level KL, less than the weight of phoneme-level KL, and less than the weight of frame-level KL, until the weight of frame-level KL increases to 1.

3. An end-to-end speech synthesis device, characterized in that: It includes HAE, HCE, HAD, and training modules; HAE includes five-level speech encoders at the frame, phoneme, subword, word, and sentence levels, and a first affine module; HCE includes a linguistic representation extraction module, three-level text encoders at the subword, word, and sentence levels; HAD includes five-level decoders at the sentence, word, subword, phoneme, and frame levels, and a second affine module; A training module is used to set the training goal to maximize the lower bound of evidence and terminate the training when the training goal is reached; The linguistic representation extraction module is used to extract phoneme-level linguistic representations and character-level linguistic representations from text samples / text to be synthesized. During model training, the phoneme-level linguistic representations are input to HAE and HAD as supervision information for obtaining subword representations. After training is completed, the phoneme-level linguistic representations are input to HAD as supervision information for obtaining subword representations during speech synthesis. Subword-level, word-level, and sentence-level text encoders are used to downsample the extracted linguistic representations step by step to obtain subword-level, word-level, and sentence-level linguistic representations; and during model training, they are used to input the subword-level and word-level linguistic representations into HAE and HAD as supervision information for obtaining their respective next-level representations. After training is completed, they are used to input the subword-level and word-level linguistic representations into HAD as supervision information for obtaining their respective next-level representations when performing speech synthesis; A frame-level speech encoder is used to extract frame-level representations from the linear spectra corresponding to text samples; Phoneme-level, subword-level, word-level, and sentence-level speech encoders for downsampling each level of representation from the frame-level representation to obtain phoneme-level, subword-level, word-level, and sentence-level representations; wherein the phoneme duration is used when downsampling to obtain the subword-level representation; A first affine module is used to obtain corresponding latent variables at each level from the representations output by the speech encoders at each level using affine transformation; The sentence-level decoder is used to upsample the sentence-level linguistic representation and the sentence-level latent variable input by the HAE to obtain the word-level representation during model training. When performing speech synthesis after training, it is used to upsample the sentence-level linguistic representation and the sentence-level latent variable output by the second affine module to obtain the word-level representation. The word-level, subword-level, and phoneme-level decoders are used to upsample the feature combinations of the corresponding level representations and the corresponding level latent variables of the HAE input from the word level to obtain subword, phoneme, and frame-level representations during model training. When performing speech synthesis after training, they are used to upsample the feature combinations of the corresponding level representations and latent variables from the word level to obtain subword, phoneme, and frame-level representations. The subword-level decoder is also used to reconstruct the phoneme duration. The second affine module is used to obtain sentence-level latent variables from the sentence-level linguistic representation using affine transformation; and obtain corresponding latent variables at each level from the representation output by each level of decoder using affine transformation; The frame-level decoder is used to reconstruct the speech waveform based on the feature combination of the frame-level representation and the frame-level latent variables of the HAE input during model training; when performing speech synthesis after training is completed, it is used to synthesize the speech waveform based on the feature combination of the frame-level latent variables and the frame-level representation.

4. The device according to claim 3, characterized in that The training module is also used to gradually increase the weights of KL at each level during training, and maintain the weight of sentence-level KL < weight of word-level KL < weight of subword-level KL < weight of phoneme-level KL < weight of frame-level KL, until the weight of frame-level KL increases to 1.

Images