Improved method for a speech synthesis system, electronic device and storage medium

Abstract

The embodiment of the application discloses an improved method of a speech synthesis system, an electronic device and a storage medium, wherein the method is used for a VALL-E model, and the method comprises the following steps: on the basis of the VALL-E model, a plurality of prediction heads are used to predict more tokens in combination with an advanced speculation decoding strategy and a tolerant mechanism, wherein the advanced speculation decoding strategy comprises an effective lossless speculation decoding method based on an autoregressive Transformer model; and a verification mechanism is used to ensure that reasoning sampling is consistent with the VALL-E model. The embodiment of the application generally retains the original VALL-E model, uses more prediction heads to predict more tokens in combination with speculation sampling and the tolerant mechanism, and then uses the verification mechanism to ensure that the reasoning sampling is consistent with the original model. In this way, the performance of the original model can be ensured, more distant tokens can be predicted through multi-head prediction, and the stability and generation effect of the model are optimized. The "tolerant" mechanism makes the results of more prediction heads acceptable, significantly enhances the acceleration effect, and improves the speech generation performance.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of speech synthesis, and particularly relates to an improved method of a speech synthesis system, an electronic device and a storage medium. BACKGROUND

[0002] In the related art, there is no speculative decoding strategy applied in the field of speech generation. More relevantly, the inference acceleration methods of the following autoregressive speech generation model are as follows: (1) using low-bit-rate acoustic coding to shorten the discrete speech sequence and accelerate inference; (2) using acoustic BPE (Byte-Pair Encoding) to shorten the discrete speech sequence and accelerate inference; (3) directly using a merging discrete speech coding method in VALL-E 2 (an optimization scheme for improving the sampling strategy and reducing the training discrete speech coding bit rate by using merging coding training based on VALL-E, where VALL-E is a speech generation architecture combining autoregressive and non-autoregressive discrete speech language models) to accelerate inference. Among them, the low-bit-rate acoustic coding is a new technology that compresses speech into a discrete representation with a lower bit rate, so that the same length of speech can be represented by a shorter discrete sequence, thereby reducing the inference steps during autoregressive inference and achieving acceleration effect; in acoustic BPE and VALL-E 2, the original discrete coding is first merged into a new coding form, the new coding is inferred in an autoregressive manner, and then the original coding is restored through a simple decoding step, which also reduces the autoregressive inference steps.

[0003] The inventors found that these technologies change the training strategy and have a great impact on the model itself, which may affect the quality of speech synthesis and increase instability. SUMMARY

[0004] The embodiments of the present application provide an improved method of a speech synthesis system, an electronic device and a storage medium, which are used to at least solve one of the above technical problems.

[0005] In a first aspect, the embodiments of the present application provide an improved method of a speech synthesis system, which is used for a VALL-E model and includes: on the basis of the VALL-E model, combining an advanced speculative decoding strategy and a tolerant mechanism, and using multiple prediction heads to predict more tokens, wherein the advanced speculative decoding strategy includes an effective lossless speculative decoding method based on an autoregressive Transformer model; and ensuring that the inference sampling is consistent with the VALL-E model through a verification mechanism.

[0006] In a second aspect, an embodiment of the present application further provides a computer program product, which comprises a computer program stored on a non-volatile computer readable storage medium, the computer program comprising program instructions which, when executed by a computer, cause the computer to perform the steps of the training method of the improved system of the speech synthesis system of any embodiment of the present application.

[0007] In a third aspect, an embodiment of the present application further provides an electronic device, comprising at least one processor, and a memory connected to the at least one processor in communication, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the steps of the method of the first aspect.

[0008] In a fourth aspect, an embodiment of the present application further provides a storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the steps of the method of the first aspect.

[0009] The embodiments of the present application generally retain the original VALL-E model in its entirety, combine the "MEDUSA" speculative sampling and the lenient mechanism, use more prediction heads to predict more tokens, and then guarantee that the inference sampling is consistent with the original model through the verification mechanism. In this way, the performance of the original model can be guaranteed, and a kind of beam search effect can be realized by predicting more distant tokens through multiple heads, thereby optimizing the stability and generation effect of the model. The "lenient" mechanism makes the results of more prediction heads acceptable, significantly enhances the acceleration effect, and improves the speech generation performance. BRIEF DESCRIPTION OF DRAWINGS

[0010] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced below. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0011] Figure 1 A flowchart of a training method of an improved system of a speech synthesis system according to an embodiment of the present application is provided.

[0012] Figure 2 A VADUSA decoding with tolerance τ = 2 according to an embodiment of the present application is provided.

[0013] Figure 3 The performance of AR TTS inference according to an embodiment of the present application is provided.

[0014] Figure 4Performance of different VAD USA configurations under the same base model trained based on HuBERT+k-means2048 labeling for an embodiment of the present application

[0015] Figure 5 Accelerated performance and average number of accepted labels when the number set increases in the tolerance strategy for an embodiment of the present application

[0016] Figure 6 Inference results of only the base model and the strategy with the best performance for an embodiment of the present application in different label types

[0017] Figure 7 Effectiveness of the embodiment method of the present application in amplifying the data set for an embodiment of the present application

[0018] Figure 8 Structural schematic diagram of an electronic device for an embodiment of the present application. DETAILED DESCRIPTION

[0019] To make the objectives, technical solutions, and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described below in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0020] The inventors found that the defects of the related art are mainly caused by the following reasons: (1) lower bit rate compression will lose more acoustic information; (2) training uses merged tokens, which increases the risk of single-step token prediction errors during inference, reducing synthesis performance and stability.

[0021] The present solution is the first to apply speculative sampling strategies to autoregressive speech generation systems. Speculative sampling strategies are commonly used in natural language processing tasks, but how to adapt them to the generation process of speech discrete representations requires further algorithm design and experimental exploration. In addition, our method proposes a "tolerant" strategy for speech sequences, making the acceleration effect of the speculative sampling method more significant.

[0022] The embodiment of the present application proposes a speculative sampling method named "VADUSA". The original VALL-E model is generally retained in its entirety. The "MEDUSA" speculative sampling and the tolerance mechanism are combined to predict more tokens using more prediction heads. Then, the verification mechanism is used to ensure that the inference sampling is consistent with the original model. In this way, the performance of the original model can be guaranteed, and a kind of beam search effect can be achieved by predicting more tokens using multiple heads, thereby optimizing the stability and generation effect of the model. The "tolerance" mechanism is proposed, which allows more prediction head results to be accepted, significantly enhances the acceleration effect, and improves the speech generation performance.

[0023] In the embodiment of the present application, the VADUSA decoding with a tolerance of 2: we use the absolute position ID of each layer from left to right to represent the top-3 tree node sampling result. The sparse tree is a subset of the complete top-3 tree. We gradually splice the results by layer and input them into the auto-regressive (AR, Auto-Regressive) decoder for decoding operation. The constructed attention mask ensures that each node can only access the information of its ancestor nodes. The tolerance mechanism is described in detail in the "2-tolerancePredictions" module in Figure 2 , which means that when the tolerance is 2, the AR head will perform two polynomial samplings in the core sampling process. As shown in Figure 2 , after comparing the AR head sampling results with the VADUSA head results in the comparison verification stage, the leftmost branch is preferentially selected in the acceptable branch of the sparse tree.

[0024] In the process of implementing the present application, the inventors have also proposed some alternative versions, which may be experiments under different parameter settings or fine-tuning in algorithm implementation. The following are several possible beta versions: when constructing the sparse tree, the expected value can be calculated not only by the node acceptance rate, but also by multiplying the depth of the node with the expected value to construct a sparse tree that values the length of the accepted nodes.

[0025] In addition, before the final product is formed, the inventors have also used the following beta version: constructing a sparse tree that values the length of the accepted nodes: when the real acceptance rate decreases, the decoding process either decodes many tokens at a time or decodes very few or even one token. It can decode faster in some cases, but the fluctuation is large.

[0026] Please refer to Figure 1 , which shows a flowchart of an embodiment of a training method of an improved system of a speech synthesis system of the present application. The training method is used for the VALL-E model of the foregoing embodiments.

[0027] As shown in Figure 1As shown, in step 101, on the basis of the VALL-E model, an advanced speculative decoding strategy and a tolerance mechanism are combined, and multiple prediction heads are used to predict more tokens, wherein the advanced speculative decoding strategy includes an effective lossless speculative decoding method based on an autoregressive Transformer model.

[0028] In step 102, the inference sampling is ensured to be consistent with the VALL-E model through a verification mechanism.

[0029] In some optional embodiments, the advanced speculative decoding strategy includes: using the absolute position ID of the top-3 tree node arranged from left to right in each layer to represent the sampling result; for a subset of the top-3 tree, a sparse tree, the results are spliced step by step by layer, and the spliced results are input into an autoregressive decoder for a decoding operation, wherein an attention mask is constructed to ensure that each node can only access the information of its ancestor nodes.

[0030] The embodiments of the present application generally retain the original VALL-E model, combine the MEDUSA speculative sampling and the tolerance mechanism, use more prediction heads to predict more tokens, and then ensure that the inference sampling is consistent with the original model through a verification mechanism. In this way, the performance of the original model can be guaranteed, and a kind of beam search effect can be realized by predicting more tokens through multiple heads, thereby optimizing the stability and generation effect of the model. The proposal of the "tolerance" mechanism makes the results of more prediction heads acceptable, significantly enhances the acceleration effect, and improves the speech generation performance.

[0031] In some optional embodiments, the advanced speculative decoding strategy includes:

[0032] Under tree attention, a draft head is used to predict future tokens, and the VALL-E model is expected to accept a candidate path with a higher acceptance probability.

[0033] In some optional embodiments, the tolerance mechanism includes: when the tolerance degree is 2, the autoregressive head performs two polynomial samplings in the core sampling process.

[0034] In some optional embodiments, the verification mechanism includes:

[0035] After comparing the autoregressive head sampling result with the VADUSA head result in the contrast verification stage, the leftmost branch is preferentially selected in the acceptable branch of the sparse tree.

[0036] It should be noted that the above method steps are not used to limit the execution order of each step, and in fact, some steps can be executed simultaneously or in the opposite order defined by the steps, which is not limited by the present application.

[0037] The scheme of the present application is described below by means of a specific embodiment, so that those skilled in the art can better understand the scheme of the present application.

[0038] Autoregressive architectures, such as GPT, are widely used in modern Text-to-Speech (TTS) systems. However, due to the challenge of next-token prediction for long sequence speech tokens, it results in high inference time. In the embodiments of the present application, we propose VADUSA, the first pioneering method to accelerate autoregressive TTS through speculative decoding. Our research results show that VADUSA not only significantly improves the inference speed, but also improves the performance by introducing a draft head to autoregressively predict future speech content. In addition, the tolerance mechanism is added during the sampling process to speed up the inference, while ensuring that the speech quality is not affected. Our method has shown strong generalization ability on large-scale datasets and different types of speech coding.

[0039] Large language models (LLMs) using autoregressive (AR) architectures have achieved great success in recent years. They generate speech Byte Pair Encoding (BPE) tokens using a next-token prediction strategy. This strategy samples the next token from a multi-modal distribution generated based on the history tokens, which is simple but effective in establishing coherence in a long context. With the invention of discrete neural audio tokens, speech passages can also be encoded as discrete token sequences. Subsequently, the paradigm of LLMs was introduced into the field of speech modeling and synthesis. Studies such as SPEAR-TTS, VALL-E, BASE-TTS, etc. use large-scale training data to realize a pure decoder AR architecture for generating speech codecs, which can generate natural-sounding speech.

[0040] However, there are significant differences between speech data and text data: for the same sentence, the speech token sequence is often much longer than the text token sequence. For example, a 10-second speech requires 500 HuBERT tokens (50Hz) to represent, while the speech transcription of the same speech only requires about 20 to 40 text BPE tokens. This is reasonable because speech has a higher information density, and speech can capture finer acoustic details over time, resulting in longer token sequences. In addition, the trivial AR architecture only predicts one token at each inference step, resulting in particularly long speech generation time. This brings challenges between the low-latency spoken synthesis requirement and the long speech sequence modeling.

[0041] To alleviate this issue, one approach in the past is to reduce the length of input / output sequences through distillation or better information compression, i.e., to establish low bit-rate phonetic labels. Many studies on neural codecs have attempted to reduce the bit-rate through reasonably configured vector quantization (VQ) modules and carefully designed training procedures. Another promising study is aBPE, which applies BPE to discrete phonetic labels.

[0042] This method applies BPE to discrete phonetic labels, achieving lossless compression, and shows great potential in accelerating speech synthesis. However, these low bit-rate labels either affect the accuracy of speech reconstruction or cause frame shift variation, thus affecting the naturalness and quality of speech synthesis.

[0043] Another approach is to predict more phonetic labels in one decoding step, which is the focus of this application. This idea has been developed since the RNN era, starting with Subscale WavRNN. It folds the wav label sequence in a subscale manner, thus achieving parallel prediction of multiple labels. Recently, VALL-E 2 uses a blocking manner to generate a large block of future labels (usually 2 labels) in one AR iteration. However, this approach usually leads to quality degradation because the historical condition is insufficient when predicting at least half of a sequence. In contrast, speculative decoding, widely used in LLM inference, can provide acceleration without sacrificing performance by simply adding an extra sketch model.

[0044] Given the potential of speculative decoding, we first apply the advanced speculative decoding strategy MEDUSA to the AR TTS model VALL-E and conduct comprehensive experiments on various discrete labels. In the process of trying, we found great challenges. The inherent complexity and variability of speech synthesis often lead to undesirable prediction accuracy and acceptance rate, making it difficult to achieve the expected acceleration and quality. This prompted us to innovate beyond adaptivity, and ultimately developed the VADUSA method. By integrating a tolerance mechanism, VADUSA not only speeds up the decoding process, but also improves the robustness and quality of overall synthesis by effectively managing the inherent uncertainty of speculative decoding. It turns out that this new method is crucial in bridging the gap between fast decoding and maintaining high synthesis fidelity.

[0045] VADUSA: Fast and High-Quality AR TTS

[0046] In this section, we introduce a new AR TTS decoding method, VADUSA, which can simultaneously achieve fast decoding and high-quality speech synthesis. In this section, we first introduce MEDUSA, an efficient lossless speculative decoding method based on the AR Transformer model. Then, we delve into the unsatisfactory combination of VALL-E and MEDUSA and analyze its limitations. After that, we propose VADUSA, a non-trivial integration of MEDUSA and VALL-E, which achieves decoding acceleration and improves synthesis quality. Finally, we briefly introduce the design of the sparse candidate tree for TTS.

[0047] A. Speculative Decoding and MEDUSA

[0048] Speculative decoding is an efficient and lossless technique for accelerating AR decoding. The key idea is to use a small and fast draft model to generate predictions, which are then verified in parallel by the target model. To avoid the expensive process of training a separate draft model, MEDUSA introduces several additional draft heads based on the original model. Specifically, in the first pass, assuming that the original head predicts the first next token t 1,1 , the i-th draft head is responsible for predicting the (i+1)-th next token t 1,i+1 (i ≥ 1), so that multiple tokens can be generated in parallel. The first next token and the draft tokens are concatenated in order and input into the same model for the next forward pass. In the second pass, the model will almost simultaneously verify the draft tokens. For example, if t 1,2 matches the "correct token" sampled from the distribution generated by the original head from t 1,1 , it will be accepted. Similarly, if t 1,2 has been accepted and matches the first next token of t 1,2 , then t 1,3 will be accepted. Assuming that t 1,3 is accepted, t 1,4 is rejected, then the original head and the draft heads of t 1,3 will predict t 2,1 , t 2,2 ,..., providing information for the third pass, and the second-pass draft acceptance length is 2.

[0049] To make better “guesses”, MEDUSA also considers multiple candidate tokens for each head and builds a draft tree, where the root node represents the first next token, each node represents a candidate token, and each root path represents a candidate continuation. For each draft head, the top-k tokens with the largest log-probabilities are selected as candidate tokens. Each non-leaf node has k children. The (i+1)-th layer has k i-1 nodes, filled by the top-k tokens produced by the i-th draft head. All root-to-leaf paths form all possible combinations of candidate tokens. With a carefully constructed tree mask, the tokens of the entire tree can be validated in one forward pass (named tree attention). This approach greatly improves the draft’s receptive length, and the additional computational cost is reasonable by promoting sparse trees, which will be introduced in the following content. The simplicity of MEDUSA draft heads and their compatibility with any Transformer-based AR model make MEDUSA highly adaptable and efficient.

[0050] B. Vanilla Combination and Re-thinking

[0051] In the vanilla version of VADUSA, we simply use the VALL-E TTS model as the base model for speech synthesis and integrate multiple MEDUSA heads on top of it to speed up the generation. However, in the initial experiments, we found that its receptive length could not compare with the results of MEDUSA on speech LLMs. This is because the speech tokens generated by systems such as EnCodec, HuBERT, and wav2vec 2.0 are obtained through vector quantization or ID clustering, rather than having clear semantic differences like text tokens. Under the AR language modeling paradigm, multiple speech tokens often have very similar statistical characteristics. This means that the multinomial distributions predicted by models such as VALL-E are often more “average”, and selecting the top-k tokens with the highest probability using draft heads can only cover a very small range of probabilities. Given that the original prediction head is only sampled once as the “correct token” during validation, it is challenging for the draft heads in VALL-E to make accurate predictions.

[0052] In addition, from the perspective of improving sample quality, tree attention can be considered as an efficient version of beam search. In beam search, we can choose the combination with the highest probability by predicting the next few steps of tokens, thus improving the sample quality. Similarly, under tree attention, we use the draft head to predict future tokens and hope that the model will accept a candidate path with a higher acceptance probability.

[0053] Figure 2 The VADUSA decoding with tolerance τ = 2 is shown: we use the absolute position ID of the nodes in the top-3 tree (ordered from left to right at each level) to represent the sampling results. The sparse tree is a subset of the complete top-3 tree, and we concatenate the results layer by layer in order to input them into the AR decoder for single-step decoding. We construct an attention mask to ensure that each node can only access the information of its ancestors. The tolerance mechanism, detailed in the "2-tolerance predictions" module in the figure, refers to the fact that when τ = 2, the AR head performs two multinomial samplings during the core sampling process. As shown in the figure, for branches in the sparse tree that are considered acceptable results, the leftmost branch is given priority.

[0054] In the figure, the English-Chinese translation is as follows: Auto-Regressive Decoder: self-recurrent decoder; AR head: original model prediction head; Head123: VADUSA prediction head 123; Attention Mask: attention mask; 2-tolerance Predictions: 2-tolerance predictions; verify: verify. In an embodiment, in combination with the figure, the specific decoding process is as follows: first, obtain the original model decoding result through the self-recurrent decoder, and the topk sampling result of the draft head, then construct the left-biased sparse tree and the attention mask as shown in the figure, combine the results in the sparse tree and input them into the self-recurrent decoder for decoding again, and verify the 2-tolerance decoding result of the candidate sequence through the original prediction head compared with the corresponding draft head decoding result of the last time, if the draft head decoding result is among the 2-tolerance decoding results, then it is passed. If there are multiple decoding result paths that pass the verification, we select the path with a higher acceptance probability, i.e., the left-biased branch path of the sparse tree.

[0055] C. VADUSA Decoding with the Tolerance Mechanism

[0056] We propose a VADUSA decoding method with the tolerance mechanism to help the VALL-E model achieve higher acceleration and better synthesis quality simultaneously. The tolerance mechanism allows the original head to sample multiple times, which means that the "correct label" is also multiple. Figure 2 We show the process of our method at τ = 2. For the root node and each accepted node, at most 2 child nodes are accepted. If there are multiple paths accepted, we choose the one with the largest acceptance probability. In this way, on the one hand, the cost of the validation process is slightly higher, on the other hand, the acceptance length is greatly increased, and finally a higher real acceleration rate is achieved. On the other hand, compared with the simple single sampling strategy, the sampling quality is improved by choosing the best candidate path among all accepted paths.

[0057] Figure 3 : Performance of AR TTS inference. "Tuned" refers to whether the base model (VADUSA-wot or VADUSA-wt) is fine-tuned, and "Acc." refers to the average number of accepted labels. Settings: Settings; Tuned: Whether to fine-tune the model; UTMOS: Proper noun, a kind of audio quality evaluation index.

[0058] Please note that VADUSA can still be compatible with any improved sampling strategy. For example, methods such as avoiding consecutive repetitions in VALL-E 2 and constraining the attention window in ACI can be combined with VADUSA by simply using the sampling results of these strategies as the prediction results of the original prediction head. If there are multiple tolerances, we can use the above strategies for multiple sampling. In this way, both the sampling quality and the acceleration effect can be guaranteed. In the experiment, we will only use the normal sampling method for evaluation.

[0059] D. TTS-Oriented Sparse Tree Design

[0060] In practice, k is usually set to 10, i.e., each non-leaf candidate node has 10 children, which leads to an exponential explosion of the number of nodes in the entire candidate tree. To reduce the computational cost of tree attention, we need to design a sparse tree that only contains a small fraction of the candidate tree. Within a limited number of nodes, we want the sparse tree to help the model make as many acceptances as possible. Therefore, the design of the sparse tree depends on the selection of the calibration dataset and the output distribution of the draft head (otherwise, we cannot obtain the acceptance probabilities). Unfortunately, the existing sparse trees provided by MEDUSA are based on speech datasets and BPE tokenizers, which may not be suitable for the speech synthesis scenario.

[0061] We use the standard of maximizing the expected value of the acceptance length to build TTS-oriented sparse trees using off-the-shelf greedy algorithms in MEDUSA. Specifically, for each type of discrete audio token we use, we train a separate VADUSA model. Then, we run forward passes of the full tree on a subset of LibriTTS to obtain the acceptance probabilities for each node. Finally, we run the greedy algorithm to build a sparse tree for each token type.

[0062] Experiments

[0063] All experiments were conducted on an NVIDIA A800-SXM4-80GB GPU, including training using different strategies and evaluating objective and subjective metrics. Our implementation is adapted from open-source software libraries.

[0064] A. Experimental Setup

[0065] 1) Architecture of TTS system and draft heads: The whole system is composed of a speech and speech tokenizer, a codec language model, and a speech codec-based vocoder. The codec language model is the main part of the system, which consists of 12 transformer layers with 16 attention heads, 1024 hidden dimensions, and 4096 feed-forward dimensions. We use a grapheme-to-phoneme converter as the speech tokenizer. We use 2048 k-means clustering to label the speech based on the features of the last layer of the HuBERT-large model. In the ablation study, we also use two other speech labels: wav2vec 2.0, which uses the IDs of the first 16000 statistically occuring frequency groups on LibriTTS for 2-group vector quantization; EnCodec, which uses 8-layer residual vector quantization with 1024 size per codebook. All the models used in the tokenizer are pre-trained. The vocoder CTX-vec2wav is composed of two conformer blocks with 2 layers and 184 attention dimensions and a HifiGAN (trained to convert semantic labels to waveforms). The VADUSA draft head is connected to the last transformer layer of the codec language model, which is built by a residual block with a linear layer and a SiLU activation layer.

[0066] 2) Training Settings: We conducted experiments on 50k hours of English speech data from LibriHeavy and 585 hours of English speech data from LibriTTS. For the codec language model, we trained two versions of the base model as the base training stage, respectively: one epoch on Libriheavy and 20 epochs on LibriTTS with a learning rate of 0.05 and a warm-up step of 200. Then, we trained the VADUSA head on LibriTTS for 10 epochs with a fixed base model (called VADUSA-wot, without adjustment) and a fine-tuned base model (called VADUSA-wt, with adjustment), both with a learning rate of 0.002 and a warm-up step of 40. In terms of the vocoder, for each semantic label of LibriTTS, we trained the CTX-vec2wav for each label type for 1 million steps.

[0067] 3) Configurations in decoding strategy: We applied nucleus sampling and top-p with temperature set to 0.9 and 1.0 on the original prediction head of the base model and the draft heads of VADUSA. The number of draft heads was set to 4 or 6. For the draft tree, the top-k value was fixed to 10 to define the branching factor, with 64 nodes as the default choice for candidate nodes, and 96 and 128 nodes for ablation studies. We calculated the candidate expectation using more than 3000 audio samples in the LibriTTS dataset, each of which was more than 6 seconds long, to build a sparse tree. For the tolerance mechanism, we set the τ value to vary from 1 to 4 in all configurations.

[0068] B. Evaluation

[0069] We explored the speedup in inference by measuring the number of discrete tokens generated per second during the AR inference of the codec language model. The speedup ratio was calculated by dividing the output rate of the model equipped with the VADUSA head by that of the baseline model. In addition, the mean accepted tokens were also used to evaluate the effectiveness of VADUSA decoding and the tolerance mechanism. We used the word error rates (WER) measured by the conformer model to evaluate the TTS performance. We also used the UTokyo-SaruLab MOS (UTMOS) prediction system to objectively evaluate the synthesis quality of the generated speech. As shown in Table 1, the experimental setup differed in three key aspects: (1) whether 4 VADUSA heads or only the base model were involved in the inference process; (2) whether VADUSA-wot or VADUSA-wt was used; (3) whether the tolerance mechanism was applied by setting τ = 3. Figure 3

[0070] The results show that VADUSA performs well in the AR TTS system. The inference speed is greatly accelerated, while the generation quality does not decrease--in fact, the performance improves when the tolerance mechanism is applied or the base model is adjusted. This improvement is particularly evident in models trained on smaller datasets, as the VADUSA heads help capture contextual information. This behavior contrasts with the application of similar strategies in LLMs for natural language modeling tasks.

[0071] Figure 4 ​: Performance of different VADUSA configurations under the same base model trained on HuBERT+k-means 2048 tokens. Among them, the English-Chinese translation is as follows: Accepted Tokens: Average accepted tokens; Speedup: Speedup; # of Candidates: Number of candidate tokens; 4 heads: 4 prediction heads; 6 heads: 6 prediction heads.

[0072] Figure 5 : In the tolerance strategy, the speedup performance and average accepted tokens when the number set increases, the base model is trained on HuBERT+k-means 2048 tokens, and 64 candidate tokens are selected for each decoding step. Among them, the English-Chinese translation is as follows: Speedup: Speedup; Accepted Tokens: Average accepted tokens; Tolerance: Tolerance; upperbound: Upper limit of accepted tokens.

[0073] C. Ablation Study

[0074] In this part, we introduce the results of various configurations of the proposed method. In addition, we also show the overall effect of these methods on large public datasets and different types of discrete speech tokens, which can provide valuable reference for future implementation.

[0075] 1) Effectiveness analysis of VADUSA configuration and tolerance strategy: In the sparse tree construction process, we adjust the length of the candidate node to represent the number of selected nodes. Figure 4 The results shown show the performance changes of VADUSA-wt when the draft head is set to 4, 6, and the candidate selection is set to 64, 96, 128. Contrary to expectations, increasing the number of candidates does not improve the speedup performance, as accepting more candidates increases the computational cost. The tolerance mechanism varies between 1 and 4, using VADUSA-wot or VADUSA-wt strategy to train 4 or 6 heads. As Figure 5 shown, the tolerance mechanism performs well in general VADUSA configurations, and the larger the tolerance value, the better the speedup effect.

[0076] 2) Performance of different tokenization: To demonstrate the effectiveness of these methods on different wordpieces, we conduct experiments on semantic wordpieces and acoustic wordpieces. For semantic tokenization, we use the HuBERT tokens with 2048 clusters extracted from k-means, and the wav2vec2.0 tokens with the first 16000 frequencies extracted from LibriTTS, both of which are combined with 2 sets of codebooks. For acoustic tokenization, we use the EnCodec model, which is pre-trained with 50Hz RVQ and 8 codebooks of size 1024. Figure 6 It is shown that the VADUSA training strategy and the tolerance mechanism proposed in the embodiments of the present application can adapt to both large-codebook semantic tokens and acoustic tokens. However, due to the large codebook size of wav2vec2.0 and the remaining 7 codebooks of EnCodec, the logit distributions of the prediction head are relatively flat, resulting in low prediction accuracy and thus reducing the acceptance rate of the draft head. This is the reason why the observed decoding speedup performance is not ideal in these two cases. Adjusting the top-k parameter during sparse tree construction can be a potential solution.

[0077] Figure 6 : In different tokenization types, only the inference results of the base model inference and the inference results using the best-performing strategy (VADUSA-wt strategy with 4 draft heads and τ = 3). Among them, the English-Chinese translation is as follows: Tokens: tokenization type; Strategies: strategy; WER: word error rate; Speedup: speedup ratio; Acc.: average number of accepted tokens.

[0078] Figure 7 The effectiveness of the method of the embodiments of the present application in the enlarged data set is shown. We compare the performance of the 4-head fine-tuned or non-VADUSA model (VADUSA-wt or VADUSA-wot) (τ = 3) setting. Among them, the English-Chinese translation is as follows: Dataset: training data set; Strategies: strategy; WER: word error rate; UTMOS: proper noun, an audio quality evaluation index; Speedup: speedup ratio; Acc.: average number of accepted tokens.

[0079] 3) Generalizability on large-scale datasets: Our strategy also demonstrates generalizability on the enlarged dataset LibriHeavy, which contains approximately 50,000 hours of audiobook audio. In addition to word error rates (WERs), the performance of the proposed method is consistent with that observed on the smaller LibriTTS dataset. The difference in WERs is understandable, as the ability to additionally learn contextual information from the draft head has its intrinsic limitations. This reflects the trade-off problem that arises when relying on the draft head under the constraints of the base model.

[0080] Conclusions

[0081] In summary, inspired by the integration of VALL-E and MEDUSA, VADUSA demonstrates impressive effects in accelerating AR TTS decoding and improving speech quality. This is attributed to the ability to learn contextual information from the draft head, thereby expanding the functionality of the TTS model. This approach is beneficial for AR models like VALL-E, as they often exhibit instability in speech synthesis. By selecting the most suitable token in several decoding steps, VADUSA reduces the likelihood of crashes due to token selection errors. The proposed tolerance mechanism further enhances this effect. In addition, the experimental results highlight the generality of the method, providing valuable insights for future exploration and implementation.

[0082] In other embodiments, the embodiments of the present application also provide a non-volatile computer storage medium, which stores computer executable instructions, and the computer executable instructions are executable for training the improved system of the speech synthesis system in any method embodiment described above, for the improved system of the speech synthesis system;

[0083] As an implementation, the non-volatile computer storage medium of the present application stores computer executable instructions, which are configured to:

[0084] When pre-training the language-audio, the audio encoder and the text encoder are jointly trained to align audio-text pairs with similar semantics in a shared embedding space;

[0085] On the basis of the VALL-E model, combined with an advanced speculative decoding strategy and a tolerance mechanism (tolerance mechanism), a plurality of prediction heads are used to predict more tokens, wherein the advanced speculative decoding strategy includes an effective lossless speculative decoding method based on an autoregressive Transformer model;

[0086] The inference sampling is guaranteed to be consistent with the VALL-E model through a verification mechanism.

[0087] The non-transitory computer-readable storage medium can include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application required by a function, and the data storage area can store data created according to the use of the improved method and system of the speech synthesis system, and the like. In addition, the non-transitory computer-readable storage medium can include a high-speed random access memory, and can also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the non-transitory computer-readable storage medium can optionally include a memory that is remotely arranged relative to the processor, and these remote memories can be connected to the improved method of the speech synthesis system through a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0088] The embodiment of the present application also provides a computer program product, which comprises a computer program stored on a non-transitory computer-readable storage medium, and the computer program comprises program instructions, which, when executed by a computer, cause the computer to execute any of the above-mentioned improved methods of the speech synthesis system.

[0089] Figure 8 is a structural schematic diagram of an electronic device provided by the embodiment of the present application, as shown in the figure, the device comprises one or more processors 810 and a memory 820, Figure 8 In the figure, the processor 810 is taken as an example. The device of the improved method and system of the speech synthesis system can also comprise an input device 830 and an output device 840. The processor 810, the memory 820, the input device 830 and the output device 840 can be connected through a bus or other means, Figure 8 In the figure, the connection through the bus is taken as an example. The memory 820 is the non-transitory computer-readable storage medium mentioned above. The processor 810 executes various functional applications and data processing of the server by running the non-transitory software programs, instructions and modules stored in the memory 820, that is, implements the improved method of the speech synthesis system according to the above-mentioned method embodiment. The input device 830 can receive input digital or character information, and generate key signal input related to the user settings and function control of the improved device of the speech synthesis system. The output device 840 can include a display device such as a display screen. Figure 8 The above-mentioned product can execute the method provided by the embodiment of the present application, has the corresponding functional modules and beneficial effects of executing the method. Technical details not described in detail in the present embodiment can be referred to the method provided by the embodiment of the present application.

[0090]

[0091] ​As an implementation form, the electronic device is applied to an improved system of a speech synthesis system, comprising: at least one processor; and a memory in communication connection with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to:

[0092] On the basis of the VALL-E model, a plurality of prediction heads are used to predict more tokens in combination with an advanced speculative decoding strategy and a tolerant mechanism, wherein the advanced speculative decoding strategy comprises an effective lossless speculative decoding method based on an autoregressive Transformer model.

[0093] The inference sampling is guaranteed to be consistent with the VALL-E model through a verification mechanism.

[0094] The electronic device of the embodiments of the present application exists in various forms, including but not limited to:

[0095] (1) Mobile communication device: the feature of this kind of device is to have mobile communication function and to provide voice and data communication as the main target. This kind of terminal includes: smart phone, multimedia phone, functional phone, and low-end phone, etc.

[0096] (2) Ultra-mobile personal computer device: this kind of device belongs to the category of personal computers, has computing and processing functions, and generally also has the characteristics of mobile Internet. This kind of terminal includes: PDA, MID and UMPC device, etc.

[0097] (3) Portable entertainment device: this kind of device can display and play multimedia content. This kind of device includes: audio and video player, handheld game console, electronic book, and smart toy and portable car navigation device.

[0098] (4) Server: a device providing computing services, the composition of the server includes processor, hard disk, memory, system bus, etc. The server is similar to the general computer architecture, but due to the need to provide high-reliability services, the requirements for processing capability, stability, reliability, security, scalability, manageability, etc. are higher.

[0099] (5) Other electronic devices with data interaction function.

[0100] The device embodiments described above are merely illustrative, wherein the units illustrated as separate components can or can not be physically separated, and the components illustrated as units can or can not be physical units, i.e., can be located in one place, or can be distributed to multiple network units. Part or all of the modules can be selected to achieve the purposes of the embodiments according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0101] Through the description of the above embodiments, those skilled in the art can clearly understand that the embodiments can be realized by means of software and the necessary general hardware platform, and of course can also be realized by hardware. Based on such understanding, the above technical solutions can be embodied in the form of software products, and the computer software products can be stored in a computer readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and include a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods of the embodiments or some parts of the embodiments.

[0102] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. An improved method of a speech synthesis system for a VALL-E model, comprising: combining an advanced speculation decoding strategy and a leniency mechanism on the basis of the VALL-E model to predict more tokens with multiple prediction heads, wherein the advanced speculation decoding strategy comprises an effective lossless speculation decoding method based on an autoregressive Transformer model; ensuring that inference sampling is consistent with the VALL-E model through a verification mechanism.

2. The method of claim 1, wherein, The advanced speculation decoding strategy comprises: using the absolute position ID of the top-3 tree nodes arranged from left to right in each layer to represent the sampling result; for a subset of the top-3 tree, a sparse tree, the results are spliced step by step by layer, and the spliced results are input into an autoregressive decoder for a decoding operation, wherein an attention mask is constructed to ensure that each node can only access the information of its ancestor nodes.

3. The method of claim 1, wherein, The advanced speculation decoding strategy comprises: Under tree attention, use the draft head to predict future tokens, and hope that the VALL-E model accepts a candidate path with a higher acceptance probability.

4. The method of claim 1, wherein, The leniency mechanism comprises: when the leniency is 2, the autoregressive head performs two polynomial samplings in the core sampling process.

5. The method of claim 1, wherein, The verification mechanism comprises: After comparing and verifying the autoregressive head sampling result and the VADUSA head result, in the acceptable branch of the sparse tree, the leftmost branch is preferentially selected.

6. An electronic device comprising: At least one processor and a memory connected in communication with the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the steps of any one of claims 1-5.

7. A storage medium having stored thereon a computer program, characterized in that The computer program is executed by the processor to implement the steps of any one of claims 1-5.

Images