Improvement method of speech synthesis system, electronic equipment and storage medium

Abstract

The embodiment of the invention discloses an improvement method of a speech synthesis system, electronic equipment and a storage medium, the method is used for a VALL-E model, and comprises the following steps: on the basis of the VALL-E model, combining an advanced speculation decoding strategy and a tolerance mechanism, and using a plurality of prediction headers to predict more marks, the advanced speculation decoding strategy comprises an effective lossless speculation decoding method based on an autoregression Transform model; and inference sampling is ensured to be consistent with the VALL-E model through a verification mechanism. According to the embodiment of the invention, the original VALL-E model is completely reserved on the whole, more tokens are predicted by using more prediction heads in combination with speculation sampling and a tolerance mechanism, and reasoning sampling is ensured to be consistent with the original model through a verification mechanism. In this way, the original model performance can be ensured, and the stability and the generation effect of the model are optimized by predicting farther tokens through multiple heads. Through the proposal of a'tolerance 'mechanism, more head prediction results are accepted, the acceleration effect is obviously enhanced, and meanwhile, the voice generation performance is improved.

Description

Technical Field

[0001] The invention belongs to the technical field of speech synthesis, and in particular relates to an improved method, electronic equipment and storage medium of a speech synthesis system. Background Art

[0002] In the related technologies, speculative decoding strategies have not been applied in the field of speech generation. The more relevant ones are the following inference acceleration methods for autoregressive speech generation models: (1) Using low-bitrate acoustic coding to shorten discrete speech sequences and accelerate inference; (2) Using acoustic BPE (Byte-Pair Encoding) to shorten discrete speech sequences and accelerate inference; (3) VALL-E 2 (based on VALL-E, the sampling strategy is improved and merged coding training is used to reduce the discrete speech coding bit rate during training. VALL-E is a speech generation architecture that combines autoregressive and non-autoregressive discrete speech language models) directly uses the merged discrete speech coding method to accelerate inference. Among them, low-bitrate acoustic coding uses new technology to compress speech into a discrete representation with a lower bit rate, so that speech of equal length can have a shorter discrete sequence representation, thereby reducing the inference steps during autoregressive inference and achieving an acceleration effect; in both acoustic BPE and VALL-E 2, the original discrete coding is first merged into a new coding form, the new coding is autoregressively inferred, and then restored to the original coding through simple decoding steps, which also reduces the autoregressive inference steps.

[0003] The inventors found that these technologies change the training strategy and make significant changes to the model itself, which may affect the quality of speech synthesis and increase instability. Summary of the invention

[0004] The embodiments of the present invention provide an improved method of a speech synthesis system, an electronic device and a storage medium, which are used to solve at least one of the above technical problems.

[0005] In a first aspect, an embodiment of the present invention provides an improved method for a speech synthesis system, which is used for a VALL-E model, including: based on the VALL-E model, combining an advanced speculative decoding strategy and a tolerance mechanism, using multiple prediction heads to predict more tags, wherein the advanced speculative decoding strategy includes an effective lossless speculative decoding method based on an autoregressive Transformer model; and ensuring that the inference sampling is consistent with the VALL-E model through a verification mechanism.

[0006] In a second aspect, an embodiment of the present invention further provides a computer program product, which includes a computer program stored on a non-volatile computer-readable storage medium, and the computer program includes program instructions, which, when executed by a computer, cause the computer to execute the steps of the training method for improving the speech synthesis system of any embodiment of the present invention.

[0007] In a third aspect, an embodiment of the present invention further provides an electronic device comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the steps of the method described in the first aspect.

[0008] In a fourth aspect, an embodiment of the present invention further provides a storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the method described in the first aspect.

[0009] The embodiment of the present application retains the original VALL-E model in its entirety, combines the "MEDUSA" speculative sampling and tolerance mechanism, uses more prediction heads to predict more tokens, and then uses a verification mechanism to ensure that the inference sampling is consistent with the original model. This can ensure the performance of the original model, and achieve a beam search effect by predicting more distant tokens through multiple heads, optimizing the stability and generation effect of the model. The introduction of the "tolerance" mechanism makes the results of more prediction heads acceptable, significantly enhances the acceleration effect, and improves the speech generation performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying any creative work.

[0011] Figure 1 A flowchart of a training method for an improved system of a speech synthesis system provided by an embodiment of the present invention; Figure 2 VADUSA decoding with a tolerance τ=2 provided by an embodiment of the present invention; Figure 3 The performance of AR TTS reasoning provided by an embodiment of the present invention; Figure 4The performance of different VADUSA configurations under the same base model based on HuBERT+k-means2048 label training provided by an embodiment of the present invention; Figure 5 The acceleration performance and average number of accepted marks when the number set increases in the tolerance strategy provided by one embodiment of the present invention; Figure 6 Inference results of only the basic model reasoning and the best performing strategy among different label types provided by an embodiment of the present invention; Figure 7 The effectiveness of the method of the present application provided in an embodiment of the present invention in amplifying a data set; Figure 8 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0012] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0013] The inventors found that the defects of the related art are mainly due to the following reasons: (1) compression with lower bit rate will lose more acoustic information; (2) training uses merged tokens, which increases the risk of single-step token prediction errors during inference, reducing synthesis performance and stability.

[0014] This is the first solution to apply the speculative sampling strategy to the autoregressive speech generation system. The speculative sampling strategy is usually widely used in natural language processing tasks. However, how to adapt it to the generation process of discrete speech representation requires further algorithm design and experimental exploration. In addition, our method proposes a "tolerant" strategy for speech sequences, which makes the acceleration effect of the speculative sampling method more significant.

[0015] The embodiment of the present application proposes a speculative sampling method named "VADUSA", which retains the original VALL-E model in its entirety, combines the "MEDUSA" speculative sampling and tolerance mechanism, uses more prediction heads to predict more tokens, and then uses a verification mechanism to ensure that the inference sampling is consistent with the original model. This can ensure the performance of the original model, and achieve a beam search-like effect by predicting more distant tokens through multiple heads, optimizing the stability and generation effect of the model. The introduction of the "tolerance" mechanism makes the results of more prediction heads acceptable, significantly enhances the acceleration effect, and improves the speech generation performance.

[0016] In the embodiment of the present application, VADUSA decoding with a tolerance of 2: We use the absolute position ID of the top-3 tree nodes arranged from left to right in each layer to represent the sampling result. The sparse tree is a subset of the complete top-3 tree. We gradually concatenate the results by level and input them into the auto-regressive (AR) decoder for a decoding operation. The constructed attention mask ensures that each node can only access the information of its ancestor node. The tolerance mechanism is Figure 2 This mechanism means that when the tolerance is 2, the AR head will perform two polynomial samplings during the kernel sampling process. Figure 2 As shown, after comparing the AR head sampling results with the VADUSA head results in the verification phase, the leftmost branch is preferentially selected among the acceptable branches of the sparse tree.

[0017] In the process of implementing this application, the inventor has also proposed some alternative versions, which may be experiments conducted under different parameter settings or fine-tuning of the algorithm implementation. The following are several possible beta versions: When constructing a sparse tree, it is possible to calculate the expectation not only based on the node acceptance rate, but also multiply the depth of the node by the expectation to construct a sparse tree that emphasizes the length of acceptance.

[0018] In addition, before the final product was formed, the inventor also used the following beta version: construct a sparse tree that emphasizes the length of acceptance: the real acceptance rate decreases, and the decoding process either decodes many tokens in one step, or very few or even no token is accepted. It can decode faster in some cases, but the fluctuation is large.

[0019] Please refer to Figure 1 , which shows a flow chart of an embodiment of a training method for an improved system of a speech synthesis system of the present application. The training method is used for the VALL-E model of the aforementioned embodiment.

[0020] like Figure 1As shown, in step 101, based on the VALL-E model, combined with advanced speculative decoding strategy and tolerance mechanism, multiple prediction heads are used to predict more tags, wherein the advanced speculative decoding strategy includes an effective lossless speculative decoding method based on an autoregressive Transformer model; In step 102, a verification mechanism is used to ensure that the inference sampling is consistent with the VALL-E model.

[0021] In some optional embodiments, the advanced speculative decoding strategy includes: using the absolute position ID of the top-3 tree nodes arranged from left to right in each layer to represent the sampling results; for a subset sparse tree of the top-3 trees, stepwise splicing the results by level, and inputting the splicing results into an autoregressive decoder for a decoding operation, wherein an attention mask is constructed to ensure that each node can only access the information of its ancestor node.

[0022] The embodiment of the present application retains the original VALL-E model in its entirety, combines the "MEDUSA" speculative sampling and tolerance mechanism, uses more prediction heads to predict more tokens, and then uses a verification mechanism to ensure that the inference sampling is consistent with the original model. This can ensure the performance of the original model, and achieve a beam search effect by predicting more distant tokens through multiple heads, optimizing the stability and generation effect of the model. The introduction of the "tolerance" mechanism makes the results of more prediction heads acceptable, significantly enhances the acceleration effect, and improves the speech generation performance.

[0023] In some optional embodiments, the method comprises: Under tree attention, we use the scratch head to predict future tokens and hope that the VALL-E model accepts a candidate path with a higher acceptance probability.

[0024] In some optional embodiments, the tolerance mechanism includes: when the tolerance is 2, the autoregressive head performs two polynomial samplings during the kernel sampling process.

[0025] In some optional embodiments, the ensuring that the inference sampling is consistent with the VALL-E model through a verification mechanism includes: After comparing the sampling results of the autoregressive head in the validation phase with the results of the VADUSA head, the leftmost branch is preferentially selected among the acceptable branches of the sparse tree.

[0026] It should be noted that the above method steps are not used to limit the execution order of each step. In fact, some steps may be executed simultaneously or in the opposite order to the specified step order, and this application has no limitation on this.

[0027] The solution of the present application is described below through a specific embodiment so that those skilled in the art can better understand the solution of the present application.

[0028] Autoregressive architectures (such as GPT) are widely used in modern text-to-speech (TTS) systems. However, it can lead to high inference time due to the challenge of next token prediction for long sequences of speech tokens. In this embodiment of the present application, we propose VADUSA, which is the first pioneering method to accelerate autoregressive TTS through speculative decoding. Our results show that VADUSA can not only significantly improve the inference speed, but also improve the performance by introducing a draft header to automatically regressively predict future speech content. In addition, the addition of a tolerance mechanism in the sampling process speeds up inference while ensuring that speech quality is not affected. Our method demonstrates strong generalization capabilities on large-scale datasets and different types of speech coding.

[0029] Large language models (LLMs) with auto-regressive (AR) architectures have achieved great success in recent years. They use a next-token prediction strategy to generate speech byte pair encoding (BPE) tags. This strategy samples the next tag from a multi-branch distribution generated based on historical tags. Although simple, it can effectively establish coherence in a longer context. With the invention of discrete neural audio tags, speech discourse can also be encoded as a sequence of discrete tags. Subsequently, the LLM paradigm was introduced into the field of speech modeling and synthesis. Studies such as SPEAR-TTS, VALL-E, and BASE-TTS have implemented a pure decoder AR architecture for generating speech codecs using large-scale training data, which can generate natural-sounding speech.

[0030] However, there is a significant difference between speech data and voice data: for the same sentence, the speech token sequence is often much longer than the speech token sequence. For example, a 10-second speech requires 500 HuBERT tokens (50Hz) to represent, while the speech transcription of the same speech only requires about 20 to 40 HuBERT tokens. This is reasonable because speech has a higher information density and speech captures finer sound details over time, resulting in longer token sequences. In addition, the trivial AR architecture predicts only one token in each reasoning step, resulting in particularly long speech generation time. This poses a challenge between the need for low-latency spoken language synthesis and the modeling of long speech sequences.

[0031] To alleviate this problem, one approach in the past was to reduce the length of the input / output sequence through distillation or better information compression, that is, to create low-bitrate speech tags. Many studies on neural codecs have attempted to reduce the bit rate through properly configured vector quantization (VQ) modules and carefully designed training processes. Another encouraging study is aBPE, which applies BPE to discrete speech tags.

[0032] This method applies BPE to discrete speech markers to achieve lossless compression, showing great potential in accelerating speech synthesis. However, these low-bitrate markers either affect the accuracy of speech reconstruction or cause frame shift variations, thus affecting the naturalness and quality of speech synthesis.

[0033] Another approach is to predict more speech tokens in one decoding step, which is the focus of this application. This idea has been developed since the RNN era, starting with Subscale WavRNN. It folds the wav token sequence in a subscale manner, allowing parallel prediction of multiple tokens. More recently, VALL-E 2 uses a block-by-block approach to generate a large block of future tokens (usually 2 tokens) in one AR iteration. However, this approach usually leads to a loss in quality because the historical conditions are insufficient when predicting at least half of a sequence. In contrast, speculative decoding, which is widely used in LLM inference, can provide speedups without losing performance by simply adding an additional draft model.

[0034] Given the potential of speculative decoding, we first applied our advanced speculative decoding strategy MEDUSA to the AR TTS model VALL-E and conducted comprehensive experiments on a variety of discrete tokens. During the experiment, we found huge challenges. The inherent complexity and variability of speech synthesis often lead to suboptimal prediction accuracy and acceptance rate, making it difficult to achieve the expected acceleration and quality. This prompted us to innovate beyond adaptability and eventually developed the VADUSA method. By integrating a tolerance mechanism, VADUSA not only speeds up the decoding process, but also improves the robustness and quality of the overall synthesis by effectively managing the uncertainty inherent in speculative decoding. This new approach has proven to be crucial in narrowing the gap between fast decoding and maintaining high synthesis fidelity.

[0035] VADUSA: FAST AND HIGH-QUALITY AR TTS In this section, we introduce a novel AR TTS decoding method VADUSA in detail, which can achieve both fast decoding and high-quality speech synthesis. This section first introduces MEDUSA, an effective lossless speculative decoding method based on the AR Transformer model. Then, we deeply study the unsatisfactory illusory combination of VALL-E and MEDUSA and analyze its limitations. After that, we propose VADUSA, a non-trivial integration of MEDUSA and VALL-E, combined with a tolerance mechanism, to achieve decoding acceleration and improved synthesis quality. Finally, we briefly introduce the design of sparse candidate trees for TTS.

[0036] A. Speculative Decoding and MEDUSA Speculative decoding is an efficient and lossless technique for accelerating AR decoding. The key idea is to use a small and fast draft model to generate predictions, which are then verified in parallel by the target model. To avoid the expensive process of training the draft model separately, MEDUSA introduces several additional draft heads on top of the original model. Specifically, in the first pass, it is assumed that the original head predicts the first next token t 1,1 , then the i-th draft head is responsible for predicting the (i+1)th next token t 1,i+1 (i ≥ 1), so that multiple tokens can be generated in parallel. The first next token and draft tokens generated will be concatenated in sequence and fed into the same model for the next forward pass. In the second pass, the model will verify the draft tokens almost in parallel. For example, if t 1,2 With from t 1,1 If the sampled “correct token” matches the distribution generated by the original header, then it will be accepted. Similarly, if t 1,2 has been accepted and is consistent with t 1,2 The first next token of 1,3 will be accepted. Assume that t 1,3 Accepted, t 1,4 If rejected, then 1,3 The original prediction head and the draft head will predict t 2,1 ,t 2,2 ,......, provide information for the third pass, the draft acceptance length of the second pass is 2.

[0037] In order to make a better "guess", MEDUSA also considers multiple candidate tokens for each head and builds a draft tree, where the root node represents the first next token, each node represents a candidate token, and each root path represents a candidate continuation. For each draft head, the k tokens with the largest logarithm are selected as candidate tokens. Each non-leaf node has k child nodes. The (i+1)th layer has k i-1 nodes, filled with the first k tokens produced by the i-th draft head. All root-to-leaf paths constitute all possible combinations of candidate nodes. Through a carefully constructed tree mask, the tokens of the entire tree can be verified in a single forward pass (named tree attention). This approach greatly improves the acceptance length of the draft, and the additional computational cost is justified by facilitating sparse trees that will be introduced later. The simplicity of the MEDUSA draft head and its compatibility with any Transformer-based AR model make MEDUSA highly adaptable and efficient.

[0038] B. Vanilla Combination and Re-thinking In the original version of VADUSA, we simply use the VALL-E TTS model as the base model for speech synthesis, and integrate multiple MEDUSA heads on top of it to speed up generation. However, in initial experiments, we found that its acceptance length could not be compared with the results of MEDUSA on speech LLM. This is because the speech tokens generated by systems such as EnCodec, HuBERT, and wav2vec 2.0 are obtained through vector quantization or ID clustering, rather than having obvious abstract semantic differences like text tokens. Under the AR language modeling paradigm, multiple speech tokens tend to have very similar statistical characteristics. This means that the multinomial distributions predicted by models such as VALL-E tend to be more "average", while using the draft head to select the top-k tokens with the highest probability can only cover a small probability range. Given that the original prediction head is only sampled once as a "correct token" during verification, it is challenging for the draft heads in VALL-E to make accurate predictions.

[0039] In addition, from the perspective of improving sample quality, tree attention can be seen as an efficient version of beam search. In beam search, by predicting the tokens of the next few steps, we can select the combination with the highest probability, thereby improving sample quality. Similarly, under tree attention, we use the draft head to predict future tokens and hope that the model accepts a candidate path with a higher acceptance probability.

[0040] Figure 2 VADUSA decoding with tolerance τ=2 is shown: We use the absolute positional ID of the node in the top-3 tree (each layer is ordered from left to right) to represent the sampling results. The sparse tree is a subset of the full top-3 tree. We concatenate the results layer by layer in order and input them into the AR decoder for single-step decoding. We construct an attention mask to ensure that each node can only access the information of its ancestors. The tolerance mechanism is detailed in the "2-Tolerance Prediction" module in the figure, which means that when τ=2, the AR head performs two multinomial samplings during the kernel sampling process. As shown in the figure, for branches in the sparse tree that are considered acceptable results, the leftmost branch is given priority.

[0041] In the figure, the Chinese and English correspondence is as follows: Auto-Regressive Decoder: Auto-Regressive Decoder; AR head: original model prediction head; Head123: VADUSA prediction head 123; Attention Mask: Attention Mask; 2-tolerance Predictions: 2 tolerance prediction; verify: verification. In one embodiment, in conjunction with the figure, the specific decoding process is as follows: first, the original model decoding result and the topk sampling result of the draft header are obtained through the auto-regressive decoder, and then the left-biased sparse tree and attention mask are constructed as shown in the figure, and the results in the sparse tree are merged and input into the auto-regressive decoder for decoding again, and the 2-tolerance decoding result of the original prediction head in the candidate sequence is compared with the last corresponding draft header decoding result for verification. If the draft header decoding result is within the 2-tolerance decoding result, it passes. If multiple decoding result paths pass the verification, we select the path with a higher probability of acceptance, that is, the branch path of the sparse tree that is more biased to the left.

[0042] C. VADUSA Decoding with the Tolerance Mechanism We propose a VADUSA decoding method with a tolerance mechanism to help the VALL-E model achieve both higher acceleration and better synthesis quality. The tolerance mechanism allows the original prediction head to be sampled multiple times, which means that the “correct labeling” is also multiple times. Figure 2 The process of our method when τ=2 is shown. For the root node and each accepted node, at most 2 child nodes will be accepted. If multiple paths are accepted, we will choose the one with the highest acceptance probability. In this way, on the one hand, the cost of the verification process is slightly higher, and on the other hand, the acceptance length is greatly increased, and finally a higher real acceleration rate is achieved. On the other hand, compared with the simple single sampling strategy, the sampling quality is improved by selecting the best candidate path among all accepted paths.

[0043] Figure 3 : AR TTS inference performance. "Tuned" refers to whether the base model (VADUSA-wot or VADUSA-wt) is fine-tuned, and "Acceptance (Acc.)" refers to the average number of accepted tokens. Settings: Settings; Tuned: whether to fine-tune the model; UTMOS: proper noun, proper noun, an audio quality evaluation indicator.

[0044] Note that VADUSA is still compatible with any improved sampling strategy. For example, methods such as avoiding consecutive repetitions in VALL-E 2 and constraining the attention window in ACI can be combined with VADUSA by simply using the sampling results of these strategies as the prediction results of the original prediction head. If there are multiple tolerances, we can use the above strategy for multiple sampling. This can ensure the sampling quality and achieve acceleration effects. In the experiment, we will only use the ordinary sampling method for evaluation.

[0045] D. TTS-Oriented Sparse Tree Design In practical applications, k is usually set to 10, that is, each non-leaf candidate node has 10 child nodes, which will cause the number of nodes in the entire candidate tree to explode exponentially. In order to reduce the computational cost of tree attention, we need to design a sparse tree that only contains a small part of the candidate tree. Within the limited number of nodes, we hope that the sparse tree can help the model make as many acceptances as possible. Therefore, the design of the sparse tree depends on the choice of the calibration dataset and the output distribution of the draft head (otherwise, we cannot obtain the acceptance probability). Unfortunately, the existing sparse trees provided by MEDUSA are based on speech datasets and BPE tokenizers, which may not be suitable for speech synthesis scenarios.

[0046] We use an off-the-shelf greedy algorithm in MEDUSA to build sparse trees for TTS, adopting the criterion of maximizing the expected value of acceptance length. Specifically, for each discrete audio token we use, we train a VADUSA model separately. Then, we run forward passes of the full tree on a subset of LibriTTS to obtain the acceptance probability of each node. Finally, we run the greedy algorithm to build sparse trees for each token type.

[0047] experiment All experiments are conducted on an NVIDIA A800-SXM4-80GB GPU, including training with different strategies and evaluation with objective and subjective metrics. Our implementation is adapted from an open source software library.

[0048] A. Experimental Setup 1) Architecture of TTS system and draft heads: The whole system consists of a cascade of speech and voice tokenizers, a codec language model, and a vocoder based on the speech codec. The codec language model is the main part of the system and consists of 12 conversion layers with 16 attention heads, 1024 hidden dimensions, and 4096 feedforward dimensions. We use a grapheme-to-phoneme converter as the speech tokenizer. We label the speech using 2048 k-means clustering based on the features of the last layer of the HuBERT-large model. Two other speech tokenizers are also used in the ablation study: wav2vec 2.0, which uses 2 sets of vector quantization using the IDs of the first 16,000 statistically occurring frequency groups on LibriTTS; and EnCodec, which uses 8 layers of residual vector quantization with a size of 1024 per codebook. All models used in the tokenizer are pre-trained. The vocoder CTX-vec2wav consists of two conformer blocks with 2 layers and 184 attention dimensions and a HifiGAN trained to convert semantic tokens to waveforms. The VADUSA scratch head is connected to the last transformer layer of the encoder-decoder language model, which is built from a residual block with a linear layer and a SiLU activation layer.

[0049] 2) Training Settings: We conducted experiments on 50k hours of English speech data from LibriHeavy and 585 hours of English speech data from LibriTTS. For the codec language model, we trained two versions of the base model as the base training stage: one epoch on LibriHeavy and 20 epochs on LibriTTS, with a learning rate of 0.05 and warm-up steps set to 200. We then trained the VADUSA head on LibriTTS for 10 epochs with a fixed base model (called VADUSA-wot, without adjustment) and a fine-tuned base model (called VADUSA-wt, with adjustment), both with a learning rate of 0.002 and a warm-up step of 40. For the vocoder, CTX-vec2wav was trained for 1 million steps for each semantic token in LibriTTS for each token type.

[0050] 3) Configurations in decoding strategy: We applied kernel sampling and top-p on the original prediction heads of the base model and the scratch heads of VADUSA, setting the temperature to 0.9 and 1.0. The number of scratch heads was set to 4 or 6. For the scratch tree, the top-k value was fixed to 10 to define the branching factor, 64 nodes were selected as candidate nodes by default, and 96 and 128 nodes were used for ablation studies. We used more than 3000 audio samples from the LibriTTS dataset, each of which was more than 6 seconds, to calculate the candidate expected values ​​to build a sparse tree. For the tolerance mechanism, we set τ values ​​ranging from 1 to 4 in all configurations.

[0051] B. Evaluation We explored the improvement in inference speed by measuring the number of discrete tokens generated per second during the AR inference process of the codec language model. The speedup is calculated by dividing the output rate of the model equipped with the VADUSA head by the output rate of the baseline model. In addition, the mean number of accepted tokens is also used to evaluate the effectiveness of the VADUSA decoding and tolerance mechanism. We use the word error rate (WER) measured by the conformal transducer model to evaluate the TTS performance. We also used the UTokyo-SaruLab MOS (UTMOS) prediction system to objectively evaluate the synthesis quality of the generated speech. Figure 3 As shown in Figure 2, the experimental settings differ in three key aspects: (1) whether 4 VADUSA heads or only the base model are involved in the inference process; (2) whether VADUSA-wot or VADUSA-wt is used; and (3) whether the tolerance mechanism is applied by setting τ = 3.

[0052] Results show that VADUSA performs well in AR TTS systems. Inference speed is greatly accelerated without degradation in generation quality - in fact, performance improves when applying tolerance mechanisms or tuning the base model. The improvement is particularly pronounced in models trained on smaller datasets, as the VADUSA head helps capture contextual information. This behavior is in stark contrast to the application of similar strategies in LLM for natural language modeling tasks.

[0053] Figure 4:The performance of different VADUSA configurations under the same base model based on HuBERT+k-means2048 label training. The Chinese and English correspondence is as follows: Accepted Tokens: average number of accepted tokens; Speedup: speedup ratio; # ofCandidates: number of candidate tokens; 4heads: 4 prediction heads; 6heads: 6 prediction heads.

[0054] Figure 5 : Speedup performance and average number of accepted tokens when the number of sets increases in the tolerance strategy. The base model is trained on HuBERT+k-means 2048 tokens, and 64 candidate tokens are selected in each decoding step. The Chinese and English translations are as follows: Speedup: speedup ratio; Accepted Tokens: average number of accepted tokens; Tolerance: tolerance. upperbound: upper limit of the number of accepted tokens.

[0055] C. Ablation studies In this section, we present the results of various configurations of the proposed methods. In addition, we show the overall performance of these methods on large public datasets and different types of discrete speech tokens, which can provide valuable references for future implementations.

[0056] 1) Effectiveness Analysis of VADUSA Configuration and Tolerance Strategy: During the sparse tree construction process, we adjust the length of candidate nodes to represent the number of selected nodes. Figure 4 The results shown show the performance variation of VADUSA-wt when the number of scratch heads is set to 4, 6, and the number of candidate selections is set to 64, 96, 128. Contrary to expectations, increasing the number of candidates does not improve the speedup performance since accepting more candidates increases the computational cost. The tolerance mechanism is varied between 1 and 4, and 4 or 6 heads are trained using either the VADUSA-wot or VADUSA-wt strategy. Figure 5 As shown, the tolerance mechanism works well in general VADUSA configurations, and the larger the tolerance value, the better the acceleration effect.

[0057] 2) Performance of different speech tags: To demonstrate the effectiveness of these methods on different morphemes, we experimented with semantic morphemes and acoustic morphemes. For semantic tags, we used HuBERT tags with 2048 clusters extracted from k-means and wav2vec2.0 tags with the top 16,000 frequencies extracted from LibriTTS, both of which were combined from 2 sets of codebooks. For acoustic tags, we used the EnCodec model, which was pre-trained with 50HzRVQ and 8 codebooks of size 1024. Figure 6 It is shown that the VADUSA training strategy and the tolerance mechanism suggested by the embodiments of the present application can adapt to both large-codebook semantic tokens and acoustic tokens. However, due to the large codebook sizes of wav2vec2.0 and the remaining 7 codebooks of EnCodec, the logit distributions of the prediction heads are relatively flat, resulting in low prediction accuracy, which reduces the acceptance rate of the draft heads. This is the reason for the unsatisfactory decoding acceleration performance observed in both cases. Adjusting the top-k parameters during sparse tree construction may be a potential solution.

[0058] Figure 6 : Inference results of only the base model and the best performing strategy (VADUSA-wt strategy with 4 scratch heads and τ=3) in different token types. The Chinese and English correspondence is as follows: Tokens: token type; Strategies: strategy; WER: word error rate; Speedup: speedup ratio; Acc.: average number of accepted tokens.

[0059] Figure 7 The effectiveness of the method of the embodiment of the present application in the enlarged data set is shown. We compared the performance under the setting of 4-head fine-tuning or non-VADUSA model (VADUSA-wt or VADUSA-wot) (τ=3). The Chinese and English correspondences are as follows: Dataset: training data set; Strategies: strategy; WER: word error rate; UTMOS: proper noun, an audio quality evaluation indicator; Speedup: speedup ratio; Acc.: average number of accepted tokens.

[0060] 3) Generalizability on Large-Scale Datasets: Our strategy also demonstrates generalizability on the enlarged dataset LibriHeavy, which contains about 50,000 hours of audiobook audio. The performance of the proposed method is consistent with that observed on the smaller LibriTTS dataset, except for word error rates (WERs). The difference in WER is understandable, as the ability of appending word headers to learn contextual information has its inherent limitations. This reflects the trade-offs that arise when relying on draft headers given the constraints of the base model.

[0061] in conclusion In summary, inspired by the integration of VALL-E and MEDUSA, VADUSA shows impressive results in accelerating AR TTS decoding and improving speech quality. This is thanks to the ability of the scratch head to learn contextual information, thereby extending the capabilities of the TTS model. This approach benefits AR models like VALL-E, which often exhibit instabilities in speech synthesis. By selecting the most appropriate tokens in several decoding steps, VADUSA reduces the likelihood of crashes due to wrong token selection. The proposed tolerance mechanism further enhances this effect. In addition, the experimental results highlight the versatility of the method and provide valuable inspiration for future exploration and implementation.

[0062] In some other embodiments, the embodiments of the present invention further provide a non-volatile computer storage medium, the computer storage medium stores computer executable instructions, the computer executable instructions can execute the training method of the improved system of the speech synthesis system in any of the above method embodiments, and is used for the improved system of the speech synthesis system; As an implementation mode, the non-volatile computer storage medium of the present invention stores computer executable instructions, and the computer executable instructions are configured as follows: In contrastive language-audio pre-training, the audio encoder and text encoder are jointly trained to align semantically similar audio-text pairs in a shared embedding space; Based on the VALL-E model, an advanced speculative decoding strategy and a tolerance mechanism are combined to use multiple prediction heads to predict more tags, wherein the advanced speculative decoding strategy includes an effective lossless speculative decoding method based on an autoregressive Transformer model; The inference sampling is ensured to be consistent with the VALL-E model through a verification mechanism.

[0063] The non-volatile computer-readable storage medium may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the improved method of the speech synthesis system and the use of the system, etc. In addition, the non-volatile computer-readable storage medium may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the non-volatile computer-readable storage medium may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the improved method of the speech synthesis system via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0064] An embodiment of the present invention also provides a computer program product, which includes a computer program stored on a non-volatile computer-readable storage medium. The computer program includes program instructions. When the program instructions are executed by a computer, the computer executes any of the above-mentioned methods for improving a speech synthesis system.

[0065] Figure 8 is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention, such as Figure 8 As shown, the device includes: one or more processors 810 and a memory 820, Figure 8 A processor 810 is used as an example. The improved method of the speech synthesis system and the device of the system may also include: an input device 830 and an output device 840. The processor 810, the memory 820, the input device 830 and the output device 840 may be connected via a bus or other means. Figure 8 In the example, the connection through the bus is taken. The memory 820 is the above-mentioned non-volatile computer-readable storage medium. The processor 810 executes various functional applications and data processing of the server by running the non-volatile software programs, instructions and modules stored in the memory 820, that is, the improved method of the speech synthesis system of the above-mentioned method embodiment is realized. The input device 830 can receive input digital or character information, and generate key signal input related to user settings and function control of the improved device of the speech synthesis system. The output device 840 may include display devices such as display screens.

[0066] The above product can execute the method provided by the embodiment of the present invention, and has the functional modules and beneficial effects corresponding to the execution method. For technical details not described in detail in this embodiment, please refer to the method provided by the embodiment of the present invention.

[0067] As an implementation mode, the electronic device is applied to an improved system of a speech synthesis system, comprising: at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can: Based on the VALL-E model, multiple prediction heads are used to predict more tags in combination with advanced speculative decoding strategies and tolerance mechanisms, wherein the advanced speculative decoding strategies include an effective lossless speculative decoding method based on an autoregressive Transformer model; The inference sampling is ensured to be consistent with the VALL-E model through a verification mechanism.

[0068] The electronic device of the embodiment of the present application exists in various forms, including but not limited to: (1) Mobile communication equipment: This type of equipment is characterized by having mobile communication functions and its main purpose is to provide voice and data communications. This type of terminal includes: smart phones, multimedia phones, functional phones, and low-end phones.

[0069] (2) Ultra-mobile personal computer devices: These devices fall into the category of personal computers, have computing and processing functions, and generally also have mobile Internet access features. These terminals include: PDA, MID and UMPC devices, etc.

[0070] (3) Portable entertainment devices: These devices can display and play multimedia content. They include audio and video players, handheld game consoles, e-books, smart toys, and portable car navigation devices.

[0071] (4) Server: A device that provides computing services. The server consists of a processor, hard disk, memory, system bus, etc. The server is similar to a general computer architecture, but because it needs to provide highly reliable services, it has higher requirements in terms of processing power, stability, reliability, security, scalability, and manageability.

[0072] (5) Other electronic devices with data interaction functions.

[0073] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, i.e., they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative effort.

[0074] Through the description of the above implementation modes, those skilled in the art can clearly understand that each implementation mode can be implemented by means of software plus a necessary general hardware platform, or of course by hardware. Based on such an understanding, the above technical solution can essentially or in other words be embodied in the form of a software product that contributes to the prior art. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of each embodiment or some parts of the embodiment.

[0075] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for improving a speech synthesis system, for use with a VALL-E model, comprising: Based on the VALL-E model, multiple prediction heads are used to predict more tags in combination with advanced speculative decoding strategies and tolerance mechanisms, wherein the advanced speculative decoding strategies include an effective lossless speculative decoding method based on an autoregressive Transformer model; The inference sampling is ensured to be consistent with the VALL-E model through a verification mechanism.

2. The method according to claim 1, wherein: The advanced speculative decoding strategy includes: The sampling results are represented by the absolute position IDs of the top-3 tree nodes arranged from left to right in each layer; For a subset sparse tree of the top-3 trees, the results are concatenated step by step by level, and the concatenated results are input into an autoregressive decoder for a decoding operation, wherein an attention mask is constructed to ensure that each node can only access the information of its ancestor node.

3. The method according to claim 1, wherein: Said include: Under tree attention, we use the scratch head to predict future tokens and hope that the VALL-E model accepts a candidate path with a higher acceptance probability.

4. The method according to claim 1, wherein: The tolerance mechanism includes: when the tolerance is 2, the autoregressive head performs two polynomial samplings in the kernel sampling process.

5. The method according to claim 1, wherein: The verification mechanism for ensuring that the inference sampling is consistent with the VALL-E model includes: After comparing the sampling results of the autoregressive head in the validation phase with the results of the VADUSA head, the leftmost branch is preferentially selected among the acceptable branches of the sparse tree.

6. An electronic device, comprising: At least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the steps of the method described in any one of claims 1 to 5.

7. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images