Voice large model training and reasoning architecture optimization method
Through the combination of autoregressive generation, reinforcement learning and adaptive potential representation learning, the speech big model training and inference architecture is optimized, and the problem of scarcity of labeled data in low-resource languages is solved, the quality and efficiency of speech generation is improved, and it is suitable for low-resource languages and scarce data set scenarios.
Patent Information
- Application Number
- CN202510623449.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-15
- Publication Date
- 2025-08-15
AI Technical Summary
In speech synthesis tasks in low-resource languages or dialects, the scarcity of labeled data leads to limited model training effects, and it is difficult for the existing technology to effectively use a small amount of labeled data to improve the generalization ability of the model and the generated speech quality.
The autoregressive generation method is used for training, combining reinforcement learning and adaptive potential representation learning, and through token-by-token sampling and mixed pseudo-label prompts, Win-Lose Pair is built, DPO method is introduced, model generation process is optimized, and reweighted label smoothing loss and multi-stage training framework are adopted to reduce dependence on manual design loss functions.
Under limited data conditions, the quality and consistency of speech generation is improved, the risk of overfitting is reduced, the generation efficiency and stability is improved, and it is suitable for low-resource languages and scarce data set scenarios. The generated speech is more natural and smoother, and the dependence on large amounts of labeled data is reduced.
Smart Images
Figure CN120496497A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of large language model optimization, and in particular to a method for optimizing a large speech model training and inference architecture. Background Art
[0002] The text analysis and encoding components of TTS are crucial to speech generation quality. The primary task of text analysis is to transform raw text information into an internal representation suitable for subsequent speech synthesis. During this process, the text's grammar, intonation, and prosody must be effectively extracted to ensure the naturalness and fluency of subsequent speech synthesis.
[0003] The lack of labeled data is a common problem in training TTS models. This is especially true for speech synthesis tasks in low-resource languages or dialects, where the scarcity of labeled data further limits model training effectiveness. To overcome this bottleneck, a large amount of labeled data is typically required to ensure the model can learn high-quality speech generation capabilities. However, obtaining high-quality labeled data is often labor-intensive, costly, and time-consuming. In this context, making full use of the limited amount of labeled data, or improving the model's generalization capabilities through techniques such as unsupervised learning, semi-supervised learning, and data augmentation, has become a key area of current TTS research. Summary of the Invention
[0004] The purpose of the present invention is to provide a method for optimizing the training and inference architecture of a large speech model, which can improve the efficiency of the training process and reduce the risk of overfitting; improve the quality and consistency of speech generated by the model; and perform more effective learning under limited data conditions, thereby improving generation effects and training efficiency.
[0005] The technical solution adopted by the method for optimizing a large speech model training and reasoning architecture disclosed in the present invention is:
[0006] A method for training and optimizing a large speech model and inference architecture includes the following steps:
[0007] Autoregressive generation: During the training process, an autoregressive approach is used for generation. In the first stage, the model is trained using a traditional teacher-forcing approach: complete text and voice prompts are used as input to generate the first round of prediction sequences. In subsequent stages, the model gradually blends its own predictions with the original ground-truth prompts to serve as pseudo-label prompts for the next stage. This blending process is iteratively performed using token-by-token sampling.
[0008] Reinforcement Learning Algorithm: This framework improves the quality and consistency of speech by strengthening the consistency between the training and inference processes. This framework not only relies on external text prompts to generate speech, but also allows the generated speech itself to become a new prompt, forming a self-consistency prompt mechanism, thereby better capturing contextual information and generating high-quality speech.
[0009] As a preferred solution, each time the current token is generated, the generation result of the previous token is used as a conditional input to ensure the sequentiality and dependency of the model generation process. During the training process, the model simulates the generation logic during inference, so that it can better restore natural and fluent speech during the inference stage.
[0010] As a preferred solution, an adaptive latent representation learning strategy is used to process output tokens during training. This strategy can accurately identify unnatural early termination phenomena. Once such anomalies are detected, the decoding process will stop immediately and backpropagation will be triggered immediately. This mechanism can dynamically adjust the decoding depth according to the output quality, thereby improving training efficiency while enhancing the stability of the generation process.
[0011] As a preferred solution, in order to further improve the generation capability of the model, the DPO method is introduced, combined with the reinforcement learning framework, to optimize the decision-making in the model generation process through a multi-stage reward mechanism. The specific optimization process is as follows:
[0012] Constructing Win-Lose Pairs: During the generation process, a win-lose pair can be constructed by comparing the differences between the generated speech and the real speech. The goal is to continuously optimize the model by comparing the quality of the model's generation results with the target speech.
[0013] Adaptive mechanism of reinforcement learning: The introduction of reinforcement learning enables the model to adjust its generation strategy based on feedback from win-lose pairs, thereby optimizing the speech generation process. By learning the win or lose judgment results of each generation, the model can gradually grasp which features and strategies can make the generated speech closer to real speech, thereby improving the quality and consistency of speech generation.
[0014] As a preferred solution for model optimization, a reweighted label smoothing loss is used as the training objective. The final loss is normalized based on the batch size or sequence length to ensure the stability and comparability of the training process. Based on this, a multi-stage training framework is proposed. In the first stage, the model is trained using full teacher forcing, that is, all inputs are real speech tokens.
[0015] Starting from the second stage, a sampling-based hybrid strategy was introduced: a variable-length segment is randomly sampled from the speech token predicted by the model and used to replace the corresponding part of the ground-truth speech prompt, while the text prompt remains unchanged. The hybrid prompt is used as the input for the next round of training, so that the model gradually adapts to the disturbance caused by its own prediction. The hybrid prompt will be used in the next round of training, allowing the model to gradually adapt to the real reasoning scenarios that continue to be generated based on some historical prediction results.
[0016] The beneficial effects of the method for optimizing a large speech model training and inference architecture disclosed in the present invention are: better capturing the dependencies between contexts through autoregressive generation, making speech synthesis more natural and fluent in intonation, rhythm and pronunciation; combining the large-scale parallel processing capabilities of non-autoregressive generation methods, greatly improving the efficiency of speech generation while ensuring the stability of generated speech; in low-resource language or scarce data set scenarios, through small sample learning and transfer learning technology, significantly improving the quality of speech generation; through the direct preference optimization method in reinforcement learning, eliminating the reliance on manually designed loss functions in traditional TTS systems. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] Figure 1 This is a framework flow chart of a method for optimizing a large speech model training and inference architecture according to the present invention.
[0018] Figure 2 This is a DPO reinforcement learning optimization diagram of a speech large model training and inference architecture optimization method of the present invention.
[0019] Figure 3 This is a training-inference consistency research framework diagram of a large speech model training and inference architecture optimization method of the present invention.
[0020] Figure 4 This is a MOS evaluation result diagram of a large speech model training and inference architecture optimization method of the present invention. DETAILED DESCRIPTION
[0021] The present invention will be further described and explained below in conjunction with specific embodiments and accompanying drawings:
[0022] Please refer to Figure 1 A method for optimizing a large speech model training and inference architecture includes the following steps:
[0023] Autoregressive generation: Autoregressive generation is used during training. In the first stage, the model is trained using a traditional teacher-forcing method: complete text prompts and speech prompts are used as input to generate the first round of prediction sequences (denoted as seq1). This stage provides clear supervision signals, which helps the model converge stably in the early stages of training and lays a good foundation.
[0024] After entering the subsequent stage, the model gradually mixes its own generated prediction results (such as seq1) with the original ground-truth prompts and uses them as pseudo-ground truth prompts in the next stage. This mixing process is iterated by token-by-token sampling. This strategy enables the model to smoothly transition during training, gradually migrating from full teacher forcing to free running, and improving the robustness of the model to its own predictions.
[0025] Each time a token is generated, the previous token's generation result is used as a conditional input, ensuring the sequential and interdependent nature of the model generation process. During training, the model simulates the generation logic used during inference, enabling better reproduction of natural, fluent speech during inference. By employing an autoregressive training strategy, each training iteration of the model strictly follows a step-by-step generation process. While this approach increases training time compared to traditional non-autoregressive strategies, it significantly improves generation performance. The model is able to learn closer dependencies between contexts, resulting in more natural and fluent generated speech.
[0026] Traditional autoregressive model generation frameworks maintain consistency well over short periods of time, but often lose contextual consistency for long texts or complex conversations. This is because each time speech is generated, it relies solely on initial external cues, which can only provide feedback within a limited context and lack inherent consistency feedback. Generated speech, on the other hand, can naturally reflect the current state of reasoning and carry more contextual information than external cues.
[0027] During training, an adaptive latent representation learning strategy is used to process output tokens. This strategy can accurately identify unnatural early termination phenomena. Once such anomalies are detected, such as premature generation of end characters (EOS), the decoding process will be stopped immediately and backpropagation will be triggered immediately. This mechanism can dynamically adjust the decoding depth according to the output quality, thereby improving training efficiency while enhancing the stability of the generation process.
[0028] Please refer to Figure 2 Reinforcement Learning Algorithm: This framework improves the quality and consistency of speech by strengthening the consistency between the training process and the inference process. Specifically, the framework not only relies on external text prompts to generate speech, but also allows the generated speech itself to become a new prompt, forming a self-consistency prompt mechanism, thereby better capturing contextual information and generating high-quality speech.
[0029] In order to further improve the generation capability of the model, the DPO method is introduced, combined with the reinforcement learning framework, and a multi-stage reward mechanism is used to optimize the decision-making in the model generation process. The specific optimization process is as follows:
[0030] Constructing Win-Lose Pairs: During the generation process, a win-lose pair can be constructed by comparing the differences between the generated speech and the real speech. The goal is to continuously optimize the model by comparing the quality of the model's generation results with the target speech.
[0031] Adaptive mechanism of reinforcement learning: The introduction of reinforcement learning enables the model to adjust its generation strategy based on feedback from win-lose pairs, thereby optimizing the speech generation process. By learning the win or lose judgment results of each generation, the model can gradually grasp which features and strategies can make the generated speech closer to real speech, thereby improving the quality and consistency of speech generation.
[0032] In terms of model optimization, a reweighted label smoothing loss is used as the training objective. The final loss is normalized according to the batch size or sequence length to ensure the stability and comparability of the training process.
[0033] Based on this, a multi-stage training framework is proposed. In the first stage, the model is trained using full teacher forcing, that is, all inputs use real speech tokens.
[0034] Starting from the second stage, a sampling-based hybrid strategy was introduced: a variable-length segment is randomly sampled from the speech token predicted by the model and used to replace the corresponding part of the ground-truth speech prompt, while the text prompt remains unchanged. The hybrid prompt is used as the input for the next round of training, so that the model gradually adapts to the disturbance caused by its own prediction. The hybrid prompt will be used in the next round of training, allowing the model to gradually adapt to the real reasoning scenarios that continue to be generated based on some historical prediction results.
[0035] For example: Reference Figure 3 .
[0036] The original ground-truth speech token is [A,B,C,D,E,F,G]
[0037] The model predicts [A',B',C',D',E',F',G']
[0038] In stage 2 training, [C',D',E'] may be randomly selected to replace the real [C,D,E], and the mixed input becomes [A,B,C',D',E',F,G]
[0039] Through this iterative training process, the model dynamically adjusts its generation strategy based on the results of the previous generation, thereby optimizing the semantic coherence of speech. The TTS large model framework enables the TTS large model to consistently produce high-quality speech output that meets expectations when faced with complex contexts and tasks.
[0040] To verify the effectiveness of the proposed hybrid generation framework, we conducted empirical experiments based on the typical TTS model CosyVoice. In terms of training scheduling strategies, we implemented and compared three token replacement strategies commonly used in existing work, including:
[0041] Linear Decay
[0042] Exponential Decay
[0043] Inverse Sigmoid Decay
[0044] These strategies are used to control the replacement ratio between ground-truth tokens and model prediction tokens. Figure 4 ,The experimental results show that the exponential decay strategy shows the most ,stable performance during the training process.
[0045] The present invention provides a method for training a large speech model and optimizing the inference architecture.
[0046] 1) Improve the naturalness and fluency of speech generation
[0047] Effect: Autoregressive generation is used to better capture the dependencies between contexts, making speech synthesis more natural and fluent in intonation, rhythm, and pronunciation.
[0048] Advantages: It avoids the speech discontinuity or unsmoothness problems that may be caused by non-autoregressive generation in traditional TTS models.
[0049] Reinforcement learning directly optimizes the auditory quality of generated speech, making it more in line with human perceptual preferences.
[0050] (2) Improve the efficiency and stability of speech generation
[0051] Effect: Combining the large-scale parallel processing capabilities of non-autoregressive generation methods, the efficiency of speech generation is greatly improved while ensuring the stability of the generated speech.
[0052] Advantages: While maintaining high-quality speech output, it significantly reduces inference latency and is suitable for real-time speech generation scenarios.
[0053] It solves the problem of cumulative error that may occur in traditional autoregressive generation when generating long sequences.
[0054] (3) Enhance adaptability in low-resource environments
[0055] Results: In low-resource language or scarce dataset scenarios, the quality of speech generation is significantly improved through small sample learning and transfer learning techniques.
[0056] Advantages: Reduces dependence on large amounts of labeled data and can generate high-quality speech even in languages or domains where data is scarce.
[0057] It provides efficient technical support for scenarios such as minority language protection and dialect inheritance.
[0058] (4) Directly optimize the generated quality and reduce the reliance on manual tuning
[0059] Effect: Through the direct preference optimization (DPO) method in reinforcement learning, the reliance on manually designed loss functions in traditional TTS systems is eliminated.
[0060] Advantages: The model can learn from samples a generation pattern that is more in line with human auditory preferences, further improving the intelligibility and naturalness of speech. The optimization process is more efficient, significantly reducing the cost of manual parameter adjustment and trial and error.
[0061] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the scope of protection of the present invention. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the essence and scope of the technical solutions of the present invention.
Claims
1. A method for training and optimizing a large speech model and inference architecture, characterized in that: The following steps are involved: Autoregressive generation: During the training process, an autoregressive approach is used for generation. In the first stage, the model is trained using a traditional teacher-forcing approach: complete text and voice prompts are used as input to generate the first round of prediction sequences. In subsequent stages, the model gradually blends its own predictions with the original ground-truth prompts to serve as pseudo-label prompts for the next stage. This blending process is iteratively performed using token-by-token sampling. Reinforcement Learning Algorithm: This framework improves the quality and consistency of speech by strengthening the consistency between the training and inference processes. This framework not only relies on external text prompts to generate speech, but also allows the generated speech itself to become a new prompt, forming a self-consistency prompt mechanism, thereby better capturing contextual information and generating high-quality speech.
2. A method for optimizing a large speech model training and reasoning architecture according to claim 1, characterized in that: Each time the current token is generated, the generation result of the previous token is used as a conditional input to ensure the sequentiality and dependency of the model generation process. During the training process, the model simulates the generation logic during inference, so that it can better restore natural and fluent speech during the inference stage.
3. A method for optimizing a large speech model training and reasoning architecture according to claim 2, characterized in that: During training, an adaptive latent representation learning strategy is used to process output tokens. This strategy can accurately identify unnatural early termination phenomena. Once such anomalies are detected, the decoding process will be stopped immediately and backpropagation will be triggered immediately. This mechanism can dynamically adjust the decoding depth according to the output quality, thereby improving training efficiency while enhancing the stability of the generation process.
4. A method for optimizing a large speech model training and reasoning architecture according to claim 1, characterized in that: In order to further improve the generation capability of the model, the DPO method is introduced, combined with the reinforcement learning framework, and a multi-stage reward mechanism is used to optimize the decision-making in the model generation process. The specific optimization process is as follows: Constructing Win-Lose Pairs: During the generation process, a win-lose pair can be constructed by comparing the differences between the generated speech and the real speech. The goal is to continuously optimize the model by comparing the quality of the model's generation results with the target speech. Adaptive mechanism of reinforcement learning: The introduction of reinforcement learning enables the model to adjust its generation strategy based on feedback from win-lose pairs, thereby optimizing the speech generation process. By learning the win or lose judgment results of each generation, the model can gradually grasp which features and strategies can make the generated speech closer to real speech, thereby improving the quality and consistency of speech generation.
5. A method for optimizing a large speech model training and reasoning architecture according to claim 4, characterized in that: In terms of model optimization, a reweighted label smoothing loss is used as the training objective. The final loss is normalized according to the batch size or sequence length to ensure the stability and comparability of the training process. On this basis, a multi-stage training framework is proposed. In the first stage, the model is trained using full teacher forcing, that is, all inputs use real speech tokens. Starting from the second stage, a sampling-based hybrid strategy was introduced: a variable-length segment is randomly sampled from the speech token predicted by the model and used to replace the corresponding part of the ground-truth speech prompt, while the text prompt remains unchanged. The hybrid prompt is used as the input for the next round of training, so that the model gradually adapts to the disturbance caused by its own prediction. The hybrid prompt will be used in the next round of training, allowing the model to gradually adapt to the real reasoning scenarios that continue to be generated based on some historical prediction results.