Emotional speech generation stability control method and system based on multi-stream discrete audio tokens

Abstract

The application discloses a kind of stable control method and system based on multi-flow discrete audio token of emotional speech generation.The method constructs text stream and multiple audio streams by to-be-synthesized text, emotional text prompt, speaker reference audio and target audio token, inputs large language model or Transform model to generate audio token;Additional audio stream is constrained by multi-token prediction branch, stop loss is trained by independent stop prediction head, and stop bias is applied to audio end token during inference period;During training, replace part of audio prefix with model self-generated token, detect low-entropy repetition, minority token set and mute attraction state during inference, and apply prediction score penalty;Finally, speech waveform is recovered by audio token decoder.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence speech synthesis, and in particular to a stable control method and system for generating emotional speech based on multi-stream discrete audio tokens. Background Technology

[0002] With the development of neural audio encoding / decoding and large language models, speech generation methods based on discrete audio tokens have gradually become an important technical route for text-to-speech synthesis. These methods typically first encode speech into discrete audio tokens, then generate audio tokens based on text conditions using an autoregressive model, and finally input the generated audio tokens into an audio token decoder to recover the speech waveform. Compared with traditional Mel-spectrum-based speech synthesis methods, discrete audio token methods are more suitable for unified modeling with language models and are easier to extend to multi-speaker, multilingual, and emotion control scenarios.

[0003] However, existing discrete audio token speech generation systems still suffer from insufficient stability in scenarios involving emotional speech generation and long audio generation. Firstly, existing systems typically rely on audio end tokens or sequence end tokens to implicitly represent the stopping point, requiring the model to indirectly learn when to stop from the audio token cross-entropy loss. This approach easily leads to premature stopping, delayed stopping, or failure to stop, especially in the generation of long texts, strong emotional expressions, or multi-layered audio tokens, where drift in the stopping position can render the generated audio unusable.

[0004] Secondly, existing autoregressive training typically employs a teacher-forced training mechanism, where historical input during training primarily comes from real audio tokens, while historical input during inference comes from the model's own output. If an incorrect token is generated early in inference, subsequent states may deviate from the training distribution and accumulate errors, leading to repetitions, silences, noise, or pronunciation drift. While traditional self-sampling replacement can alleviate exposure bias in sequence prediction, it does not address the encoding / decoding layers, audio prefix block replacement, and stop control coordination in multi-stream discrete audio token speech generation.

[0005] Furthermore, multi-layered codec tokens often carry acoustic information at different levels, making single-step prediction or single audio stream prediction insufficient to adequately constrain subsequent audio trajectories. For long audio speech synthesis, the model may be reasonable in local probabilities, but globally it gradually enters a low-entropy repetitive state or a state attracted by a few audio tokens, ultimately resulting in repetitive segments, silence lock-in, or meaningless audio. Ordinary top-k, top-p, or temperature sampling mainly adjusts the sampling distribution, lacking online detection and correction mechanisms for generation state collapse.

[0006] Therefore, a stable control method is needed for multi-stream discrete audio token emotional speech generation, which reduces the difference between training and inference distributions during the training phase, explicitly controls the stopping timing and identifies collapse risks during the inference phase, thereby improving the stability of long audio and emotional speech generation. Summary of the Invention

[0007] The purpose of this invention is to address the shortcomings of existing technologies by proposing a stable control method for emotion-based speech generation based on multi-stream discrete audio tokens.

[0008] The objective of this invention is achieved through the following technical solution: a stable control method for emotion-based speech generation based on multi-stream discrete audio tokens, comprising: Construct a multi-stream input sequence containing the text stream and multiple audio streams based on the text and discrete audio tokens; The multi-stream input sequence is input into an autoregressive model to obtain the hidden state, the text token prediction probability distribution, and the main audio token prediction probability distribution; based on the hidden state, multi-token prediction is performed, and at least one additional audio token prediction probability distribution is output. The stopping probability is output by combining the hidden state and the original audio prediction score with a stopping prediction head that is independent of the audio token prediction probability distribution; A joint loss is constructed based on the audio token prediction probability distribution and the stopping probability. The parameters of the stopping prediction head, multi-token prediction and autoregressive model are trained using the joint loss. During the training process, the sampling audio prefix is ​​replaced based on the sampling probability that changes with the number of training steps. The trained model is used to generate audio tokens for multiple audio streams through autoregression. After reaching a preset minimum number of stopping steps, a stopping bias is applied to the predicted score of the audio end token for each audio stream according to the stopping probability. The collapse risk is detected based on the distribution state of the audio tokens during the generation process. When a collapse risk is detected, the predicted score of the audio tokens identified as attracting tokens is penalized. The audio tokens of the multiple audio streams are synthesized into a codec token sequence and combined with the speaker reference audio to generate the target speech waveform through the audio token decoder.

[0009] Furthermore, constructing a multi-stream input sequence comprising the text stream and the plurality of audio streams based on the text and discrete audio tokens includes: Obtain discrete audio tokens corresponding to the text to be synthesized, the emotional text cue, the speaker reference audio, and the target audio for training; encode the text to be synthesized and the emotional text cue into a text stream, divide the discrete audio tokens into multiple audio streams according to the encoding / decoding layer or token level, and construct a multi-stream input sequence containing the text stream and multiple audio streams; the mapping from the target audio encoding / decoding token to the multiple audio streams is specifically as follows: in, Indicates the target audio codec token. This indicates the end of audio. Indicates an audio fill token. This indicates that in order to make the total number of tokens available The number of fill tokens added due to integer division; Indicates the first One audio stream, Indicates the number of audio streams. Indicates the first The audio stream in the first A token for each time step. This indicates the length of each audio stream after splitting.

[0010] Furthermore, the model used for the multi-token prediction includes a decoding layer, a normalization layer, and an audio output projection layer; the additional audio prediction probability distribution is obtained step by step from the hidden state through the decoding layer and the projection layer, and the additional audio corresponds to the remaining audio streams other than the main audio.

[0011] Furthermore, the stopping prediction head includes a hidden state normalization layer, an original audio prediction score projection layer, and a multilayer perceptron, and the calculation process includes: in, Indicates a hidden state. Represents the original audio prediction score. Represents the normalization function. and Indicates projection parameters, This indicates that you should stop predicting scores. This represents the probability of stopping.

[0012] Furthermore, the joint loss specifically includes: in, Indicates total training loss. Indicates the loss of basic training. Indicates stopping the loss weight. An indicator that indicates whether stopping loss should be turned off during training of the sampled audio prefix. To stop the loss; The basic training loss is calculated using the following formula: in, Indicates the first Cross-entropy loss of each audio stream, Indicates the first The audio stream in the first A time step for the real audio token The predicted probability; Indicates the loss of basic training. An indicator of whether the text stream loss is used in training. Indicates the text stream loss. This represents the multi-token prediction loss decay coefficient; Stop loss is calculated using the following formula: in, Indicates the first The target of cessation of monitoring at each time step, This indicates whether the time step is a valid monitoring position. This indicates the end of audio. Indicates that the label should be ignored. This indicates that the positive sample weights are being stopped. This represents the probability of stopping.

[0013] Furthermore, the replacement of the sampled audio prefix based on the sampling probability that changes with the number of training steps includes: When training is enabled, the sampling probability is calculated as follows: in, The initial probability of the sampled audio prefix. This represents the probability of the audio prefix ending. This represents the current number of training calls. This represents the number of warmup steps; if the sampling probability is greater than 0, candidate positions are selected from the suffix window of valid audio tag positions, and the candidate positions are evaluated with probability. The replacement is performed. During the replacement, the system obtains the audio prediction probability distribution using the current multi-stream input in a gradient-free state, and obtains a self-generated audio token using a greedy method. The self-generated audio token is then written to the input position of the corresponding audio stream after layer offset. The multi-stream input sequence after the above sampling audio prefix replacement processing is sent to the model for forward propagation training.

[0014] Further, applying a stop bias to the audio end token prediction score of each audio stream based on the stop probability includes: The stopping probability is multiplied by the stopping bias coefficient to obtain the stopping bias, and this stopping bias is added to the audio end token prediction score of each audio stream. If any audio stream generates an audio end token, it can be determined that the audio generation has ended. No stopping bias is applied before the minimum stopping step is reached. After the minimum stopping step is reached, the stopping prediction head does not directly replace the audio token generation, but participates in the stopping decision by changing the audio end token probability.

[0015] Furthermore, the detection of collapse risk based on the audio token distribution state during the generation process includes: Calculate the top-1 token, top-1 probability, distribution entropy, and audio end token probability based on the current audio prediction score; combine the current statistics with the historical window to determine if the collapse condition is met. in, This indicates the number of distinct values ​​for the top-1 token within the most recent window. Indicates the threshold of the minority set. This represents the maximum entropy within the most recent window. Represents the entropy threshold. This indicates the number of times the advantage token has appeared in the most recent window. Indicates the threshold for the number of times the dominance is reached. This indicates a continuous state of silence at the tail. This indicates that the preset attraction token has reached a threshold number of times within the window. This indicates that the current top-1 token belongs to the preset attraction token and the current entropy is below the threshold. This indicates that the current top-1 probability has reached the peak threshold and the current entropy is below the threshold. This indicates the result of the collapse risk assessment; If a collapse risk is detected, an attraction token is determined, and a preset penalty is applied to its predicted score. After the penalty, an audio token is reselected based on the corrected predicted score.

[0016] According to another aspect of the specification, a stable control system for emotion-based speech generation based on multi-stream discrete audio tokens is also provided for implementing the method, comprising: The input and data construction module is used to obtain the discrete audio tokens corresponding to the text to be synthesized, the emotional text prompts, the speaker reference audio, and the target audio for training, and to construct the text stream and multiple audio streams; The multi-stream generation module is used to input the text stream and the multiple audio streams into a large language model or a Transformer model to obtain hidden states, text token prediction probability distributions, and audio token prediction probability distributions. A multi-token prediction module is used to output at least one additional audio token prediction probability distribution based on the hidden state and to calculate the audio prediction loss. The stop control module is used to output a stop probability based on the hidden state and the original audio prediction score, calculate the stop loss during the training period, and apply a stop bias to the audio end token prediction score during the inference period. The sampling audio prefix training module is used to replace some real audio tokens in the multiple audio streams with model-generated audio tokens according to the sampling probability that changes with the number of training steps during the training phase. The collapse-aware decoding module is used to detect collapse risk during the inference phase based on the distribution state of audio tokens during the generation process, and to impose a penalty on the prediction score of audio tokens identified as attractive tokens. The audio recovery module is used to synthesize the audio tokens of multiple generated audio streams into a codec token sequence, and combine it with the speaker reference audio to generate a target speech waveform through an audio token decoder.

[0017] According to another aspect of the specification, an emotional speech generation stabilization control device based on multi-stream discrete audio tokens is also provided, including a memory and one or more processors. The memory stores executable code, and when the processor executes the executable code, it implements the emotional speech generation stabilization control method based on multi-stream discrete audio tokens.

[0018] The beneficial effects of this invention are: 1. This invention sets up an independent stop prediction head and uses audio end tokens to construct a stop supervision target during the training period. During the inference period, the stop probability is converted into an audio end token prediction score bias, thus enabling the stop control to be transformed from implicit token prediction to explicit stop decision. 2. This invention uses sampled audio prefix training. During the training phase, some real audio prefixes are replaced with audio tokens generated by the model itself, so that the model can be exposed to the historical input states that may occur during the inference phase in advance. 3. This invention simultaneously constrains the main audio stream and the supplementary audio stream or future audio tokens through multi-token prediction branches, which is beneficial to enhancing the timing consistency between audio tokens in multi-layer codecs. 4. This invention detects the risk of collapse during the inference stage and applies a prediction score penalty to the attraction token, thereby enabling online correction of low-entropy repetition or silent attraction states during the generation process. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of the structure of a stable control system for emotion-based speech generation based on multi-stream discrete audio tokens according to the present invention. Figure 2This is a training flowchart of a stable control method for emotion-based speech generation based on multi-stream discrete audio tokens according to the present invention. Figure 3 This is a flowchart illustrating the reasoning process of a stable control method for emotion-based speech generation based on multi-stream discrete audio tokens, as described in this invention. Figure 4 This is a schematic diagram of the structure of the stop prediction head of the present invention; Figure 5 This is a flowchart of the sampling audio prefix training process of the present invention; Figure 6 This is a flowchart of the collapse sensing decoding process of the present invention; Figure 7 This is a schematic diagram of the multi-token prediction branch structure of the present invention; Figure 8 This is a schematic diagram of an emotional speech generation stabilization control device based on multi-stream discrete audio tokens according to the present invention. Detailed Implementation

[0020] The specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0021] A stable control method for emotion-based speech generation based on multi-stream discrete audio tokens is applied to speech generation systems including large language models or Transformer models. The method includes: Step 1: Obtain the text to be synthesized, the emotional text prompt, the speaker reference audio, and the discrete audio tokens pre-trained by the audio codec encoder corresponding to the target audio for training. The training process additionally includes the target text and the target audio. Step 2: Encode the text to be synthesized and the emotional text prompt into a text stream, divide the discrete audio token into multiple audio streams according to the encoding / decoding layer or token level, and construct a multi-stream input sequence containing the text stream and the multiple audio streams; The multiple audio streams comprise C audio streams obtained by hierarchically splitting the target audio codec tokens, where C is an integer greater than 1. Each audio stream includes an input identifier token, a padding token, an audio end token, and an answer identifier token, and the text stream includes a text input identifier token, a text padding token, a text end token, and a text answer identifier token. To map the target audio codec tokens into multiple audio streams suitable for parallel modeling, the original audio token sequence needs to be appended with an end marker and padded with length, and then interleaved and split according to a preset number of streams C.

[0022] In this embodiment, the padding process of the target audio codec token sequence can be represented by the following formula: in, This represents the original codec audio token. This indicates the end of audio. Indicates an audio fill token. This indicates that in order to make the total number of tokens available The number of fill tokens added when the number is divisible by an integer.

[0023] Will Interleaving is split according to the preset flow number C, the first... An audio stream can be represented as: in, Indicates the first One audio stream, Indicates the first The audio stream in the first Audio tokens at each time step, This represents the length of each audio stream after splitting. This formula is used to define the deterministic mapping relationship between multi-layer codec tokens and multi-stream audio sequences. In this way, the original one-dimensional audio token sequence can be deterministically mapped into multiple time-aligned audio streams, thereby supporting subsequent multi-stream parallel generation and hierarchical modeling.

[0024] Step 3: Input the multi-stream input sequence into the large language model or Transformer model to perform autoregressive modeling on the text stream and multiple audio streams to obtain the hidden state, text token prediction probability distribution and main audio token prediction probability distribution; The text token prediction probability distribution is naturally generated by the large language model or Transformer model during forward propagation. In audio generation tasks, since the input text is already provided as conditional completeness, text prediction constitutes information redundancy, and including it in the loss would dilute the effective gradient signal of audio prediction. Therefore, in this embodiment, the supervision target corresponding to the text token in the loss function is set as an ignore flag, so that the text token prediction probability distribution does not participate in the loss function calculation, and the model training is entirely optimized based on the audio token prediction loss and the stopping prediction loss.

[0025] It should be noted that the existence of the text token prediction probability distribution is beneficial to maintaining the text understanding ability of the large language model or the Transformer model and improving the quality of the audio token prediction probability distribution. The text stream participates in forward propagation through multi-stream fusion and receives the gradient backpropagation of the audio token prediction. It also shares the hidden states generated by the large language model or the Transformer model when processing the text to be synthesized and the sentiment text cues, and is directly used for the prediction of the audio token.

[0026] Step 4: Through the multi-token prediction branch, output at least one additional audio token prediction probability distribution based on the hidden state, and calculate the audio prediction loss based on the main audio token prediction probability distribution and the additional audio token prediction probability distribution. The multi-token prediction branch includes at least one decoding layer, a normalization layer, and an audio output projection layer corresponding to the large language model or the Transformer model; the multi-token prediction branch takes the hidden state as input and outputs an additional audio token prediction probability distribution for predicting different audio streams or future audio tokens.

[0027] The main audio token prediction probability distribution is directly generated from the output of the large language model or the Transformer model, corresponding to the main audio stream; the supplementary audio token prediction probability distribution is derived step-by-step from the hidden state by the multi-token prediction branches, corresponding to the remaining audio streams respectively. Together, they cover all audio streams, providing a prediction probability distribution for calculating the loss in each audio stream during the training phase, jointly constraining the temporal consistency of the multi-layered token structure, and providing token sampling criteria for each audio stream during the inference phase, selecting the token with the highest probability as the generated result at each step.

[0028] Step 5, using a stopping prediction head independent of the audio token prediction probability distribution, such as... Figure 4 As shown, the stopping probability is output based on the hidden state and the original audio prediction score; The stopping prediction head includes a hidden state normalization layer, an original audio prediction score projection layer, and a multilayer perceptron; wherein, the hidden state normalization layer is used to normalize the current hidden state, the original audio prediction score projection layer is used to project the original audio prediction score onto a preset dimension, and the multilayer perceptron is used to output a stopping prediction score based on the splicing result of the normalized hidden state and the projected original audio prediction score.

[0029] In this embodiment, the output of the stopping prediction head can be represented by the following formula: in, This represents a multilayer perceptron used for classification. Indicates the first The hidden state at each time step Indicates the hidden state normalization layer, semicolon This represents vector concatenation. Indicates the first The original audio prediction score at each time step. and This represents the projection parameters of the original audio prediction score, used to map the high-dimensional original audio prediction score to a preset low-dimensional feature space; Indicates the stopping probability. This represents the sigmoid function. This represents the stopping prediction score. The formula uses both the semantic hidden state and the original audio prediction probability distribution for the stopping decision.

[0030] In one formalized embodiment, the stopping prediction head can be represented as: in, , , , Indicates the parameters of the multilayer perceptron. This represents a non-linear activation function; the meanings of other variables remain the same as described above. This formula provides the complete computational chain from the hidden state and audio prediction score to the stopping probability for the stopping prediction head.

[0031] Step Six, during the training phase, such as Figure 2 As shown, the system acquires training input and constructs multi-stream input sequences according to steps one to two, performs forward propagation based on the model structure defined in steps three to five, performs sampled audio prefix training, calculates training loss, and jointly optimizes model parameters.

[0032] In the sampling audio prefix training, according to the sampling probability that changes with the number of training steps, the effective audio position is selected from the audio prefixes of the multiple audio streams, and the corresponding real audio token is replaced with the audio token generated by the speech generation system to obtain the sampling audio prefix input. The sampling probability is calculated based on the starting probability of the sampled audio prefix, the ending probability of the sampled audio prefix, and the number of warmup steps, and gradually changes from the starting probability to the ending probability with the number of training calls. Both the starting probability and the ending probability are preset hyperparameters determined through experimental tuning.

[0033] In this embodiment, the sampling probability can be calculated using the following formula: in, Indicates the first The sampling probability during the next training call. Indicates the initial probability of the sampled audio prefix. Indicates the probability of the sampling audio prefix ending. This indicates the number of warmup steps. This formula is used to gradually increase the proportion of the model's self-generated prefixes, avoiding excessive replacement in the early stages of training that could lead to unstable optimization.

[0034] The step of replacing the corresponding real audio token with an audio token generated by the speech generation system includes: if the sampling probability is greater than 0, selecting a candidate position in the suffix window of the valid audio tag position, and performing a probability-based replacement on the candidate position. During replacement, the system obtains the audio prediction probability distribution using the current multi-stream input in a gradient-free state, and preferably uses a greedy method to obtain the self-generated audio token, then writes the self-generated audio token into the input position of the corresponding audio stream after layer offset. This step does not involve gradient calculation or parameter updates, but only modifies the input sequence of subsequent training steps.

[0035] In this embodiment, the self-generated token replacement can be represented by the following formula: in, This represents a self-generated audio token obtained from the probability distribution predicted at the previous time step. Indicates the first The audio stream was in the previous time step on the first Predicted probability of each audio token A Bernoulli random variable indicating whether to perform the substitution. This indicates the replacement probability given the current number of training calls. Indicates the first The layer offset corresponding to each audio stream Indicates the original input token. This represents the replaced input token. The formula is used to probabilistically mix the model's self-generated states into the training input, thereby more closely approximating the historical input distribution during the inference phase, constructing a training prefix that closely resembles the inference state while maintaining the validity of the vocabulary hierarchy.

[0036] The system feeds the multi-stream input sequence, after the aforementioned sampled audio prefix replacement processing, into the model for forward propagation. It then combines the basic training loss and stopping loss into a total loss to update the model parameters. Depending on the training strategy, the main body of the language model, the stopping prediction head, the multi-token prediction module, or other adapted modules can be frozen or unfrozen.

[0037] In the calculation of training loss, since the input text is already provided as conditional completeness, text prediction constitutes information redundancy, and including it in the loss would dilute the effective gradient signal of audio prediction. Therefore, in this embodiment, the supervision target corresponding to the text token in the loss function is set as an ignore flag, so that the probability distribution of text token prediction does not participate in the calculation of the loss function, and the model training is entirely optimized based on the audio token prediction loss and the stopping prediction loss.

[0038] The audio prediction loss includes the cross-entropy loss of multiple audio streams, and the losses of different audio streams or different prediction branches are weighted and summed according to the attenuation coefficient.

[0039] In this embodiment, the audio prediction loss and the basic training loss can be calculated using the following formulas: in, Indicates the first Cross-entropy loss of each audio stream, Indicates the first The audio stream in the first A time step for the real audio token The predicted probability distribution; Indicates the loss of basic training. An indicator of whether the text stream loss is used in training. Indicates the text stream loss. This represents the multi-token prediction loss decay coefficient. This formula is used to normalize and merge text supervision and supervision from multiple audio branches, avoiding instability in the loss scale caused by changes in the number of branches, and enabling different audio streams or different prediction branches to participate in joint optimization with controllable weights.

[0040] The stopping supervision objective is obtained by determining whether at least one audio stream label at each time step contains an audio end token. If any audio stream label at a given time step is an audio end token, the stopping objective for that time step is 1; otherwise, it is 0. The stopping loss is a binary cross-entropy loss with positive sample weights, and it is only used for training at valid audio label positions to alleviate the problem of sparse samples at stopping positions.

[0041] In this embodiment, the stopping monitoring target and the stopping loss can be calculated using the following formula: in, Indicates the first The target of cessation of monitoring at each time step, This indicates whether the time step is a valid monitoring position. This indicates the end of audio. Indicates that the label should be ignored. This indicates that the positive sample weights are being stopped. Indicates the stopping probability. This represents the stopping loss. This formula is used to enhance the stopping supervision signal when the audio end position is sparse.

[0042] In this step, the total loss can be expressed as: in, Indicates total training loss. Indicates the loss of basic training. Indicates stopping the loss weight. This indicates whether stopping loss should be turned off during training of the sampled audio prefix. This formula is used to coordinate audio generation capability and stopping control capability within the same optimization objective.

[0043] Step seven, in the reasoning stage, such as Figure 3 As shown, the system acquires inference input and constructs a multi-stream input sequence according to steps one and two. Based on the model structure of steps three to five, it performs forward propagation and autoregressively generates audio tokens for multiple audio streams. After reaching a preset minimum stopping step count, a stopping bias is applied to the predicted scores of the audio end tokens for each audio stream according to the stopping probability. The system also detects collapse risk based on the audio token distribution state during the generation process, and penalizes the predicted scores of audio tokens identified as attracting tokens when a collapse risk is detected. The generated audio tokens from multiple audio streams are synthesized into a codec token sequence, and combined with the speaker reference audio, a target speech waveform is generated through an audio token decoder. Specifically, this includes performing stopping bias control, collapse-aware decoding, updating the generation state, and restoring the speech waveform.

[0044] Stop bias control is implemented. When the stop prediction head is enabled and the current step count reaches the preset minimum stop step count, the stop control module calculates the stop probability based on the current hidden state and the original audio prediction score. This stop probability is multiplied by a stop bias coefficient to obtain the stop bias, which is then added to the audio end token prediction score of each audio stream. If any audio stream generates an audio end token, the audio generation is considered complete. The system does not apply a stop bias before reaching the minimum stop step count; after reaching the minimum stop step count, the stop prediction head does not directly replace audio token generation but participates in the stop decision by changing the audio end token probability.

[0045] In this embodiment, the stop bias control can be expressed by the following formula: in, Indicates the first Stop bias at each time step This indicates the preset minimum number of stopping steps. Indicates the stop bias coefficient. Indicates the stopping probability. and ...

[0046] Perform collapse-aware decoding. The collapse risk detection includes: maintaining the history of the top-1 audio tokens and the history of distribution entropy within the most recent window of each audio stream, and combining at least two of the following factors to determine whether there is a collapse risk: a small set of top-1 tokens, low-entropy states, dominant tokens, a preset set of attractive tokens, and a continuous state of silence at the end.

[0047] In this embodiment, the audio distribution statistics can be calculated using the following formula: in, Indicates the first The audio stream pair The probability of an audio token. This indicates the corresponding predicted score. This indicates the top-1 audio token. This represents the distribution entropy. This formula is used to transform the generated distribution into a statistic that can be used by the collapse detection module.

[0048] The statistics within the most recent window can be represented as: in, Indicates the size of the collapse detection window. This indicates the number of different values ​​for the top-1 token within the window. This indicates the number of times the token appears most frequently within the window. Indicates token Number of times it appears in the window This represents the maximum entropy within the window. This formula is used to characterize repetitive states with low change and low entropy states.

[0049] In this embodiment, the distribution entropy and collapse risk can be calculated using the following formula: in, This indicates the number of distinct values ​​for the top-1 token within the most recent window. Indicates the threshold of the minority set. This represents the maximum entropy within the most recent window. Represents the entropy threshold. This indicates the number of times the advantage token has appeared in the most recent window. Indicates the threshold for the number of times the dominance is reached. This indicates a continuous state of silence at the tail. This indicates that the preset attraction token has reached a threshold number of times within the window. This indicates that the current top-1 token belongs to the preset attraction token and the current entropy is below the threshold. This indicates that the current top-1 probability has reached the peak threshold and the current entropy is below the threshold. This indicates the collapse risk assessment result. The formula is used to jointly detect multiple conditions, including repetition, low entropy, silence, and attracting token states, and unify them into a computable online risk assessment.

[0050] In embodiments where tail audio debugging decoding exists, the tail mute state can be calculated using the following formula: in, This represents the audio sample value within the tail window. Indicates the number of sampling points in the tail window. Indicates the silence threshold. This indicates the result of the tail-end silence determination. This formula is used to convert the tail-end energy of the generated audio into an auxiliary condition for collapse detection.

[0051] The penalty applied to the predicted score of the audio tokens identified as attraction tokens includes: after determining that there is a risk of collapse, identifying attraction tokens from the audio tokens that have appeared a threshold number of times in the most recent window, and subtracting a preset penalty value from the predicted score of attraction tokens that are neither audio end tokens nor fill tokens.

[0052] In this embodiment, the attraction token penalty can be used to correct the predicted scores of the audio end token and the attraction token using the following formula: in, Indicates the first The audio stream in the first The set of attraction tokens identified at each time step Indicates token Number of times it appears in the most recent window Indicates the threshold number of times tokens can be attracted; The original prediction score representing the audio end token. This represents the predicted score after stopping the bias. Indicates the minimum number of stopping steps. Indicates the stopping bias coefficient; Indicates the first The audio stream in the first The time step for the first The predicted score for each token. This represents the penalty value for attracting tokens. This indicates the end of audio. Indicates an audio fill token. This represents the corrected predicted score. This formula is used to suppress non-ending, non-filling attracting tokens, reducing the probability of generation entering low-change cycles.

[0053] Update the generation state. The system adds the generated audio token to the audio stream region in the generation history, and adds the text-fixed padding token to the text stream region to maintain the alignment format of the multi-stream sequence. It also updates the input, attention mask, and buffer state for the next time step. If the teacher-forced audio prefix debugging mode is enabled, the system can use the audio token corresponding to the current time step in the target audio token as the conditional input for the next step to analyze the differences between freely generated and true prefix conditions.

[0054] The audio waveform is then restored. After generation, the audio restoration module 170 interleaves or merges multiple audio streams into a one-dimensional codec token sequence according to the codec level, truncates the content after the audio end token, and replaces the padding token with a token acceptable to the audio token decoder. Subsequently, the audio restoration module 170 extracts the prompt token, acoustic features, and speaker embedding from the speaker reference audio, and generates the target speech waveform through the audio token decoder.

[0055] In this step, the process of synthesizing multi-stream audio tokens into a codec token sequence can be represented by the following formula: in, Indicates the first The audio stream in the first Audio tokens generated at each time step This represents the synthesized one-dimensional encoded / decoded token sequence. This indicates an audio token decoder. This represents the prompt token extracted from the speaker's reference audio. Indicates the acoustic characteristics of the reference audio. Indicates speaker embedding, This represents the recovered speech waveform. This formula is used to illustrate the process by which a multi-stream discrete audio token is ultimately recovered into a playable speech signal.

[0056] System Structure like Figure 1 As shown, in one embodiment, the emotion-based speech generation stability control system 100 based on multi-stream discrete audio tokens includes an input and data construction module 110, a multi-stream generation module 120, a multi-token prediction module 130, a stop control module 140, a sampled audio prefix training module 150, a collapse-aware decoding module 160, and an audio recovery module 170.

[0057] The input and data construction module 110 is used to receive sample data. The sample data may include the text to be synthesized, emotional text prompts, speaker reference audio, training target audio, and discrete audio tokens corresponding to the training target audio. The emotional text prompts may be natural language text, such as text describing emotion categories, emotion intensities, or speech styles. The speaker reference audio is used to provide the audio token decoder with speaker timbre and acoustic conditions.

[0058] The input and data construction module 110 encodes the text to be synthesized and the emotional text prompt into a text stream, and divides the discrete audio tokens into multiple audio streams according to the encoding / decoding layer or token level. Preferably, when the encoding / decoding token has C levels, the target audio token is split into C audio streams according to the levels, where C is an integer greater than 1. Each audio stream may include an audio input identifier token, an audio padding token, an audio end token, and an audio answer identifier token; the text stream may include a text input identifier token, a text padding token, a text end token, and a text answer identifier token.

[0059] In a more specific implementation, the multi-token layer offset and embedding fusion can be expressed by the following formula: in, Indicates the first The audio stream in the first The original audio token at each time step, The audio token after the layer offset. Indicates the text vocabulary offset. Indicates the step size between audio word layers. Represents a text stream token. This represents a text embedding function. This represents the audio embedding function. This represents the fused embedding fed into a large language model or Transformer model. This formula is used to make audio tokens from different codec layers distinguishable in a unified vocabulary space and to enable joint modeling of text and audio streams at the same time step.

[0060] The multi-stream generation module 120 includes a large language model or a Transformer model for autoregressive modeling of the text stream and multiple audio streams. Preferably, the multi-stream input sequence is converted into an embedding representation before entering the model, and the embeddings of the multiple streams can be obtained as the current time step input embedding through averaging, concatenation, projection, or other fusion methods. The multi-stream generation module 120 outputs the hidden state, the text token prediction probability distribution, and the main audio token prediction probability distribution.

[0061] The multi-token prediction module 130 is used to predict the distribution of additional audio tokens based on the hidden state output by the multi-stream generation module 120. The multi-token prediction module 130 may include one or more decoding layers, normalization layers, and audio output projection layers. For multi-layer codec tokens, the distribution of additional audio tokens may correspond to other audio streams besides the main audio stream; for single audio stream scenarios, the distribution of additional audio tokens may also correspond to audio tokens at several future time steps.

[0062] In a more specific implementation, the output of the multi-token prediction branch can be expressed by the following formula: in, Indicates as of the date The model's hidden state sequence at each time step. Indicates the first A multi-token prediction decoding layer, Indicates the first A normalization layer, and Indicates the first Parameters of the audio output projection layer Indicates the first The additional prediction branch in the first Predicted score at each time step This indicates that the branch is related to the first... The formula describes the predicted probability of each audio token. It illustrates how the additional audio prediction probability distribution is obtained step-by-step from the hidden states through the decoding and projection layers.

[0063] The stop control module 140 is used to learn and execute stop control. The stop control module 140 includes a stop prediction head, which comprises a hidden state normalization layer 142, an original audio prediction score projection layer 144, and a multilayer perceptron 146. The hidden state normalization layer 142 normalizes the hidden states, the original audio prediction score projection layer 144 projects the original audio prediction scores onto a preset dimension, and the multilayer perceptron 146 outputs a stop prediction score based on the concatenation of the two, and obtains the stop probability using the sigmoid function.

[0064] The sampled audio prefix training module 150 is used to construct an input that more closely resembles the inference state during the training phase. This module calculates the sampling probability based on the number of training calls, selects a suffix chunk from the valid audio tag positions, generates an audio token using the current model in a gradient-free state, and then replaces the corresponding real audio token with the generated audio token to obtain the sampled audio prefix input.

[0065] The collapse-aware decoding module 160 is used to monitor the generation process during the inference phase. This module maintains the history of the top-1 audio tokens and the history of distribution entropy within the most recent window of each audio stream, and combines a few top-1 token sets, low-entropy states, dominant tokens, a preset set of attractive tokens, and a continuous state of tail silence to determine whether there is a collapse risk. When a collapse risk is detected, the module determines the attractive tokens and applies a penalty to their predicted scores.

[0066] The audio restoration module 170 is used to resynthesize the generation tokens of multiple audio streams into a one-dimensional codec token sequence, and combine it with the speaker reference audio to generate a target speech waveform through an audio token decoder. The audio token decoder can extract the prompt token, acoustic features, and speaker embedding of the reference audio, and perform a token-to-waveform process accordingly.

[0067] Sampled audio prefix training example like Figure 5 As shown, the audio prefix training module 150 first calculates the sampling probability based on the number of training steps, and then selects a suffix chunk from the valid audio tag positions. For the candidate positions that are sampled and hit, the system uses the current model to predict the audio token for the previous position and writes the predicted audio token into the current input position. Since multi-stream audio tokens may have layer offsets, the offset relationship of the corresponding layer is preserved when writing, so that the replaced input is still within the range of the valid vocabulary.

[0068] In a preferred embodiment, the sampling method is greedy, the sampling probability gradually increases from 0 to 0.2, the warmup step count is 5000, and the chunk length is 16. The above values ​​are only one implementation method; the actual system can be adjusted according to the model size, data size, and audio length.

[0069] Collapse-sensing decoding example like Figure 6 As shown, the collapse-aware decoding module 160 calculates distribution statistics for each audio stream at each inference time step. These distribution statistics include the current top-1 audio token, top-1 probability, distribution entropy, audio end token probability, and top-k candidates. This module maintains the top-1 token history and entropy history for the most recent W time steps.

[0070] If the number of distinct values ​​of the top-1 token within the most recent window is less than or equal to the minority set threshold, or the entropy within the window is all below the entropy threshold, or the number of times the same token appears reaches the dominance threshold, or the current top-1 token belongs to a preset attractive token set and has low entropy, or the tail silence state appears consecutively and reaches the patience threshold, then the system determines that there is a risk of collapse. The system determines attractive tokens from audio tokens that appear a certain number of times within the window. For attractive tokens that are neither audio end tokens nor filling tokens, the system subtracts a preset penalty value from their predicted score.

[0071] In one formalized embodiment, the collapse risk assessment can be expressed as: in, Indicates the threshold of the minority set. Represents the entropy threshold. Indicates the threshold for the number of times the dominance is reached. This indicates that the tail is in a silent state. This indicates that the preset attraction token has reached a threshold number of times within the window. This indicates that the current top-1 token belongs to the preset attraction token and the current entropy is below the threshold. This indicates that the current top-1 probability has reached the peak threshold and the current entropy is below the threshold. This formula is used to combine multiple collapse indicators into a single Boolean risk assessment result.

[0072] After determining the collapse risk, the attraction token set and the predicted score correction can be expressed as: in, Indicates the threshold for the number of times tokens can be attracted. This represents the penalty value for attracting tokens. This represents the set of attracting tokens. This formula is used to probabilistically suppress non-ending, non-filled tokens that lead to low-change cycles.

[0073] In one formalized embodiment, for an audio token with three codec layers, it can be made The multi-token prediction loss is written as: in, Indicates the loss of the main audio stream. and This represents the loss between two additional audio streams or prediction branches. This represents the loss attenuation coefficient. This formula is used to give the specific weighted form of multi-token prediction in a three-layer codec token scenario.

[0074] In a preferred embodiment, the window size is 8, the minority set threshold is 2, the entropy threshold is 2.0, the tail silence patience value is 3, the attraction token penalty value is 1.5, and the minimum detection steps are 4. These values ​​can be adjusted according to the specific encoding / decoding vocabulary and data distribution.

[0075] Multi-token prediction implementation like Figure 7 As shown, the multi-token prediction module 130 may include multiple audio prediction branches. The main branch directly outputs the prediction probability distribution of the main audio stream from the multi-stream generation module 120; the supplementary branches output the audio prediction probability distribution of other audio streams or future time steps based on the hidden state. For audio tokens with three codec layers, the system can set up two supplementary branches to predict the two audio streams other than the main audio stream, respectively.

[0076] During training, the system calculates the cross-entropy loss for each audio stream and weights it by the decay factor. In this way, the model not only learns the next token of the current main audio stream, but also learns the joint constraints between multiple audio streams, thus providing more comprehensive acoustic trajectory supervision for long audio generation.

[0077] In one formalized embodiment, for an audio token with three codec layers, it can be made The multi-token prediction loss is written as: in, Indicates the loss of the main audio stream. and This represents the loss between two additional audio streams or prediction branches. This represents the loss attenuation coefficient. This formula is used to give the specific weighted form of multi-token prediction in a three-layer codec token scenario.

[0078] Alternative implementation methods In another embodiment, the multi-token prediction module 130 can be used in a single audio stream model, where additional branches predict audio tokens for future time steps, rather than audio tokens from different codec layers. In another embodiment, the collapse-aware decoding module 160 can be enabled only on debug samples, long audio samples, or inputs labeled as difficult samples, or it can be enabled on all inference samples. In another embodiment, the audio token decoder can be any decoder capable of recovering discrete audio tokens into speech waveforms, and speaker reference audio can be used to extract one or more conditions from prompt tokens, acoustic features, and speaker embeddings. In another embodiment, the emotional text prompt can be directly input by the user or mapped from preset emotional tags to natural language prompts.

[0079] The following describes the experimental process of the method according to the embodiments of the present invention.

[0080] The experiments were conducted on a single NVIDIA A6000 GPU and Linux operating system, using Python 3.10 as the primary development language and Conda virtual environment for dependency management. The deep learning framework used was PyTorch 2.4.1, audio processing used torchaudio 2.4.1 and librosa 0.10.2, Transformer model-related modules used transformers 4.43.4, and frameworks for distributed and high-performance training included DeepSpeed ​​0.14.2, accelerate 0.34.2, and PyTorch Lightning 2.6.1. During the experiments, datasets 3.3.2 was used for data loading and management, and tensorboard 2.14.0 and wandb 0.25.1 were used for training log recording and metric visualization. The inference and deployment phases used ONNX Runtime GPU 1.16.0 and triton 3.0.0 for inference acceleration. Audio encoding and decoding related processing is implemented using libraries such as soundfile, ffmpy, and kaldiio.

[0081] In the experiment, Qwen2.5-1.5B was used as the base language model, with a hidden dimension of 1536. The audio token decoder used the CosyVoice decoding module. The target audio was encoded into multi-layer discrete codec audio tokens and split into multiple parallel audio streams for joint generation. The number of codec audio layers was set to 3, the total audio vocabulary size was set to 4160, and the total vocabulary size was set to 156160. Hydra configuration was used to manage parameters for both training and inference. The AdamW optimizer was used during model training, combined with a cosine decay learning rate scheduling strategy. The batch size during training was dynamically set according to GPU memory, and gradient pruning was used to improve training stability.

[0082] During the inference phase, this embodiment sets the maximum generated length to 256, the text repetition penalty coefficient to 1.2, the audio repetition penalty coefficient to 1.2, and uses greedy decoding to generate the target audio token, where top_p is set to 1.0, top_k to 0, and temperature to 1.0. The generated audio sampling rate is set to 22050Hz.

[0083] The minimum stopping step size for the stopping prediction head is set to 48, and the stopping bias coefficient is set to 4.0. The collapse detection window for the collapse-aware decoding mechanism is 8, the minority token set threshold is 2, the entropy threshold is 2.0, the tail silence tolerance length is 3, the attracting token penalty coefficient is 1.5, and the minimum effective step size for the attracting penalty is 4.

[0084] This embodiment utilizes the publicly available emotional speech dataset EmoVoice-DB, a high-quality emotional speech dataset built for emotional speech synthesis tasks. It contains text-speech paired samples with text annotations, emotion category annotations, and expressive speech, comprising over 40 hours of English emotional speech data covering various emotional states. The dataset contains 66,300 data entries, with 10,458 entries for anger, 9,807 for happiness, 9,522 for sadness, 9,216 for surprise, 8,883 for embarrassment, 8,850 for embarrassment, and 9,564 for neutrality. During the testing phase, the model is validated using the 700 test entries officially defined by the EmoVoice dataset to evaluate its performance in long speech generation, emotional expression, and generation stability.

[0085] To verify the effectiveness of this invention in terms of long speech generation stability, emotional expression, and collapse suppression, several representative speech generation models were selected as comparison methods, and a unified experimental evaluation was conducted on the EmoVoice dataset test set. All models generated target speech under the same test data and unified inference conditions, and subjective naturalness, speech content consistency, and emotional expression consistency were selected as the main evaluation indicators. Among them, UTMOS was used to evaluate the naturalness and sound quality of the generated speech, and the higher the value, the closer the generated speech is to the real human voice; WER (Word Error Rate) was used to evaluate the content consistency between the generated speech and the target text, and the lower the value, the lower the speech content error rate; EmotionSimilarity was used to evaluate the emotional expression consistency between the generated speech and the reference speech. In this embodiment, the Emo2Vec-Large emotional representation model was used to extract the emotional embedding, and the cosine similarity between the generated speech and the reference speech was calculated. The higher the value, the closer the emotional expression is to the target speech. The experimental results are shown in Table 1.

[0086] Table 1 Comparison of the performance of the method of the present invention and existing representative speech generation models on the EmoVoice-DB test set. As shown in the table, the method of this invention achieved the highest result in the emotional consistency index, reaching 0.9157, which is higher than existing methods such as EmoVoice (1.5B), CosyVoice, PromptStyle, and PromptTTS. This indicates that the method of this invention can more accurately learn and maintain the emotional expression features in the target speech, and has higher consistency with the reference speech in terms of emotional intensity, prosodic variation, and emotional tone. In terms of speech naturalness, the method of this invention achieved a UTMOS index of 4.40, which is at a high level compared with existing high-performance models, only slightly lower than CosyVoice2's 4.42, but better than most comparative models overall. This shows that the method of this invention does not significantly lose speech naturalness and sound quality while improving generation stability. In terms of content consistency, the method of this invention achieved a WER index of 2.57, which is significantly better than models such as CosyVoice, CosyVoice2, and PromptStyle. The PromptStyle model achieved a WER of 15.51, indicating its susceptibility to text deviation, repetition, or anomalous generation in long speech or complex emotional scenarios. In contrast, the method described in this invention effectively reduces error accumulation and low-entropy collapse during the generation process through a self-generated prefix substitution mechanism, a stopping control mechanism, and a collapse risk detection mechanism, thereby improving the stability and text consistency of long speech generation. Compared to the basic EmoVoice model, this invention further improves EmotionSimilarity while maintaining a lower WER, demonstrating that the proposed multi-stream autoregressive generation control mechanism not only enhances speech generation stability but also improves emotional expression, making the generated speech closer to the real target speech in terms of semantic content, prosodic variation, and emotional expression.

[0087] Corresponding to the aforementioned embodiment of the emotional speech generation stabilization control method based on multi-stream discrete audio tokens, the present invention also provides an embodiment of an emotional speech generation stabilization control device based on multi-stream discrete audio tokens.

[0088] See Figure 8 The present invention provides an emotional speech generation stabilization control device based on multi-stream discrete audio tokens, comprising a memory and one or more processors. The memory stores executable code, and when the processor executes the executable code, it is used to implement an emotional speech generation stabilization control method based on multi-stream discrete audio tokens as described in the above embodiments.

[0089] The embodiment of the emotion-based speech generation stabilization control device based on multi-stream discrete audio tokens provided by this invention can be applied to any device with data processing capabilities, such as a computer. The device embodiment can be implemented in software, hardware, or a combination of both. Taking software implementation as an example, as a logical device, it is formed by the processor of any data-processing device loading the corresponding computer program instructions from non-volatile memory into memory for execution. From a hardware perspective, such as... Figure 8 The diagram shown is a hardware structure diagram of any device with data processing capabilities, which is a stable control device for emotion-based speech generation based on multi-stream discrete audio tokens provided by the present invention. (Except for...) Figure 8 In addition to the processor, memory, network interface, and non-volatile memory shown, any data processing device in the embodiment may also include other hardware depending on the actual function of the data processing device, which will not be described in detail here.

[0090] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0091] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the present invention according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0092] This invention also provides a computer-readable storage medium storing a program thereon, which, when executed by a processor, implements a stable control method for emotion speech generation based on multi-stream discrete audio tokens as described in the above embodiments.

[0093] The computer-readable storage medium can be an internal storage unit of any data processing device described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be an external storage device of any data processing device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units and external storage devices of any data processing device. The computer-readable storage medium is used to store the computer program and other programs and data required by the data processing device, and can also be used to temporarily store data that has been output or will be output.

[0094] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the aforementioned method for stable control of emotion-based speech generation based on multi-stream discrete audio tokens.

[0095] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of this application are indicated by the claims.

[0096] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this application. This application is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A stable control method for emotion-based speech generation based on multi-stream discrete audio tokens, characterized in that, include: Construct a multi-stream input sequence containing the text stream and multiple audio streams based on the text and discrete audio tokens; The multi-stream input sequence is input into an autoregressive model to obtain the hidden state, the text token prediction probability distribution, and the main audio token prediction probability distribution; based on the hidden state, multi-token prediction is performed, and at least one additional audio token prediction probability distribution is output. The stopping probability is output by combining the hidden state and the original audio prediction score with a stopping prediction head that is independent of the audio token prediction probability distribution; A joint loss is constructed based on the audio token prediction probability distribution and the stopping probability. The parameters of the stopping prediction head, multi-token prediction and autoregressive model are trained using the joint loss. During the training process, the sampling audio prefix is ​​replaced based on the sampling probability that changes with the number of training steps. The trained model is used to generate audio tokens for multiple audio streams through autoregression. After reaching a preset minimum number of stopping steps, a stopping bias is applied to the predicted score of the audio end token for each audio stream according to the stopping probability. The collapse risk is detected based on the distribution state of the audio tokens during the generation process. When a collapse risk is detected, the predicted score of the audio tokens identified as attracting tokens is penalized. The audio tokens of the multiple audio streams are synthesized into a codec token sequence and combined with the speaker reference audio to generate the target speech waveform through the audio token decoder.

2. The method for stable control of emotion-based speech generation based on multi-stream discrete audio tokens according to claim 1, characterized in that, The construction of a multi-stream input sequence containing the text stream and the multiple audio streams based on the text and discrete audio tokens includes: Obtain discrete audio tokens corresponding to the text to be synthesized, the emotional text cue, the speaker reference audio, and the target audio for training; encode the text to be synthesized and the emotional text cue into a text stream, divide the discrete audio tokens into multiple audio streams according to the encoding / decoding layer or token level, and construct a multi-stream input sequence containing the text stream and multiple audio streams; the mapping from the target audio encoding / decoding token to the multiple audio streams is specifically as follows: ; ； ； in, Indicates the target audio codec token. This indicates the end of audio. Indicates an audio fill token. This indicates that in order to make the total number of tokens available The number of fill tokens added due to integer division; Indicates the first One audio stream, Indicates the number of audio streams. Indicates the first The audio stream in the first A token for each time step. This indicates the length of each audio stream after splitting.

3. The method for stable control of emotion-based speech generation based on multi-stream discrete audio tokens according to claim 1, characterized in that, The model used for multi-token prediction includes a decoding layer, a normalization layer, and an audio output projection layer; the additional audio prediction probability distribution is obtained step by step from the hidden state through the decoding layer and the projection layer, and the additional audio corresponds to the remaining audio streams other than the main audio.

4. The method for stable control of emotion-based speech generation based on multi-stream discrete audio tokens according to claim 1, characterized in that, The stopping prediction head includes a hidden state normalization layer, an original audio prediction score projection layer, and a multilayer perceptron. The calculation process includes: ; ； in, Indicates a hidden state. Represents the original audio prediction score. Represents the normalization function. and Indicates projection parameters, This indicates that you should stop predicting scores. This represents the probability of stopping.

5. The method for stable control of emotion-based speech generation based on multi-stream discrete audio tokens according to claim 1, characterized in that, The joint loss specifically includes: ; in, Indicates total training loss. Indicates the loss of basic training. Indicates stopping the loss weight. An indicator that indicates whether stopping loss should be turned off during training of the sampled audio prefix. To stop the loss; The basic training loss is calculated using the following formula: ； ； in, Indicates the first Cross-entropy loss of each audio stream, Indicates the first The audio stream in the first A time step for the real audio token The predicted probability; Indicates the loss of basic training. An indicator of whether the text stream loss is used in training. Indicates the text stream loss. This represents the multi-token prediction loss decay coefficient; Stop loss is calculated using the following formula: ； ； in, Indicates the first The target to be monitored at each time step. This indicates whether the time step is a valid monitoring position. This indicates the end of audio. Indicates that the label should be ignored. This indicates that the positive sample weights are stopped. This represents the probability of stopping.

6. The method for stable control of emotion-based speech generation based on multi-stream discrete audio tokens according to claim 1, characterized in that, The replacement of the sampled audio prefix based on the sampling probability that changes with the number of training steps includes: When training is enabled, the sampling probability is calculated as follows: ； in, The initial probability of the sampled audio prefix. This represents the probability of the audio prefix ending. This represents the current number of training calls. This represents the number of warmup steps; if the sampling probability is greater than 0, candidate positions are selected from the suffix window of valid audio tag positions, and the candidate positions are evaluated with probability. The replacement is performed. During the replacement, the system obtains the audio prediction probability distribution using the current multi-stream input in a gradient-free state, and obtains a self-generated audio token using a greedy method. The self-generated audio token is then written to the input position of the corresponding audio stream after layer offset. The multi-stream input sequence after the above sampling audio prefix replacement processing is sent to the model for forward propagation training.

7. The method for stable control of emotion-based speech generation based on multi-stream discrete audio tokens according to claim 1, characterized in that, Applying a stop bias to the audio end token prediction score of each audio stream based on the stop probability includes: The stopping probability is multiplied by the stopping bias coefficient to obtain the stopping bias, and this stopping bias is added to the audio end token prediction score of each audio stream. If any audio stream generates an audio end token, it can be determined that the audio generation has ended. No stopping bias is applied before the minimum stopping step is reached. After the minimum stopping step is reached, the stopping prediction head does not directly replace the audio token generation, but participates in the stopping decision by changing the audio end token probability.

8. The method for stable control of emotion-based speech generation based on multi-stream discrete audio tokens according to claim 1, characterized in that, The method of detecting collapse risk based on the audio token distribution status during the generation process includes: Calculate the top-1 token, top-1 probability, distribution entropy, and audio end token probability based on the current audio prediction score; combine the current statistics with the historical window to determine if the collapse condition is met. ； in, This indicates the number of distinct values ​​for the top-1 token within the most recent window. Indicates the threshold of the minority set. This represents the maximum entropy within the most recent window. Represents the entropy threshold. This indicates the number of times the advantage token has appeared in the most recent window. Indicates the threshold for the number of times the dominance is reached. This indicates a continuous state of silence at the tail. This indicates that the preset attraction token has reached a threshold number of times within the window. This indicates that the current top-1 token belongs to the preset attraction token and the current entropy is below the threshold. This indicates that the current top-1 probability has reached the peak threshold and the current entropy is below the threshold. This indicates the result of the collapse risk assessment; If a collapse risk is detected, an attraction token is determined, and a preset penalty is applied to its predicted score. After the penalty, an audio token is reselected based on the corrected predicted score.

9. A stable control system for emotion-based speech generation based on multi-stream discrete audio tokens, used to implement the method described in any one of claims 1-8, characterized in that, include: The input and data construction module is used to obtain the discrete audio tokens corresponding to the text to be synthesized, the emotional text prompts, the speaker reference audio, and the target audio for training, and to construct the text stream and multiple audio streams; The multi-stream generation module is used to input the text stream and the multiple audio streams into a large language model or a Transformer model to obtain hidden states, text token prediction probability distributions, and audio token prediction probability distributions. A multi-token prediction module is used to output at least one additional audio token prediction probability distribution based on the hidden state and to calculate the audio prediction loss; The stop control module is used to output a stop probability based on the hidden state and the original audio prediction score, calculate the stop loss during the training period, and apply a stop bias to the audio end token prediction score during the inference period. The sampling audio prefix training module is used to replace some real audio tokens in the multiple audio streams with model-generated audio tokens according to the sampling probability that changes with the number of training steps during the training phase. The collapse-aware decoding module is used to detect collapse risk during the inference phase based on the distribution state of audio tokens during the generation process, and to impose a penalty on the prediction score of audio tokens identified as attractive tokens. The audio recovery module is used to synthesize the audio tokens of multiple generated audio streams into a codec token sequence, and combine it with the speaker reference audio to generate a target speech waveform through an audio token decoder.

10. A stable control device for emotion-based speech generation based on multi-stream discrete audio tokens, comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that, When the processor executes the executable code, it implements a stable control method for emotional speech generation based on multi-stream discrete audio tokens as described in any one of claims 1-8.

Images