An AI audio synthesis optimization method based on deep reinforcement learning

Abstract

The application discloses an AI audio synthesis optimization method based on deep reinforcement learning, comprising the following steps: collecting training corpus data; using the training corpus data to perform supervised training on an improved VITS model to obtain a baseline improved VITS model; processing text data and extracting target feature data related to sentiment annotation data and style annotation data; setting a controllable parameter interface; calling the controllable parameter interface to generate synthesized speech data; evaluating the synthesized speech data to obtain reward signal data including sentiment consistency reward, style similarity reward and prosody naturalness reward; and using an improved GRPO algorithm to update the strategy of the baseline improved VITS model. The application realizes the effect of enhancing the expression of synthesized speech in emotion and style control while maintaining the naturalness of speech synthesis, and can be applied to intelligent voice interaction, virtual person broadcasting and multi-scene voice generation tasks.

Description

Technical Field

[0001] This invention relates to the fields of speech signal processing and artificial intelligence technology, and in particular to an AI audio synthesis optimization method based on deep reinforcement learning. Background Technology

[0002] With the rapid development of applications such as human-computer interaction, virtual reality, and intelligent voice assistants, artificial intelligence audio synthesis technology has gradually become a core component of multimodal intelligent systems. Most existing speech synthesis methods rely on deep neural networks to achieve end-to-end modeling from text to speech. Among these, the Tacotron series models based on sequence-to-sequence frameworks, acoustic modeling methods based on diffusion and autoregression, and the VITS model based on variational autoencoders and generative adversarial networks are widely used. These methods have achieved significant progress in speech naturalness and intelligibility, but they still have significant shortcomings in meeting the complex application requirements such as diverse emotional expression and style transfer.

[0003] Specifically, existing technologies mainly suffer from the following problems: First, the text-to-speech mapping process often only considers the basic clarity and coherence of speech, lacking fine-grained modeling of emotional and stylistic features. This leads to synthesized speech being prone to rigidity in emotional expression and distortion in stylistic consistency. Second, although some methods attempt to introduce emotional tags or style embeddings for control, they usually rely on independent emotion or style recognition models for constraints, increasing system complexity and resulting in high training costs and low inference efficiency. Third, in the speech synthesis optimization process, traditional supervised training modes cannot fully utilize reward signals for dynamic adjustment, making it difficult to establish an effective feedback loop between the synthesis result and the target features. This results in synthesized speech remaining inflexible in terms of prosodic naturalness, pause duration, and fundamental frequency variation.

[0004] Therefore, how to provide an AI audio synthesis optimization method based on deep reinforcement learning is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] One objective of this invention is to propose an AI audio synthesis optimization method based on deep reinforcement learning. This invention fully utilizes an improved VITS model and deep reinforcement learning policy update technology, and describes in detail the steps of training a baseline model using text data, speech data, sentiment-annotated data, and style-annotated data, generating synthesized speech using a controllable parameter interface, and optimizing the model using reward signal data and an improved GRPO algorithm. This method has the advantages of high naturalness in speech synthesis, accurate emotional expression, and flexible style control.

[0006] An AI audio synthesis optimization method based on deep reinforcement learning according to an embodiment of the present invention includes the following steps:

[0007] Collect training corpus data that includes text data, speech data, sentiment annotation data, and style annotation data;

[0008] The improved VITS model was trained in a supervised manner using training corpus data to obtain the baseline improved VITS model;

[0009] The text data is regularized, phoneme-based, and pause prediction is performed, and target feature data related to sentiment annotation data and style annotation data are extracted.

[0010] Set a controllable parameter interface in the latent representation layer of the baseline improved VITS model;

[0011] Text data and target feature data are used as synthesis conditions input into the baseline improved VITS model, and the controllable parameter interface is called to generate synthesized speech data.

[0012] The synthesized speech data is evaluated to obtain reward signal data, which includes emotion consistency reward, style similarity reward and prosodic naturalness reward;

[0013] An improved GRPO algorithm is used to update the policy of the baseline improved VITS model, resulting in an optimized improved VITS model.

[0014] Optionally, the training corpus data specifically includes:

[0015] The training corpus consists of text data, speech data, sentiment-annotated data, and style-annotated data. The text data is the transcribed content of the corresponding speech data, the speech data is real speech segments recorded by multiple voice identities, the sentiment-annotated data is the sentiment category labeled for the speech data, and the style-annotated data is the speech style category labeled for the speech data.

[0016] Optionally, the step of using training corpus data to perform supervised training on the improved VITS model to obtain a baseline improved VITS model specifically includes:

[0017] An improved VITS model is constructed, which includes a text encoder, a posterior encoder, a normalized stream module, a decoder, a multi-scale discriminator, a multi-band discriminator, a speech identity embedding module, an emotion embedding module, and a style embedding module.

[0018] The text encoder consists of a text embedding layer, a one-dimensional convolutional layer, a positional encoding layer, and a feedforward network layer; the posterior encoder consists of a convolutional layer, a normalization layer, and a nonlinear activation layer; the normalization flow module consists of multiple layers of invertible mapping units; the decoder consists of convolutional units, residual connections, and nonlinear activation functions; the multi-scale discriminator consists of convolutional subnetworks with different kernel sizes; and the multi-band discriminator consists of parallel convolutional subnetworks divided into frequency bands.

[0019] The text data in the training corpus is input into the text encoder, which outputs text encoded feature data. The speech data in the training corpus is input into the posterior encoder, which outputs speech latent feature data. The speech latent feature data is input into the normalization stream module, which outputs normalized latent feature data.

[0020] A speech identity embedding module is constructed, including an embedding lookup table and a fully connected layer, which maps speech identity identifiers in the training corpus data to speech identity embedding data and concatenates them with text-encoded feature data. A sentiment embedding module is constructed, including a sentiment embedding layer and a linear mapping layer, which maps sentiment annotation data in the training corpus data to sentiment embedding data and concatenates it with text-encoded feature data. A style embedding module is constructed, which includes a style embedding layer and a multilayer perceptron, which maps style annotation data in the training corpus data to style embedding data and concatenates it with text-encoded feature data. The speech identity embedding data, sentiment embedding data, and style embedding data are jointly concatenated to obtain comprehensive conditional feature data.

[0021] The integrated conditional feature data and normalized latent feature data are input into the decoder, and the synthesized waveform data is output. The synthesized waveform data is then input into the multi-scale discriminator and the multi-band discriminator, and the discriminant feedback data is output.

[0022] Calculate the mean squared error loss of the synthesized waveform data and the speech data in the training corpus, calculate the log likelihood loss of the speech latent feature data and the normalized latent feature data, calculate the adversarial loss of the discriminative feedback data, and update the model parameters by weighting the three types of losses.

[0023] The aligner is invoked to perform duration alignment between text and speech data in the training corpus, generate duration-labeled data, train the duration prediction branch, output phoneme duration prediction data, and update the decoder prosody control parameters.

[0024] The speech data in the training corpus is randomly pruned to generate speech segments of different lengths. Corresponding text data is then pruned to form batch training data, which is input into the model for iterative training. The training is performed in stages: the first stage is reconstruction training, the second stage is adversarial training, and the third stage is joint training by combining speech identity data, sentiment annotation data, and style annotation data. After training is completed, a baseline improved VITS model is output.

[0025] Optionally, the process of performing regularization, phonemicization, and pause prediction on the text data, and extracting target feature data related to sentiment annotation and style annotation data, specifically includes:

[0026] The text data in the training corpus is regularized by removing redundant symbols and unifying character case and punctuation to obtain regularized text data.

[0027] Regularized text data is converted into phoneme sequence data according to a preset phoneme dictionary and pronunciation rules;

[0028] Perform pause prediction processing on phoneme sequence data, mark pause positions according to context, and generate pause label data;

[0029] Phoneme sequence data is combined with pause tag data to form prosodic input data;

[0030] The sentiment-annotated data is read from the training corpus and numerically processed to output sentiment feature data;

[0031] Style annotation data is read from the training corpus and numerically processed to output style feature data;

[0032] The prosody input data, sentiment feature data, and style feature data are combined and concatenated to generate target feature data.

[0033] Optionally, setting a controllable parameter interface in the latent representation layer of the baseline improved VITS model specifically includes:

[0034] In the improved VITS model, a controllable parameter interface is set, which is located in the latent representation layer, and the latent representation layer is the representation layer corresponding to the normalized latent feature data;

[0035] The duration bias parameter is input into the controllable parameter interface, and the duration bias parameter is generated from the phoneme duration prediction result;

[0036] The base frequency offset parameter is input into the controllable parameter interface, and the base frequency offset parameter is extracted by the base frequency curve generation unit;

[0037] The energy bias parameter is input into the controllable parameter interface, and the energy bias parameter is generated by the energy distribution estimation unit;

[0038] Input the speech rate ratio parameter and the pause duration parameter into the controllable parameter interface. The speech rate ratio parameter is used to globally scale the duration distribution of the phoneme sequence, and the pause duration parameter is used to locally extend or shorten the pause interval.

[0039] The emotion intensity parameter and style blending parameter are input into the controllable parameter interface. The emotion intensity parameter is a numerical scalar and the style blending parameter is a multidimensional vector.

[0040] The parameters in the controllable parameter interface are processed into a unified controllable parameter vector, and then concatenated with the normalized latent feature data to obtain the controlled latent feature data.

[0041] Optionally, the step of inputting text data and target feature data as synthesis conditions into the baseline improved VITS model and calling the controllable parameter interface to generate synthesized speech data specifically includes:

[0042] By binding the text data in the training corpus with the target feature data, a synthetic conditional input is formed.

[0043] The synthesis condition input is fed into the baseline improved VITS model, and the controllable parameter interface is called. During the synthesis process, a controllable parameter vector is generated based on the external input duration parameter, fundamental frequency parameter, energy parameter, speech rate parameter, pause parameter, sentiment parameter and style parameter.

[0044] The controllable parameter vector is fused with the latent feature data inside the baseline improved VITS model to obtain the controlled latent feature data;

[0045] The controlled latent feature data is decoded using a baseline-improved VITS model to generate synthetic speech data.

[0046] Optionally, the evaluation of the synthesized speech data to obtain reward signal data includes emotional consistency reward, style similarity reward, and prosodic naturalness reward, specifically including:

[0047] The synthesized speech data is compared with the target sentiment annotations in the training corpus data to extract the sentiment-related features of the synthesized speech and match them with the target sentiment annotations. Based on the matching results, a sentiment consistency reward value is generated and the sentiment consistency reward value is limited to between zero and one.

[0048] Style feature vectors are extracted from the synthesized speech data and their similarity is calculated with the target style feature vectors in the training corpus data. The similarity is measured by the vector similarity method. A style similarity reward value is generated based on the similarity calculation results and the style similarity reward value is limited to between zero and one.

[0049] Prosodic analysis is performed on the synthesized speech data to obtain fundamental frequency curves, energy distribution, and phoneme duration data. These data are then compared with the average fundamental frequency, average energy, and average duration data in the training corpus to obtain fundamental frequency difference values, energy difference values, and duration difference values, respectively. These values ​​are then weighted according to a preset weight ratio to obtain a comprehensive prosodic value. A prosodic naturalness reward value is generated by subtracting the comprehensive prosodic value from one, and the prosodic naturalness reward value is limited to between zero and one.

[0050] The reward values ​​for emotional consistency, style similarity, and rhythmic naturalness are weighted and combined according to a preset weighting coefficient, wherein the weighting coefficient is a non-negative number and the sum of the three is one, to generate reward signal data.

[0051] Optionally, the step of using an improved GRPO algorithm to update the policy of the baseline improved VITS model to obtain an optimized improved VITS model specifically includes:

[0052] The reward signal data is bound to the synthesized speech data to form a training sample set;

[0053] The improved GRPO algorithm is used to update the policy parameters of the baseline improved VITS model;

[0054] The improved GRPO algorithm sets up a grouped normalization update mechanism, which divides the training sample set into several subsets according to preset rules, calculates the relative advantage value between the reward signal data and the output behavior of the improved VITS model in each subset, and performs normalization processing within the subset.

[0055] The improved GRPO algorithm sets up a dynamic weight adjustment mechanism, which adaptively allocates weight coefficients based on the changes in sentiment consistency reward, style similarity reward, and prosodic naturalness reward during the training process. When the fluctuation of a certain type of reward exceeds the preset range, the corresponding weight coefficient is adjusted according to the constraint rules.

[0056] A policy distribution constraint term is introduced into the optimization objective function. The policy distribution constraint is used to measure the difference between the updated policy distribution and the original policy distribution. When the difference value exceeds a preset threshold, a penalty term is added to the objective function.

[0057] The improved GRPO algorithm sets up an adaptive learning rate adjustment mechanism, which continuously monitors the change amplitude of reward signal data during training and adjusts the learning rate according to the comparison result of the change amplitude and the preset threshold. When the change amplitude is within a stable range, the learning rate is adjusted according to the acceleration rule, and when the change amplitude exceeds the stable range, the learning rate is adjusted according to the deceleration rule.

[0058] After each batch of updates is completed, regularization constraints are applied to the parameters of the improved VITS model. These regularization constraints include restrictions on the range of parameter values ​​and control over the parameter growth rate. The controllability of parameter updates is achieved through a weight decay mechanism.

[0059] The improved GRPO algorithm is executed to complete multiple rounds of iterative training, and finally the optimized improved VITS model is output.

[0060] The beneficial effects of this invention are:

[0061] This invention introduces a controllable parameter interface for the latent representation layer based on an improved VITS model. It combines text data, sentiment-annotated data, and style-annotated data to generate target feature data as synthesis conditions, achieving fine-grained control over multi-dimensional features such as duration, fundamental frequency, energy, speech rate, pauses, emotion, and style during speech synthesis. By constructing sentiment consistency rewards, style similarity rewards, and prosodic naturalness rewards, a comprehensive synthesized speech quality evaluation system is established. An improved GRPO algorithm is used for policy updates, and mechanisms such as grouping normalization, dynamic weight adjustment, policy distribution constraints, and adaptive learning rate are utilized to improve the stability and convergence of model training. This invention not only significantly improves the shortcomings of existing speech synthesis methods in terms of naturalness, emotional expression, and style presentation, but also achieves controllable and personalized generation of synthesized speech. It can meet the application needs of intelligent voice interaction, virtual human broadcasting, and multi-scenario audio content generation, and has the beneficial effects of high synthesis quality, strong robustness, and wide application range. Attached Figure Description

[0062] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0063] Figure 1 This is a flowchart of an AI audio synthesis optimization method based on deep reinforcement learning proposed in this invention;

[0064] Figure 2 This is a framework diagram of the improved VITS model in the AI ​​audio synthesis optimization method based on deep reinforcement learning proposed in this invention;

[0065] Figure 3 This is a framework diagram of the improved GRPO algorithm in the AI ​​audio synthesis optimization method based on deep reinforcement learning proposed in this invention. Detailed Implementation

[0066] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0067] refer to Figure 1-3 An AI audio synthesis optimization method based on deep reinforcement learning includes the following steps:

[0068] Collect training corpus data that includes text data, speech data, sentiment annotation data, and style annotation data;

[0069] The improved VITS model was trained in a supervised manner using training corpus data to obtain the baseline improved VITS model;

[0070] The text data is regularized, phoneme-based, and pause prediction is performed, and target feature data related to sentiment annotation data and style annotation data are extracted.

[0071] Set a controllable parameter interface in the latent representation layer of the baseline improved VITS model;

[0072] Text data and target feature data are used as synthesis conditions input into the baseline improved VITS model, and the controllable parameter interface is called to generate synthesized speech data.

[0073] The synthesized speech data is evaluated to obtain reward signal data, which includes emotion consistency reward, style similarity reward and prosodic naturalness reward;

[0074] An improved GRPO algorithm is used to update the policy of the baseline improved VITS model, resulting in an optimized improved VITS model.

[0075] In this embodiment, the training corpus data specifically includes:

[0076] The training corpus consists of text data, speech data, sentiment-annotated data, and style-annotated data. The text data is the transcribed content of the corresponding speech data, the speech data is real speech segments recorded by multiple voice identities, the sentiment-annotated data is the sentiment category labeled for the speech data, and the style-annotated data is the speech style category labeled for the speech data.

[0077] This implementation method constructs a training corpus containing text data, speech data, sentiment annotation data, and style annotation data, providing multi-dimensional supervision signals for the model. This achieves comprehensive coverage of synthesized speech in terms of semantics, sentiment, and style, thereby improving the accuracy and diversity of speech synthesis.

[0078] In this embodiment, the step of supervising the improved VITS model using training corpus data to obtain a baseline improved VITS model specifically includes:

[0079] An improved VITS model is constructed, which includes a text encoder, a posterior encoder, a normalized stream module, a decoder, a multi-scale discriminator, a multi-band discriminator, a speech identity embedding module, an emotion embedding module, and a style embedding module.

[0080] The text encoder consists of a text embedding layer, a one-dimensional convolutional layer, a positional encoding layer, and a feedforward network layer; the posterior encoder consists of a convolutional layer, a normalization layer, and a nonlinear activation layer; the normalization flow module consists of multiple layers of invertible mapping units; the decoder consists of convolutional units, residual connections, and nonlinear activation functions; the multi-scale discriminator consists of convolutional subnetworks with different kernel sizes; and the multi-band discriminator consists of parallel convolutional subnetworks divided into frequency bands.

[0081] The text data in the training corpus is input into the text encoder, which outputs text encoded feature data. The speech data in the training corpus is input into the posterior encoder, which outputs speech latent feature data. The speech latent feature data is input into the normalization stream module, which outputs normalized latent feature data.

[0082] A speech identity embedding module is constructed, including an embedding lookup table and a fully connected layer, which maps speech identity identifiers in the training corpus data to speech identity embedding data and concatenates them with text-encoded feature data. A sentiment embedding module is constructed, including a sentiment embedding layer and a linear mapping layer, which maps sentiment annotation data in the training corpus data to sentiment embedding data and concatenates it with text-encoded feature data. A style embedding module is constructed, which includes a style embedding layer and a multilayer perceptron, which maps style annotation data in the training corpus data to style embedding data and concatenates it with text-encoded feature data. The speech identity embedding data, sentiment embedding data, and style embedding data are jointly concatenated to obtain comprehensive conditional feature data.

[0083] The integrated conditional feature data and normalized latent feature data are input into the decoder, and the synthesized waveform data is output. The synthesized waveform data is then input into the multi-scale discriminator and the multi-band discriminator, and the discriminant feedback data is output.

[0084] Calculate the mean squared error loss of the synthesized waveform data and the speech data in the training corpus, calculate the log likelihood loss of the speech latent feature data and the normalized latent feature data, calculate the adversarial loss of the discriminative feedback data, and update the model parameters by weighting the three types of losses.

[0085] The aligner is invoked to perform duration alignment between text and speech data in the training corpus, generate duration-labeled data, train the duration prediction branch, output phoneme duration prediction data, and update the decoder prosody control parameters.

[0086] The speech data in the training corpus is randomly pruned to generate speech segments of different lengths. Corresponding text data is then pruned to form batch training data, which is input into the model for iterative training. The training is performed in stages: the first stage is reconstruction training, the second stage is adversarial training, and the third stage is joint training by combining speech identity data, sentiment annotation data, and style annotation data. After training is completed, a baseline improved VITS model is output.

[0087] This implementation achieves a comprehensive improvement in the naturalness, personalization, and emotional expression of synthesized speech by designing an improved VITS model that includes a text encoder, a posterior encoder, a normalized stream module, a decoder, a discriminator, and embedding modules for speech identity, emotion, and style.

[0088] In this embodiment, the regularization, phonemicization, and pause prediction processing of the text data, and the extraction of target feature data related to sentiment annotation data and style annotation data, specifically include:

[0089] The text data in the training corpus is regularized by removing redundant symbols and unifying character case and punctuation to obtain regularized text data.

[0090] Regularized text data is converted into phoneme sequence data according to a preset phoneme dictionary and pronunciation rules;

[0091] Perform pause prediction processing on phoneme sequence data, mark pause positions according to context, and generate pause label data;

[0092] Phoneme sequence data is combined with pause tag data to form prosodic input data;

[0093] The sentiment-annotated data is read from the training corpus and numerically processed to output sentiment feature data;

[0094] Style annotation data is read from the training corpus and numerically processed to output style feature data;

[0095] The prosody input data, sentiment feature data, and style feature data are combined and concatenated to generate target feature data.

[0096] This implementation improves speech synthesis in terms of prosodic control and speech fluency by performing regularization, phonemicization, and pause prediction processing on text data and extracting emotion and style features to ensure the accuracy and standardization of input conditions.

[0097] In this embodiment, setting a controllable parameter interface in the latent representation layer of the baseline improved VITS model specifically includes:

[0098] In the improved VITS model, a controllable parameter interface is set. The controllable parameter interface is located in the latent representation layer, which is the representation layer corresponding to the normalized latent feature data. It is used to accept external control parameters and fuse the latent features before the latent features enter the decoder.

[0099] The duration bias parameter is input into the controllable parameter interface. The duration bias parameter is generated from the phoneme duration prediction result and is used to adjust the duration ratio of the phoneme unit in the latent representation layer.

[0100] The fundamental frequency offset parameter is input into the controllable parameter interface. The fundamental frequency offset parameter is extracted by the fundamental frequency curve generation unit and is used to correct the fundamental frequency profile in the latent representation layer, thereby adjusting the pitch direction of the speech.

[0101] The energy bias parameter is input into the controllable parameter interface. The energy bias parameter is generated by the energy distribution estimation unit and is used to modify the speech energy intensity in the latent representation layer to control the loudness and stress prominence.

[0102] Input the speech rate ratio parameter and the pause duration parameter into the controllable parameter interface. The speech rate ratio parameter is used to globally scale the duration distribution of the phoneme sequence, and the pause duration parameter is used to locally extend or shorten the pause interval.

[0103] The emotion intensity parameter and style mixing parameter are input into the controllable parameter interface. The emotion intensity parameter is a numerical scalar used to adjust the strength of emotion expression. The style mixing parameter is a multi-dimensional vector used to perform linear interpolation in the multi-style embedding space, so that the latent representation layer can output latent feature data with mixed style features.

[0104] The parameters in the controllable parameter interface are processed into a unified controllable parameter vector, and then concatenated with normalized latent feature data to obtain controlled latent feature data. The controlled latent feature data is used as the input of the decoder to drive the decoder to generate synthetic waveform data that meets the requirements of duration, fundamental frequency, energy, speech rate, pauses, emotion and style control.

[0105] This implementation achieves multi-dimensional controllability of the speech generation process by setting a controllable parameter interface in the latent representation layer and introducing parameters such as duration, fundamental frequency, energy, speech rate, pauses, emotion, and style, thereby enhancing the flexibility and personalized expressiveness of synthesized speech.

[0106] In this embodiment, the step of inputting text data and target feature data as synthesis conditions into the baseline improved VITS model and calling the controllable parameter interface to generate synthesized speech data specifically includes:

[0107] By binding the text data in the training corpus with the target feature data, a synthetic conditional input is formed.

[0108] The synthesis condition input is fed into the baseline improved VITS model, and the controllable parameter interface is called. During the synthesis process, a controllable parameter vector is generated based on the external input duration parameter, fundamental frequency parameter, energy parameter, speech rate parameter, pause parameter, sentiment parameter and style parameter.

[0109] The controllable parameter vector is fused with the latent feature data inside the baseline improved VITS model to obtain the controlled latent feature data;

[0110] The controlled latent feature data is decoded using a baseline-improved VITS model to generate synthetic speech data.

[0111] This implementation method binds text data and target feature data into the model and calls the controllable parameter interface to generate synthesized speech, thereby achieving effective fusion of emotional and stylistic conditions with potential features, thus ensuring the coordination of synthesized speech in terms of timbre, rhythm and expression.

[0112] In this embodiment, the evaluation of the synthesized speech data to obtain reward signal data includes emotional consistency reward, style similarity reward, and prosodic naturalness reward, specifically including:

[0113] The synthesized speech data is compared with the target sentiment annotations in the training corpus data to extract the sentiment-related features of the synthesized speech and match them with the target sentiment annotations. Based on the matching results, a sentiment consistency reward value is generated and the sentiment consistency reward value is limited to between zero and one.

[0114] Style feature vectors are extracted from the synthesized speech data and their similarity is calculated with the target style feature vectors in the training corpus data. The similarity is measured by the vector similarity method. A style similarity reward value is generated based on the similarity calculation results and the style similarity reward value is limited to between zero and one.

[0115] Prosodic analysis is performed on the synthesized speech data to obtain fundamental frequency curves, energy distribution, and phoneme duration data. These data are then compared with the average fundamental frequency, average energy, and average duration data in the training corpus to obtain fundamental frequency difference values, energy difference values, and duration difference values, respectively. These values ​​are then weighted according to a preset weight ratio to obtain a comprehensive prosodic value. A prosodic naturalness reward value is generated by subtracting the comprehensive prosodic value from one, and the prosodic naturalness reward value is limited to between zero and one.

[0116] The reward values ​​for emotional consistency, style similarity, and rhythmic naturalness are weighted and combined according to a preset weighting coefficient, wherein the weighting coefficient is a non-negative number and the sum of the three is one, to generate reward signal data.

[0117] This implementation method constructs emotional consistency rewards, style similarity rewards, and prosodic naturalness rewards, and generates a comprehensive reward signal, thereby achieving a multi-dimensional objective evaluation of the quality of synthesized speech, effectively guiding the model to optimize towards natural fluency, emotional authenticity, and clear style.

[0118] In this embodiment, the step of using the improved GRPO algorithm to update the policy of the baseline improved VITS model to obtain the optimized improved VITS model specifically includes:

[0119] The reward signal data is bound to the synthesized speech data to form a training sample set;

[0120] The improved GRPO algorithm is used to update the policy parameters of the baseline improved VITS model;

[0121] The improved GRPO algorithm sets up a grouped normalization update mechanism, which divides the training sample set into several subsets according to preset rules. Within each subset, the relative advantage value between the reward signal data and the output behavior of the improved VITS model is calculated, and normalization processing is performed within the subset to reduce the impact of sample distribution differences on the parameter update process and ensure the balance of update results in different subsets.

[0122] The improved GRPO algorithm sets up a dynamic weight adjustment mechanism. Based on the changes in sentiment consistency reward, style similarity reward, and prosodic naturalness reward during the training process, the weight coefficients are adaptively allocated. When the fluctuation of a certain type of reward exceeds the preset range, the corresponding weight coefficient is adjusted according to the constraint rules to keep the contribution of various rewards to the optimization objective balanced.

[0123] A policy distribution constraint term is introduced into the optimization objective function. The policy distribution constraint is used to measure the difference between the updated policy distribution and the original policy distribution. When the difference exceeds a preset threshold, a penalty term is added to the objective function to limit the offset of a single update and avoid abnormal jumps in parameters during the iteration process.

[0124] The improved GRPO algorithm sets up an adaptive learning rate adjustment mechanism, which continuously monitors the change amplitude of reward signal data during training and adjusts the learning rate according to the comparison result of the change amplitude with the preset threshold. When the change amplitude is within a stable range, the learning rate is adjusted according to the acceleration rule, and when the change amplitude exceeds the stable range, the learning rate is adjusted according to the deceleration rule, thereby ensuring the convergence of the update process.

[0125] After each batch of updates is completed, regularization constraints are applied to the parameters of the improved VITS model. These regularization constraints include restrictions on the range of parameter values ​​and control over the parameter growth rate to avoid instability caused by excessive parameter expansion. The controllability of parameter updates is achieved through a weight decay mechanism.

[0126] The improved GRPO algorithm is executed to complete multiple rounds of iterative training, and finally the optimized improved VITS model is output.

[0127] This implementation improves the stability and convergence of the training process by introducing grouping normalization, dynamic weight adjustment, policy distribution constraints, and adaptive learning rate mechanisms into the GRPO algorithm, thereby obtaining an optimized model that performs better in terms of emotional expression, style control, and prosodic naturalness.

[0128] Example 1:

[0129] To verify the feasibility of this invention in practice, it was applied to an intelligent customer service voice interaction scenario, specifically the online voice customer service system of a large e-commerce platform. In this system, users consult and interact with a customer service robot via voice. The robot needs to quickly generate synthesized voices with different emotions and styles in different situations. For example, it needs to maintain a calm and professional tone when explaining return policies, a warm and engaging tone when recommending promotional products, and a calm and reassuring tone when handling complaints.

[0130] Traditional text-to-speech synthesis methods, while capable of generating relatively natural and fluent speech, often lack fine-grained emotional expression and style variation control. For example, existing models like Tacotron2 and the unmodified VITS model often only exhibit a neutral style when generating customer service speech, failing to meet the differentiated needs in real-world scenarios. This invention, through an improved VITS model combined with a deep reinforcement learning optimization mechanism, significantly enhances the performance of synthesized speech in terms of emotional consistency, style similarity, and prosodic naturalness.

[0131] During the experimental data preparation process, a multi-speech corpus of 500 hours was collected, covering 50 speakers and 10 common customer service situational styles, including neutral answers, enthusiastic recommendations, reassuring explanations, formal statements, and humorous exchanges. Each speech was also sentiment-labeled, covering five major sentiment categories: positive, negative, calm, enthusiastic, and composed. The collected speech data was transcribed into text data, and then regularized, phoneme-based, and pause prediction annotations were performed to generate target feature data.

[0132] In the model training phase, supervised training is first performed using the improved VITS model proposed in this invention. Then, a controllable parameter interface is introduced, inputting a mixture of parameters including duration, fundamental frequency, energy, speech rate, pauses, emotional intensity, and style into the model. During the synthesis phase, the parameters are continuously updated through a reinforcement learning mechanism, allowing the generated speech to be continuously optimized under the guidance of the reward signal. The reward signal consists of three parts: emotional consistency reward, style similarity reward, and prosodic naturalness reward.

[0133] In the evaluation phase, a combination of objective and subjective metrics was employed. Objective metrics included sentiment consistency score, style similarity score, and prosodic naturalness score, while the subjective metric was the Mean Opinion Score (MOS). The experiment compared three methods: the traditional Tacotron2, the unmodified VITS, and the improved VITS and deep reinforcement learning optimization of this invention. The experimental results are shown in Table 1.

[0134] Table 1. Comparison of the effects of different speech synthesis methods

[0135] method Affective consistency score Style similarity score Rhythm Naturalness Score MOS Subjective Rating Tacotron2 0.62 0.58 0.65 3.4 Original VITS 0.71 0.69 0.74 3.9 Method of the present invention 0.89 0.87 0.91 4.6

[0136] As shown in Table 1, the method of this invention outperforms existing methods in all metrics. Specific analysis is as follows: In terms of emotional consistency, the method of this invention scores 0.89, a 25.4% improvement over the original VITS and a 43.5% improvement over Tacotron2, indicating that this invention can accurately express the target emotional characteristics. In terms of style similarity, the method of this invention scores 0.87, a 26.1% improvement over the original VITS and a 50.0% improvement over Tacotron2, proving that this invention can effectively maintain consistency between speech style and the target scene. In terms of prosodic naturalness, the method of this invention scores 0.91, a 23.0% improvement over the original VITS and a 40.0% improvement over Tacotron2, indicating that the synthesized speech is more natural and fluent in terms of pause distribution, speech rate variation, and fundamental frequency curve. In terms of user subjective rating, the method of this invention achieves an average MOS score of 4.6 (out of 5), 0.7 points higher than the original VITS and 1.2 points higher than Tacotron2, significantly improving the user's auditory experience.

[0137] Furthermore, experiments conducted in real-world customer service scenarios using the method of this invention demonstrate that the synthesized speech style and emotion can be quickly switched when different synthesis parameters are invoked. For example, when the "calm" emotion parameter is input, the generated synthesized speech maintains a low-frequency, stable tone, moderate speed, and natural pauses, making it suitable for handling user complaints. When the "enthusiastic" emotion parameter is input, the generated synthesized speech maintains a higher fundamental frequency and energy, with a slightly faster speed and significant emotional impact, making it suitable for promotional recommendation scenarios. In actual testing, 87% of users reported that the speech generated using the method of this invention was natural and fluent, with appropriate emotional expression in 1000 conversations, while only 65% ​​of users gave similar feedback when using the original VITS speech, and only 52% of users gave similar feedback when using the Tacotron2 speech.

[0138] The method of this invention not only achieves significant advantages in objective evaluation indicators, but also allows for flexible adjustment of the emotion and style of synthesized speech according to scenario requirements in practical applications, thereby significantly improving the user experience and application value of intelligent voice interaction systems.

[0139] As demonstrated by the above embodiments, this invention has significant beneficial effects in the following aspects: First, by introducing an improved VITS model and a deep reinforcement learning mechanism, this invention solves the problems of insufficient emotional consistency and style similarity in existing speech synthesis methods. Second, this invention achieves fine-grained adjustment of elements such as duration, fundamental frequency, energy, speech rate, and pauses through a controllable parameter interface, significantly improving the prosodic naturalness of synthesized speech. Third, this invention establishes a feedback loop through reward signals and policy update mechanisms, enabling continuous optimization of the model's synthesis effect in practical application scenarios. In summary, this invention not only improves the objective quality indicators of speech synthesis but also significantly enhances the user's subjective experience, possessing broad application prospects and promotional value.

[0140] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. An AI audio synthesis optimization method based on deep reinforcement learning, characterized in that, Includes the following steps: Collect training corpus data that includes text data, speech data, sentiment annotation data, and style annotation data; The improved VITS model was trained in a supervised manner using training corpus data to obtain the baseline improved VITS model; The text data is regularized, phoneme-based, and pause prediction is performed, and target feature data related to sentiment annotation data and style annotation data are extracted. Set a controllable parameter interface in the latent representation layer of the baseline improved VITS model; Text data and target feature data are used as synthesis conditions input into the baseline improved VITS model, and the controllable parameter interface is called to generate synthesized speech data. The synthesized speech data is evaluated to obtain reward signal data, which includes emotion consistency reward, style similarity reward and prosodic naturalness reward; An improved GRPO algorithm is used to update the policy of the baseline improved VITS model, resulting in an optimized improved VITS model.

2. The AI ​​audio synthesis optimization method based on deep reinforcement learning according to claim 1, characterized in that, The training corpus data specifically includes: The training corpus consists of text data, speech data, sentiment-annotated data, and style-annotated data. The text data is the transcribed content of the corresponding speech data, the speech data is real speech segments recorded by multiple voice identities, the sentiment-annotated data is the sentiment category labeled for the speech data, and the style-annotated data is the speech style category labeled for the speech data.

3. The AI ​​audio synthesis optimization method based on deep reinforcement learning according to claim 1, characterized in that, The process of supervising the improved VITS model using training corpus data to obtain a baseline improved VITS model specifically includes: An improved VITS model is constructed, which includes a text encoder, a posterior encoder, a normalized stream module, a decoder, a multi-scale discriminator, a multi-band discriminator, a speech identity embedding module, an emotion embedding module, and a style embedding module. The text encoder consists of a text embedding layer, a one-dimensional convolutional layer, a positional encoding layer, and a feedforward network layer; the posterior encoder consists of a convolutional layer, a normalization layer, and a nonlinear activation layer; the normalization flow module consists of multiple layers of invertible mapping units; the decoder consists of convolutional units, residual connections, and nonlinear activation functions; the multi-scale discriminator consists of convolutional subnetworks with different kernel sizes; and the multi-band discriminator consists of parallel convolutional subnetworks divided into frequency bands. The text data in the training corpus is input into the text encoder, which outputs text encoded feature data. The speech data in the training corpus is input into the posterior encoder, which outputs speech latent feature data. The speech latent feature data is input into the normalization stream module, which outputs normalized latent feature data. A speech identity embedding module is constructed, including an embedding lookup table and a fully connected layer, which maps speech identity identifiers in the training corpus data to speech identity embedding data and concatenates them with text-encoded feature data. A sentiment embedding module is constructed, including a sentiment embedding layer and a linear mapping layer, which maps sentiment annotation data in the training corpus data to sentiment embedding data and concatenates it with text-encoded feature data. A style embedding module is constructed, which includes a style embedding layer and a multilayer perceptron, which maps style annotation data in the training corpus data to style embedding data and concatenates it with text-encoded feature data. The speech identity embedding data, sentiment embedding data, and style embedding data are jointly concatenated to obtain comprehensive conditional feature data. The integrated conditional feature data and normalized latent feature data are input into the decoder, and the synthesized waveform data is output. The synthesized waveform data is then input into the multi-scale discriminator and the multi-band discriminator, and the discriminant feedback data is output. Calculate the mean squared error loss of the synthesized waveform data and the speech data in the training corpus, calculate the log likelihood loss of the speech latent feature data and the normalized latent feature data, calculate the adversarial loss of the discriminative feedback data, and update the model parameters by weighting the three types of losses. The aligner is invoked to perform duration alignment between text and speech data in the training corpus, generate duration-labeled data, train the duration prediction branch, output phoneme duration prediction data, and update the decoder prosody control parameters. The speech data in the training corpus is randomly pruned to generate speech segments of different lengths. Corresponding text data is then pruned to form batch training data, which is input into the model for iterative training. The training is performed in stages: the first stage is reconstruction training, the second stage is adversarial training, and the third stage is joint training by combining speech identity data, sentiment annotation data, and style annotation data. After training is completed, a baseline improved VITS model is output.

4. The AI ​​audio synthesis optimization method based on deep reinforcement learning according to claim 1, characterized in that, The process of performing regularization, phonemicization, and pause prediction on the text data, and extracting target feature data related to sentiment and style annotation data, specifically includes: The text data in the training corpus is regularized by removing redundant symbols and unifying character case and punctuation to obtain regularized text data. Regularized text data is converted into phoneme sequence data according to a preset phoneme dictionary and pronunciation rules; Perform pause prediction processing on phoneme sequence data, mark pause positions according to context, and generate pause label data; Phoneme sequence data is combined with pause tag data to form prosodic input data; The sentiment-annotated data is read from the training corpus and numerically processed to output sentiment feature data; Style annotation data is read from the training corpus and numerically processed to output style feature data; The prosody input data, sentiment feature data, and style feature data are combined and concatenated to generate target feature data.

5. The AI ​​audio synthesis optimization method based on deep reinforcement learning according to claim 1, characterized in that, The setting of a controllable parameter interface in the latent representation layer of the baseline improved VITS model specifically includes: In the improved VITS model, a controllable parameter interface is set, which is located in the latent representation layer, and the latent representation layer is the representation layer corresponding to the normalized latent feature data; The duration bias parameter is input into the controllable parameter interface, and the duration bias parameter is generated from the phoneme duration prediction result; The base frequency offset parameter is input into the controllable parameter interface, and the base frequency offset parameter is extracted by the base frequency curve generation unit; The energy bias parameter is input into the controllable parameter interface, and the energy bias parameter is generated by the energy distribution estimation unit; Input the speech rate ratio parameter and the pause duration parameter into the controllable parameter interface. The speech rate ratio parameter is used to globally scale the duration distribution of the phoneme sequence, and the pause duration parameter is used to locally extend or shorten the pause interval. The emotion intensity parameter and style blending parameter are input into the controllable parameter interface. The emotion intensity parameter is a numerical scalar and the style blending parameter is a multidimensional vector. The parameters in the controllable parameter interface are processed into a unified controllable parameter vector, and then concatenated with the normalized latent feature data to obtain the controlled latent feature data.

6. The AI ​​audio synthesis optimization method based on deep reinforcement learning according to claim 1, characterized in that, The process of inputting text data and target feature data as synthesis conditions into the baseline improved VITS model and calling the controllable parameter interface to generate synthesized speech data specifically includes: By binding the text data in the training corpus with the target feature data, a synthetic conditional input is formed. The synthesis condition input is fed into the baseline improved VITS model, and the controllable parameter interface is called. During the synthesis process, a controllable parameter vector is generated based on the external input duration parameter, fundamental frequency parameter, energy parameter, speech rate parameter, pause parameter, sentiment parameter and style parameter. The controllable parameter vector is fused with the latent feature data inside the baseline improved VITS model to obtain the controlled latent feature data; The controlled latent feature data is decoded using a baseline-improved VITS model to generate synthetic speech data.

7. The AI ​​audio synthesis optimization method based on deep reinforcement learning according to claim 1, characterized in that, The synthesized speech data is evaluated to obtain reward signal data, which includes emotion consistency reward, style similarity reward, and prosodic naturalness reward, specifically including: The synthesized speech data is compared with the target sentiment annotations in the training corpus data to extract the sentiment-related features of the synthesized speech and match them with the target sentiment annotations. Based on the matching results, a sentiment consistency reward value is generated and the sentiment consistency reward value is limited to between zero and one. Style feature vectors are extracted from the synthesized speech data and their similarity is calculated with the target style feature vectors in the training corpus data. The similarity is measured by the vector similarity method. A style similarity reward value is generated based on the similarity calculation results and the style similarity reward value is limited to between zero and one. Prosodic analysis is performed on the synthesized speech data to obtain fundamental frequency curves, energy distribution, and phoneme duration data. These data are then compared with the average fundamental frequency, average energy, and average duration data in the training corpus to obtain fundamental frequency difference values, energy difference values, and duration difference values, respectively. These values ​​are then weighted according to a preset weight ratio to obtain a comprehensive prosodic value. A prosodic naturalness reward value is generated by subtracting the comprehensive prosodic value from one, and the prosodic naturalness reward value is limited to between zero and one. The reward values ​​for emotional consistency, style similarity, and rhythmic naturalness are weighted and combined according to a preset weighting coefficient, wherein the weighting coefficient is a non-negative number and the sum of the three is one, to generate reward signal data.

8. The AI ​​audio synthesis optimization method based on deep reinforcement learning according to claim 1, characterized in that, The process of updating the policy of the baseline improved VITS model using the improved GRPO algorithm to obtain the optimized improved VITS model specifically includes: The reward signal data is bound to the synthesized speech data to form a training sample set; The improved GRPO algorithm is used to update the policy parameters of the baseline improved VITS model; The improved GRPO algorithm sets up a grouped normalization update mechanism, which divides the training sample set into several subsets according to preset rules, calculates the relative advantage value between the reward signal data and the output behavior of the improved VITS model in each subset, and performs normalization processing within the subset. The improved GRPO algorithm sets up a dynamic weight adjustment mechanism, which adaptively allocates weight coefficients based on the changes in sentiment consistency reward, style similarity reward, and prosodic naturalness reward during the training process. When the fluctuation of a certain type of reward exceeds the preset range, the corresponding weight coefficient is adjusted according to the constraint rules. A policy distribution constraint term is introduced into the optimization objective function. The policy distribution constraint is used to measure the difference between the updated policy distribution and the original policy distribution. When the difference value exceeds a preset threshold, a penalty term is added to the objective function. The improved GRPO algorithm sets up an adaptive learning rate adjustment mechanism, which continuously monitors the change amplitude of reward signal data during training and adjusts the learning rate according to the comparison result of the change amplitude and the preset threshold. When the change amplitude is within a stable range, the learning rate is adjusted according to the acceleration rule, and when the change amplitude exceeds the stable range, the learning rate is adjusted according to the deceleration rule. After each batch of updates is completed, regularization constraints are applied to the parameters of the improved VITS model. These regularization constraints include restrictions on the range of parameter values ​​and control over the parameter growth rate. The controllability of parameter updates is achieved through a weight decay mechanism. The improved GRPO algorithm is executed to complete multiple rounds of iterative training, and finally the optimized improved VITS model is output.

Images