Audio synthesis method based on VITS model improvement and storage medium

Abstract

The invention discloses an improved audio synthesis method based on a VITS model and a storage medium, and belongs to the technical field of speech synthesis, and the method comprises the steps: obtaining a text of to-be-synthesized audio data, and carrying out the preprocessing of the text; inputting the preprocessed text into a pre-trained adaptive speech synthesis model AdaVITS for audio synthesis; according to the output of the adaptive speech synthesis model AdaVITS, obtaining generated audio data; wherein the adaptive speech synthesis model AdaVITS is based on a speech synthesis model VITS, and a loss function of the speech synthesis model VITS is improved and increased to obtain a joint loss function of the adaptive speech synthesis model AdaVITS; and performing optimal solution on the joint loss function, thereby realizing collaborative optimization of the voice quality and the training efficiency.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of speech synthesis, in particular to an audio synthesis method based on an improved VITS model and a storage medium. BACKGROUND

[0002] With the rapid development of communication technology and the deep integration of artificial intelligence, the demand for intelligent services in the telecommunications industry is increasing. Voice interaction scenarios represented by intelligent customer service have become an important part of telecommunications services. However, traditional speech synthesis technology still faces many challenges in practical applications, and it is difficult to meet the demand for high-quality and efficient speech synthesis in the telecommunications field.

[0003] Traditional speech synthesis technology mainly includes concatenation synthesis and parameter synthesis. Concatenation synthesis relies on a large speech library to generate target speech by editing and concatenating pre-recorded speech segments. Although it can preserve the naturalness of speech, it is limited by the coverage of the speech library, has poor flexibility, and the concatenation place is prone to discontinuity or incoherence. Parameter synthesis generates speech waveforms through mathematical models (such as HMM, statistical parameter model), which has a certain flexibility, but the naturalness and emotional expression ability of the synthesized speech are weak, especially in handling multi-syllable words, complex intonation and emotional expression.

[0004] In addition, traditional methods require complex multi-stage processes (such as acoustic modeling, prosody prediction, vocoder, etc.), resulting in high system complexity, high training cost, and difficulty in adapting to multi-language, multi-dialect, and personalized speech requirements. In the telecommunications scenario, these defects directly affect the user experience of intelligent customer service, such as strong mechanical feeling of voice, high response delay, and inability to adapt to diverse user interaction needs. SUMMARY

[0005] The purpose of the present application is to provide an audio synthesis method based on an improved VITS model and a storage medium, by improving the loss function of the speech synthesis model VITS and adding the joint loss function of the adaptive speech synthesis model AdaVITS. Then, the optimal solution of the joint loss function is obtained by setting a specific value, realizing the collaborative optimization of speech quality and training efficiency. The present application is realized by the following technical solutions.

[0006] In the first aspect, the present application provides an audio synthesis method based on an improved VITS model, including the following contents:

[0007] Obtain the text of the audio data to be synthesized, and preprocess the text;

[0008] Input the preprocessed text into the pre-trained adaptive speech synthesis model AdaVITS for audio synthesis;

[0009] Audio data generated according to an output of the adaptive voice synthesis model AdaVITS;

[0010] The adaptive voice synthesis model AdaVITS is based on a voice synthesis model VITS, and a loss function of the voice synthesis model VITS is improved and increased to obtain a joint loss function of the adaptive voice synthesis model AdaVITS; and the joint loss function is optimally solved to realize collaborative optimization of voice quality and training efficiency.

[0011] The improved joint loss function is obtained by the following formula:

[0012]

[0013] In the formula, is a joint loss function, is a spectrum domain reconstruction loss function, is a discriminator loss function, is a generator loss function, is a multi-scale feature matching loss function, is a random duration predictor loss function, is a forward KL divergence regularization loss function, is a reverse KL divergence regularization loss function, and are dynamic adjustment KL divergence weight coefficients.

[0014] Optionally, the data set for training includes audio data and text data corresponding to the audio data.

[0015] The preprocessing of the data set for training includes preprocessing of the audio data and preprocessing of the text data.

[0016] The preprocessing of the audio data includes cleaning the audio data, removing abnormal audio (such as invalid signals, blank audio) and low-quality audio, detecting and cropping silent segments to ensure effective speech content, and denoising and enhancing the audio data to obtain preprocessed audio data; and existing technologies and related denoising algorithms (such as spectral subtraction) are used to reduce background noise and perform equalization processing on the speech signal to improve the signal-to-noise ratio and clarity.

[0017] ​The preprocessing of the text data includes cleaning the text data, removing special symbols, stop words and meaningless content, unifying the text format, and normalizing the text data by converting text information such as time, special symbols and date into unified standard vocabulary to obtain preprocessed text data. (For example, converting the time representation "22:30" to "twenty-two thirty", converting the unit "kg" to "kilogram", and converting the number "10086" to "one thousand eighty-six" or "one zero zero eight six").

[0018] The preprocessed text data is converted into a phoneme sequence, and a rhythm prediction is performed through existing technologies and related models, such as a bidirectional encoder representation model based on Transformer (BERT). The complete phoneme sequence is obtained by splicing according to the related vocabulary, for example:

[0019] Text data: Today the weather is really good

[0020] Rhythm prediction: today #0 day #1 day #0 q #1 really #0 good #0 ah #4

[0021] Final phoneme sequence: sil j in1 #0 t ian1 #1 t ian1 #0 q i4 #1 zh en1 #0 h ao3#0 ^ a5 #4.

[0022] The preprocessed audio data and its corresponding phoneme sequence together constitute a preprocessed data set for training.

[0023] Optionally, the spectral domain reconstruction loss function is obtained by the following formula:

[0024] ,

[0025] In the formula, represents the target mel spectrum, represents the predicted mel spectrum, represents the L1 norm.

[0026] The reconstruction loss of the traditional speech synthesis model VITS is usually directly aimed at the original waveform domain, but the present application proposes to take the mel spectrum as the reconstruction target. This method decodes the latent variable into the waveform domain , and then converts it into the mel spectrum domain Thus, the L1 loss between the predicted value and the target value is calculated. This improvement significantly reduces the dependence on waveform domain reconstruction, improves the model's ability to capture speech spectral features, and reduces computational complexity.

[0027] Optionally, the discriminator loss function and the generator loss function are calculated by the following formulas, respectively:

[0028] ,

[0029] ,

[0030] In the formula, represents the training set speech data, represents the latent variable, represents the output probability of the discriminator D, represents the output probability of the generator G, is the mathematical expectation of the joint distribution of the training set speech data and the latent variable , is the mathematical expectation of the distribution of the latent variable . This design improves the generation ability of the generator G for high-fidelity speech through dynamic game, and enhances the sensitivity of the discriminator D to detailed features.

[0031] In the training process of the present application, based on the speech synthesis model VITS, the double target loss function is improved, respectively for the discriminator and the generator . The discriminator loss is realized by maximizing the discrimination ability between the real data and the generated data, while the generator loss is optimized by minimizing the discrimination difference between the generated data and the real data.

[0032] Optionally, the multi-scale feature matching loss function is calculated by the following formula:

[0033] ,

[0034] In the formula, represents the real speech data, represents the latent variable, represents the output probability of the discriminator, represents the output probability of the generator, represents the L1 norm, represents the intermediate feature output probability of the discriminator at the layer, represents the feature dimension, represents the total number of layers of the discriminator.

[0035] To alleviate the adversarial imbalance between the generator G and the discriminator D, this invention improves the multi-scale feature matching loss function of the traditional speech synthesis model VITS. The core idea is to match the feature distribution of the generator G output with the real data in the intermediate layer of the discriminator D. This loss only applies to the generator G. This improvement effectively suppresses the mode collapse problem during training by constraining the global consistency of the generator output.

[0036] Optionally, the loss function of the random duration predictor is calculated using the following formula:

[0037] ,

[0038] In the formula, Represents the linear spectrum. Indicates text encoding. Represents the alignment matrix. Represents the given input features The posterior distribution, This represents the conditional likelihood term of the AdaVITS adaptive speech synthesis model, specifically representing the conditional likelihood term given the latent variables. After (implicit encoding representing phoneme duration), the Mel spectrum is reconstructed. The probability, Represents the posterior distribution, specifically representing the encoding of a given text. (such as phoneme embedding representation) and alignment matrix (Alignment relationship between text phonemes and speech frames) followed by latent variables The distribution of .

[0039] To avoid gradient interference, the gradient backpropagation of the random duration predictor is explicitly cut off, and phoneme durations are mapped to integer values ​​after sampling from random noise through an invertible transformation. This design significantly improves the robustness of duration prediction, especially enhancing the stability of phoneme alignment in multi-speaker scenarios.

[0040] Optionally, the positive KL divergence regularization loss function is calculated using the following formula:

[0041] ,

[0042] ,

[0043] In the formula, Let be the mean vector of the posterior distribution. Let be the standard deviation vector of the posterior distribution. For input features, Represents the given input features The posterior distribution, Indicates a given condition Latent variablesa prior distribution of represents an alignment matrix;

[0044] The reverse KL divergence regularization loss function is calculated by the following formula:

[0045] ,

[0046] In the formula, represents the reconstructed hidden variable.

[0047] The reverse KL divergence regularization loss function balances the complexity of the posterior distribution and the prior distribution, so that the forward and reverse KL divergence regularization loss functions are closer to the needs of actual application scenarios. This improvement enhances the modeling ability of the adaptive speech synthesis model AdaVITS for the text-speech coupling relationship and improves the naturalness of speech generation.

[0048] Optionally, the optimal solution of the joint loss function includes obtaining a target function by target decoupling, defining a constraint condition and constructing a Lagrange function and a dynamic updating mechanism to obtain the parameters of the updated adaptive speech synthesis model AdaVITS, and minimizing the target function and satisfying the constraint condition through the parameters of the updated adaptive speech synthesis model AdaVITS, so as to obtain the optimal solution of the joint loss function.

[0049] Optionally, the target decoupling includes separating the spectrum domain reconstruction loss function from other loss functions in the joint loss function, defining a target function and a constraint condition :

[0050] ,

[0051] ,

[0052] In the formula, is a preset target value of the spectrum domain reconstruction loss function , i represents the type of loss function, is the parameter of the adaptive speech synthesis model AdaVITS. is usually the convergence value when the HiFi-GAN vocoder in the adaptive speech synthesis model AdaVITS is independently trained .

[0053] The construction of the Lagrange function includes introducing a penalty term , constructing a Lagrange function , and the formula is as follows:

[0054] ,

[0055] In the formula, is a Lagrange multiplier, is a damping constant;

[0056] The dynamic updating mechanism comprises updating , through the updated Update , the formula is as follows:

[0057] ,

[0058] .

[0059] The parameters of the updated adaptive speech synthesis model AdaVITS , so that the objective function Minimum, while meeting the constraint condition Let The adaptive speech synthesis model AdaVITS automatically finds the Pareto optimal equilibrium point, so as to obtain the optimal solution of the joint loss function, and realizes the collaborative optimization of speech quality and training efficiency.

[0060] In the second aspect, the application provides a computer readable storage medium, which stores a computer program / instruction, and characterized in that the computer program / instruction is executed by a processor to realize the steps of the audio synthesis method based on the improved VITS model in the first aspect.

[0061] Compared with the prior art, the application has the following beneficial effects:

[0062] (1) The speech synthesis method based on the improved VITS model provided by the application solves the technical defects of the traditional speech synthesis model VITS, such as poor flexibility and insufficient naturalness of the parameter synthesis model, by improving and adding the loss function of the traditional speech synthesis model VITS.

[0063] (2) The application improves the reconstruction loss function of the speech synthesis model VITS and optimizes it into a loss function system with mel spectrum as the reconstruction target, thereby reducing the dependence on waveform domain reconstruction, improving the capture ability of the improved training model for speech spectrum characteristics, and reducing the computational complexity; the discriminator loss function, the generator loss function and the multi-scale feature matching loss function are improved, thereby improving the generation ability of the generator for high-fidelity speech and enhancing the sensitivity of the discriminator to detailed features. By constraining the global consistency of the generator output, the mode collapse problem in the adversarial training is effectively suppressed.

[0064] (3) The present application improves the loss function of the random duration predictor, models through reversible transformation and attention alignment matrix, improves the generation robustness in multi-sound word and complex tone scene, solves the common intermittent and stuttering problem in traditional method. By introducing the bidirectional KL divergence constraint of the hidden variable, the semantic coupling relationship between text and speech is strengthened, so that the generated speech has a significant improvement in phoneme alignment accuracy and prosody naturalness compared with the traditional speech synthesis model VITS. KL divergence is an index for measuring the difference between two probability distributions, commonly known as relative entropy or information divergence in Chinese. It was proposed by two statisticians, Solomon Kullback and Richard Leibler, so it is named after them.

[0065] (4) The present application proposes an adaptive hyperparameter tuning algorithm based on Lagrange multiplier method to solve the pain points of high training cost and experience-based trial and error of super parameter tuning in traditional multi-stage speech synthesis system. By decoupling the reconstruction loss from other loss terms and introducing dynamic constraints, the Pareto optimal equilibrium point is automatically optimized.

[0066] (5) The present application converts the super parameter search process in the traditional fixed weight method, which needs to be manually traversed, into a mathematical constraint optimization problem, that is, instead of directly trying different balance hyperparameters, a specific target value of the spectral domain reconstruction loss function is set to find the optimal balance, so that the model can reach the same speech quality indicators in a shorter training time. At the same time, it supports fast adaptation in multi-speaker and multi-language scenarios, significantly reducing the deployment cost of telecom operators.

[0067] (6) The present application realizes a breakthrough effect of 12% improvement in speech synthesis MOS index and 20% reduction in response delay in the intelligent customer service scene of telecommunications. Speech synthesis MOS index: MOS (Mean Opinion Score) is a subjective speech quality evaluation index, which is used to measure the naturalness and acceptability of the generated speech by the speech synthesis system. Compared with traditional technology, the adaptive speech synthesis model AdaVITS based on the traditional speech synthesis model VITS can provide clearer, more natural and accurate speech synthesis effect, significantly improving the user interaction experience. It effectively solves the problems of mechanical feeling and poor adaptability of traditional speech synthesis systems in the telecommunications scenario, provides key technical support for operators to build intelligent and personalized voice service systems, and has significant commercial value and social and economic benefits. BRIEF DESCRIPTION OF DRAWINGS

[0068] Figure 1 The figure shows a speech synthesis method flowchart based on the VITS model improvement in an embodiment of the present application. DETAILED DESCRIPTION

[0069] Further description is made in combination with the drawings and specific embodiments. In the description of the present application, it is to be understood that the terms "first", "second", etc. are only for the purpose of description and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the technical features indicated. Therefore, the features defined with "first", "second", etc. can be explicitly or implicitly included one or more of the features.

[0070] Embodiment 1 The present embodiment provides an audio synthesis method based on an improved VITS model, including the following contents:

[0071] Obtaining the text of the audio data to be synthesized, and preprocessing the text;

[0072] Inputting the preprocessed text into a pre-trained adaptive speech synthesis model AdaVITS to perform audio synthesis;

[0073] Obtaining generated audio data according to the output of the adaptive speech synthesis model AdaVITS;

[0074] Wherein, the adaptive speech synthesis model AdaVITS is based on a speech synthesis model VITS, and a joint loss function of the adaptive speech synthesis model AdaVITS is obtained by improving and adding a loss function of the speech synthesis model VITS; the joint loss function is optimally solved to realize collaborative optimization of speech quality and training efficiency;

[0075] Wherein, the improved joint loss function is obtained by the following formula:

[0076] ,

[0077] In the formula, is a joint loss function, is a spectrum domain reconstruction loss function, is a discriminator loss function, is a generator loss function, is a multi-scale feature matching loss function, is a random duration predictor loss function, is a forward KL divergence regularization loss function, is a reverse KL divergence regularization loss function, and are dynamic adjustment KL divergence weight coefficients.

[0078] Embodiment 2 Based on embodiment 1, the present embodiment introduces a specific implementation process of an audio synthesis method based on an improved VITS model, as shown in Figure 1 specifically including the following contents:

[0079] In one specific embodiment of the present application, the data set for training includes audio data and text data corresponding to the audio data.

[0080] The preprocessing of the data set for training includes preprocessing of the audio data and preprocessing of the text data.

[0081] The preprocessing of the audio data includes cleaning the audio data, removing abnormal audio (such as invalid signals, blank audio) and low-quality audio, detecting and cropping silent segments, ensuring valid speech content, and denoising and enhancing the audio data to obtain preprocessed audio data; reducing background noise by existing technology and related denoising algorithms (such as spectral subtraction), and performing equalization processing on the speech signal to improve the signal-to-noise ratio and clarity.

[0082] The preprocessing of the text data includes cleaning the text data, removing special symbols, stop words, and meaningless content, unifying the text format, and normalizing the text data by converting temporal, special symbol, and date information into uniform and standardized vocabulary to obtain preprocessed text data. (For example, converting the time representation "22:30" to "twenty-two thirty", converting the unit "kg" to "kilogram", and converting the number "10086" to "one thousand eighty-six" or "one zero zero eight six").

[0083] The preprocessed text data is converted into a phoneme sequence, prosodic prediction is performed by existing technology and related models such as the Bidirectional Encoder Representations from Transformers (BERT) based on the Transformer, and the complete phoneme sequence is obtained by splicing according to the related vocabulary, for example:

[0084] Text data: Today the weather is really good

[0085] Prosodic prediction: today #0 day #1 day #0 q #1 really #0 good #0 ah #4

[0086] Final phoneme sequence: sil j in1 #0 t ian1 #1 t ian1 #0 q i4 #1 zh en1 #0 h ao3#0 ^ a5 #4.

[0087] The preprocessed audio data and its corresponding phoneme sequence together constitute the preprocessed data set for training.

[0088] In an embodiment of the present application, the spectrum domain reconstruction loss function is calculated by the following formula:

[0089] ,

[0090] In the formula, represents the target mel spectrum, represents the predicted mel spectrum, represents the L1 norm.

[0091] The reconstruction loss of the traditional speech synthesis model VITS is usually directly aimed at the original waveform domain, but the present application proposes to take the mel spectrum as the reconstruction target. This method calculates the L1 loss between the predicted value and the target value by decoding the hidden variable into the waveform domain and then converting it into the mel spectrum domain . This improvement significantly reduces the dependence on waveform domain reconstruction, improves the model's ability to capture speech spectral features, and reduces computational complexity.

[0092] In an embodiment of the present application, the discriminator loss function and the generator loss function are calculated by the following formulas, respectively:

[0093] ,

[0094] ,

[0095] In the formula, represents the real speech data, represents the hidden variable, represents the output probability of the discriminator D, represents the output probability of the generator G, is the mathematical expectation of the joint distribution of the training set speech data and the hidden variable , is the mathematical expectation of the distribution of the hidden variable .

[0096] In this embodiment, based on the speech synthesis model VITS, the dual target loss function is improved, respectively for the discriminator and the generator . The discriminator loss is achieved by maximizing the discrimination ability between real data and generated data, while the generator loss is optimized by minimizing the discrimination difference between generated data and real data.

[0097] In a specific embodiment of the embodiment of the present application, the multi-scale feature matching loss function is calculated by the following formula:

[0098] ,

[0099] In the formula, represents real speech data, represents a latent variable, represents the output probability of the discriminator, represents the output probability of the generator, represents the L1 norm, represents the intermediate feature output probability of the first layer of the discriminator, represents the intermediate feature output probability of the first layer of the discriminator, represents the feature dimension, represents the total number of layers of the discriminator.

[0100] To alleviate the imbalance between the generator G and the discriminator D, the multi-scale feature matching loss function of the traditional speech synthesis model VITS is improved in the present application, and the core idea is to match the feature distribution of the output of the generator G with the real data in the intermediate layer of the discriminator D. The loss only acts on the generator G. The improvement effectively suppresses the mode collapse problem in training by constraining the global consistency of the generator output.

[0101] In a specific embodiment of the embodiment of the present application, the random duration predictor loss function is calculated by the following formula:

[0102] ,

[0103] In the formula, represents the linear spectrum, represents the text encoding, represents the alignment matrix, represents the posterior distribution of the given input feature , represents the conditional likelihood term of the adaptive speech synthesis model AdaVITS, specifically the probability of reconstructing the mel spectrum after giving the latent variable (representing the implicit encoding of phoneme duration), represents the posterior distribution, specifically the distribution of the latent variable after giving the text encoding (such as the embedding representation of phonemes) and the alignment matrix (the alignment relationship between text phonemes and speech frames).

[0104] To avoid gradient interference, the gradient backpropagation of the random duration predictor is explicitly cut off, and the phoneme duration is mapped to an integer value from the random noise sample through an invertible transformation. This design significantly improves the robustness of duration prediction, especially in multi-speaker scenarios, and enhances the stability of phoneme alignment.

[0105] In an embodiment of the present application, the forward KL divergence regularization loss function is calculated by the following formula:

[0106] ,

[0107] ,

[0108] In the formula, is the mean vector of the posterior distribution, is the standard deviation vector of the posterior distribution, is the input feature, denotes the posterior distribution of the given input feature , denotes the prior distribution of the latent variable under the given condition , denotes the alignment matrix;

[0109] The reverse KL divergence regularization loss function is calculated by the following formula:

[0110] ,

[0111] In the formula, denotes the reconstructed latent variable.

[0112] The reverse KL divergence regularization loss function balances the complexity of the posterior distribution and the prior distribution, making the forward and reverse KL divergence regularization loss functions more suitable for actual application scenarios. This improvement enhances the modeling ability of the adaptive speech synthesis model AdaVITS for text-speech coupling relationships, significantly improving the naturalness of speech generation.

[0113] In an embodiment of the present application, the optimal solution of the joint loss function includes obtaining a target function by target decoupling, defining a constraint condition, constructing a Lagrange function and dynamically updating the mechanism to obtain the parameters of the updated adaptive speech synthesis model AdaVITS. The parameters of the updated adaptive speech synthesis model AdaVITS make the target function minimum and satisfy the constraint condition, thereby obtaining the optimal solution of the joint loss function.

[0114] In an embodiment of the present application, the target decoupling includes defining a spectrum domain reconstruction loss function separately from other loss functions in the joint loss function and a constraint condition :

[0115] ,

[0116] ,

[0117] wherein is a preset target value of the spectrum domain reconstruction loss function , i represents a loss function type, is a parameter of the adaptive speech synthesis model AdaVITS. is usually a converged value obtained when the HiFi-GAN vocoder in the adaptive speech synthesis model AdaVITS is independently trained .

[0118] The construction of the Lagrange function includes introducing a penalty term , constructing a Lagrange function , and the formula is as follows:

[0119] ,

[0120] wherein is a Lagrange multiplier, is a damping constant;

[0121] The dynamic updating mechanism includes updating , updating by using the updated , and the formula is as follows:

[0122] ,

[0123] .

[0124] The updated parameter of the adaptive speech synthesis model AdaVITS can make the target function minimum while satisfying the constraint condition Let , the adaptive speech synthesis model AdaVITS automatically finds a Pareto optimal equilibrium point, thereby obtaining an optimal solution of the joint loss function and realizing collaborative optimization of speech quality and training efficiency. In the embodiment, the spectrum domain reconstruction loss function embodies speech quality, and the spectrum domain reconstruction loss function Other loss functions besides these reflect the training efficiency of the adaptive speech synthesis model AdaVITS.

[0125] In practical applications, Set as The reconstruction loss can then be found. The method seeks to find the optimal trade-off between hyperparameter search and other losses, thereby discovering a model configuration that maintains good speech quality while also achieving good performance in other aspects. This approach significantly reduces the trial-and-error cost of hyperparameter search while improving the model's generalization ability in complex tasks such as multi-speaker and emotion synthesis.

[0126] In optimization problems, the Pareto equilibrium point refers to the optimal point reached when, among multiple objective functions, it is impossible to further improve one objective without sacrificing another. For the speech synthesis model VITS, there exists a spectral domain reconstruction loss function. A trade-off between other losses. During training, the loss function is reconstructed by changing the spectral domain. Depending on the balance between the VITS speech synthesis model and other losses, the VITS model converges to different Pareto equalization points, and the performance of the VITS speech synthesis model varies significantly depending on which Pareto equalization point it converges to. This is especially true for models with balancing hyperparameters. Reconstruction losses The traditional method for finding the optimal solution is to consider different equilibrium hyperparameters. Iterative experiments can be conducted, but this method is very time-consuming and resource-intensive.

[0127] Therefore, this invention proposes the above method, which is not to directly try different equilibrium hyperparameters. Instead, a specific reconstruction loss is set. The goal is to find the optimal balance by setting a target value.

[0128] The decoder part of the VITS speech synthesis model is actually a HiFi-GAN, responsible for converting the intermediate representation back into the audio signal. If the HiFi-GAN performs well when trained independently as a vocoder, then its spectral domain reconstruction loss function... This likely represents an ideal level. Therefore, training HiFi-GAN alone for only the vocoder task yields a convergent reconstruction loss. target value Then, during the training of the adaptive speech synthesis model AdaVITS, we tried to adjust the reconstruction loss. converged to Thus, the spectral domain reconstruction loss function can be found. The optimal solution of the joint optimization function is obtained, and the collaborative optimization of the speech quality and the training efficiency is realized.

[0129] Embodiment 3 This embodiment introduces a computer readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps of the audio synthesis method based on the VITS model improved as introduced in embodiment 1 or 2 are implemented.

[0130] The embodiments of the present application are described above with reference to the drawings; however, the present application is not limited to the specific embodiments described above, and the specific embodiments described above are merely illustrative rather than restrictive, and a person of ordinary skill in the art can make many forms under the inspiration of the present application without departing from the purpose of the present application and the scope protected by the claims, which are all within the protection of the present application.

Claims

1. An audio synthesis method based on an improved VITS model, characterized in that, include: Obtain the text of the audio data to be synthesized, and preprocess the text; The preprocessed text is input into the pre-trained adaptive speech synthesis model AdaVITS for audio synthesis; The generated audio data is obtained based on the output of the adaptive speech synthesis model AdaVITS. The adaptive speech synthesis model AdaVITS is based on the speech synthesis model VITS. The loss function of the speech synthesis model VITS is improved and added to obtain the joint loss function of the adaptive speech synthesis model AdaVITS. The joint loss function is optimally solved to achieve synergistic optimization of speech quality and training efficiency. The improved joint loss function is calculated using the following formula: ， In the formula, For the joint loss function, Reconstruct the loss function in the spectral domain. Let the discriminator loss function be... Let be the generator loss function. For multi-scale feature matching loss function, Let the loss function be that of the random duration predictor. The loss function is the positive KL divergence regularization function. The inverse KL divergence regularization loss function is... and All are dynamically adjusted KL divergence weight coefficients.

2. The audio synthesis method based on the improved VITS model according to claim 1, characterized in that, The dataset used for training includes audio data and the corresponding text data. Preprocessing the dataset used for training includes preprocessing the audio data and preprocessing the text data; Preprocessing the audio data includes cleaning the audio data, removing abnormal and low-quality audio, detecting and cropping silent segments, and denoising and enhancing the audio data to obtain preprocessed audio data. Preprocessing the text data includes cleaning the text data, removing special symbols, stop words and meaningless content, unifying the text format, and standardizing the text data by converting time, special symbols and dates into uniform vocabulary, thus obtaining preprocessed text data.

3. The audio synthesis method based on the improved VITS model according to claim 1, characterized in that, The spectral domain reconstruction loss function is calculated using the following formula: ， In the formula, Represents the target Mel spectrum. This represents the predicted Mel spectrum. This represents the L1 norm.

4. The audio synthesis method based on the improved VITS model according to claim 1, characterized in that, The discriminator loss function and generator loss function are respectively calculated using the following formulas: ， ， In the formula, Represents real voice data. Represents latent variables. This represents the output probability of the discriminator. This represents the output probability of the generator. To analyze the training set speech data and latent variables The joint distribution takes the expected value. For latent variables The distribution takes the expected value.

5. The audio synthesis method based on the improved VITS model according to claim 1, characterized in that, The multi-scale feature matching loss function is calculated using the following formula: ， In the formula, This represents the speech data in the training set. Represents latent variables. This represents the output probability of the discriminator. This represents the output probability of the generator. Describing the L1 norm, Indicates the discriminator's first... The intermediate feature output probability of the layer Representing feature dimension, This indicates the total number of discriminator layers.

6. The audio synthesis method based on the improved VITS model according to claim 1, characterized in that, The loss function of the random duration predictor is calculated using the following formula: ， In the formula, Represents the linear spectrum. Indicates text encoding. Represents the alignment matrix. Represents the given input features The posterior distribution, This represents the conditional likelihood term of the AdaVITS adaptive speech synthesis model, specifically representing the conditional likelihood term given the latent variables. Then, the Mel spectrum was reconstructed. The probability, Represents the posterior distribution, specifically representing the encoding of a given text. and alignment matrix Afterwards, latent variables The distribution of .

7. The audio synthesis method based on the improved VITS model according to claim 1, characterized in that, The positive KL divergence regularization loss function is calculated using the following formula: ， ， In the formula, Let be the mean vector of the posterior distribution. Let be the standard deviation vector of the posterior distribution. For input features, Represents the given input features The posterior distribution, Indicates a given condition Latent variables The prior distribution, Represents the alignment matrix; The inverse KL divergence regularization loss function is calculated using the following formula: ， In the formula, This represents the hidden variables after reconstruction.

8. The audio synthesis method based on the improved VITS model according to claim 1, characterized in that, The optimal solution for the joint loss function includes obtaining the objective function through objective decoupling, defining constraints and constructing a Lagrange function and a dynamic update mechanism to obtain the updated parameters of the adaptive speech synthesis model AdaVITS, and using the updated parameters of the adaptive speech synthesis model AdaVITS to minimize the objective function and satisfy the constraints, thereby obtaining the optimal solution for the joint loss function.

9. The audio synthesis method based on the improved VITS model according to claim 8, characterized in that, The target decoupling includes reconstructing the spectral domain loss function. Separate from the other loss functions in the joint loss function, define the objective function. and constraints : ， ， In the formula, For the preset spectrum domain reconstruction loss function The target value, where i represents the type of loss function. The parameters of the adaptive speech synthesis model AdaVITS; The construction of the Lagrange function includes introducing a penalty term. Construct the Lagrange function The formula is as follows: ， In the formula, For Lagrange multipliers, Here is the damping constant; The dynamic update mechanism includes updating. Through the updated renew The formula is as follows: ， 。 10. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the audio synthesis method based on the VITS model improvement as described in any one of claims 1-9.

Images