A phoneme duration prediction method based on VITS neural spline flow improvement

Abstract

The application discloses a phoneme duration prediction method based on VITS neural spline flow improvement, introduces a method based on neural spline flow by improving the random duration predictor in the VITS model, effectively improves the accuracy of phoneme duration prediction, through detailed analysis of input data distribution and block interval division, the model can perform more effective coupling transformation in each block interval, which enables the model to predict the duration of phonemes with higher accuracy in actual speech synthesis applications. In addition, the application can speed up the training process and focus on optimizing the performance of the duration prediction module by freezing the model, this method not only improves the training efficiency, but also ensures the accuracy of the model in predicting the duration of phonemes, and the mean and variance of the error are used as evaluation indexes to further verify the prediction accuracy of the model, and the transformation interval boundary value is optimized by adjusting the hyperparameter B, realizing the optimization of the performance of the model.

Description

Technical Field

[0001] This invention belongs to the field of speech synthesis technology, specifically relating to a phoneme duration prediction method based on VITS neural spline flow improvement. Background Technology

[0002] Currently, deep learning-based speech synthesis methods are mainly divided into autoregressive and non-autoregressive methods. Autoregressive methods are among the earliest generative models used in speech synthesis, with major models including Recurrent Neural Networks (RNNs) and Long Short-Term Memory Networks (LSTMs). Non-autoregressive methods are a relatively new research direction. They attempt to generate the entire sequence in parallel, without relying on previously generated content. This approach can significantly improve generation speed, making it particularly suitable for real-time applications. However, because non-autoregressive methods do not depend on previously generated content and instead use phoneme duration prediction to determine the total length of the final speech waveform data, they may face challenges in maintaining contextual consistency, potentially leading to relatively poor generation results. In early speech synthesis systems, duration information was typically obtained through manual annotation or rule-based methods, such as statistical model methods like Hidden Markov Models and decision trees, to predict phoneme duration. The problem with this approach is that it may not adapt to changes in different speakers, contexts, or emotions, resulting in unnatural speech synthesis performance. Recently, end-to-end non-autoregressive speech synthesis systems have become increasingly popular, including end-to-end duration prediction. This means that the entire speech synthesis process, from text to speech, can be performed using a unified model, such as Tacotron, FastSpeech, and VITS (Variational Inference with adversarial learning for end-to-end Text-to-Speech).

[0003] Chinese patent application CN102222501A proposes a method for predicting speech synthesis duration parameters. It generates state duration parameters based on Hidden Markov Model (HMM) speech synthesis. For the input context-dependent HMM sequence, it generates the duration of each state of each model, i.e., the dwell time of each state. When generating state duration parameters, it combines the state duration model and the overall duration variance model from the HMM. It creates a training corpus for the overall duration variance model, uses an overall duration variance analyzer to generate training samples from the training corpus, and uses these training samples to train the overall duration variance model.

[0004] Chinese patent application CN102231276 A proposes a phoneme duration prediction method based on linear regression using a decision tree-Gaussian mixture model. This method employs a stepwise linear regression duration prediction model for contextual parameters to initially predict the duration of speech synthesis units, obtaining initial duration prediction results. The initial duration prediction results are then allocated using a decision tree-Gaussian mixture model to obtain allocated duration prediction results. This improves the accuracy of duration prediction results, enabling the synthesized speech from the speech synthesis system to possess a realistic sense of rhythm.

[0005] The methods employed by the two patented technologies mentioned above require a large amount of speech data for training to achieve good model generalization ability, which is costly. Furthermore, as the corpus expands and the phoneme sequence grows, the training and inference speed decreases significantly, failing to meet the needs of real-time speech generation. Summary of the Invention

[0006] In view of the above, the present invention provides a phoneme duration prediction method based on VITS neural spline flow improvement. It proposes an improvement to the neural spline flow in the random duration predictor used by VITS, which can couple the data transformation capability of the transformation function, thereby improving the accuracy of phoneme duration prediction.

[0007] A phoneme duration prediction method based on VITS neural spline flow improvement includes the following steps:

[0008] (1) Statistically analyze the distribution of input data for neural spline flow;

[0009] (2) Divide the transformation interval of the input data into multiple block intervals, determine the boundary of each block interval, and allocate the number of sub-blocks to each block interval;

[0010] (3) By freezing the training of the VITS model, the training process of the neural spline flow is accelerated, thereby using the neural spline flow to predict the phoneme duration and generate audio by the model.

[0011] (4) The predictive performance of neural spline flow is evaluated by duration prediction error analysis;

[0012] (5) Evaluate the quality of the audio generated by the model and the time and space complexity of the model algorithm.

[0013] Furthermore, in step (1), during the training and inference phases of the VITS model, the frequency of data occurrence in the nonlinear transformation portion of the neural spline flow input data within the transformation interval is statistically analyzed. Specifically, the transformation interval is evenly divided into 100 sub-intervals, and the frequency of data occurrence in each sub-interval is statistically analyzed.

[0014] Furthermore, the duration prediction module in the VITS model is implemented based on neural spline flow, which is a normalized flow generation model. It uses spline interpolation to replace the affine transformation function with a monotone rational quadratic function to achieve more complex data transformation, thereby enhancing the flexibility of its coupled transformation function and improving the model's generation capability. Thus, the VITS model is applied to the duration prediction module in speech synthesis to achieve highly accurate phoneme duration prediction.

[0015] Furthermore, in step (2), the hyperparameter B and the number of blocks in the transformation interval are first determined according to the distribution of the input data. The hyperparameter B is used to determine the transformation interval of the input data, i.e. the interval boundary value. Then, the entire transformation interval is divided into multiple block intervals, and the boundary of each block interval is determined through data analysis, so that the coupling transformation in each block interval is more effective.

[0016] Furthermore, the entire transformation interval of the neural spline flow input data is divided into N sub-blocks with a total of N+1 nodes. Interpolation is performed based on these nodes, with the nodes at both ends of the edge fixed. The model predicts the width, height, and first-order derivative values ​​of the N-1 nodes in the middle of each sub-block. Thus, step (2) allocates the corresponding number of sub-blocks in each block interval according to the actual data requirements to meet the requirements of nonlinear transformation.

[0017] Furthermore, in step (3), the pre-trained model of VITS is loaded first. Since the neural spline flow in the duration prediction module has changed, the parameters of the duration prediction module are not loaded. After loading the parameters of other modules, these parameters are frozen and not used for optimization. Then, the model is trained using the LJSpeech dataset, and only the parameters of the duration prediction module are optimized, thereby improving the training efficiency of the model.

[0018] Further, the specific implementation of step (4) is as follows: after the model training is completed, the accuracy of the model’s duration prediction is first verified by calculating the mean and variance of the error between the predicted duration value and the true value, where the true duration value comes from the value calculated by the monotonic alignment search during the model training process; then, by comparing the duration prediction error under different hyperparameters B, the boundary value of the transformation interval that can achieve the best performance is determined.

[0019] Furthermore, in step (5), subjective opinion scores are used as a verification index to evaluate the quality of the audio generated by the model. The subjective opinion scores are calculated using the deep learning model NISQA. In addition, the time and space complexity of the entire model algorithm are comprehensively evaluated by calculating real-time factors and parameters.

[0020] This invention improves the accuracy of phoneme duration prediction by introducing a neural spline flow-based method to the stochastic duration predictor in the VITS model. Through detailed analysis of the input data distribution and block division, the model can perform more effective coupling transformations within each block, enabling it to predict phoneme durations with higher accuracy in practical speech synthesis applications. Furthermore, this invention employs frozen training, freezing parameters other than the duration prediction module during training. This accelerates the training process and allows for focused optimization of the duration prediction module's performance. This method not only improves training efficiency but also ensures the model's accuracy in predicting phoneme durations. The mean and variance of the error are used as evaluation metrics to further verify the model's prediction accuracy, and the transformation interval boundary values ​​are optimized by adjusting the hyperparameter B, achieving optimal model performance. Ultimately, by comprehensively evaluating the quality, time complexity, and space complexity of the audio generated by the model, this invention ensures improved accuracy without significant changes in efficiency and resource consumption. This comprehensive evaluation includes using subjective opinion scores and the deep learning model NISQA to assess audio quality, as well as calculating real-time factors and parameter counts to evaluate the complexity of the model algorithm. Experimental results show that this invention can improve the quality of the synthesized audio. Attached Figure Description

[0021] Figure 1 This is a schematic diagram of the improved architecture of the phoneme duration prediction method of the present invention.

[0022] Figure 2 (a) and Figure 2 (b) are schematic diagrams of the input data distribution during the training and inference phases of the VITS model.

[0023] Figure 3 This is a schematic diagram of the rational quadratic transformation function for neural spline flow.

[0024] Figure 4 This is a schematic diagram of a rational quadratic transformation function with 10 sub-blocks.

[0025] Figure 5 This is a schematic diagram of the rational quadratic transformation function after block division. Detailed Implementation

[0026] To describe the present invention in more detail, the technical solution of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0027] The VITS stochastic duration predictor is based on neural spline flow, a normalized flow generation model that uses spline interpolation to replace affine transformation functions with monotone rational quadratic functions to achieve more complex transformations. This enhances the model's generation capability by increasing the flexibility of its coupled transformation functions. This invention applies VITS to the duration prediction module of a speech synthesis model and proposes improvements to its neural spline flow. Since the nonlinear transformation capability of coupled transformation functions is particularly important in flow models, this invention improves the algorithm by dividing the transformation function into blocks to enhance its transformation capability, thereby achieving highly accurate phoneme duration prediction.

[0028] like Figure 1 As shown, the phoneme duration prediction method based on VITS neural spline flow improved in this invention includes the following steps:

[0029] (1) For different datasets, it is necessary to analyze their data distribution and then determine the boundary intervals and the number of blocks.

[0030] This invention, during the training and inference phases of the VITS model, identifies the main concentrated regions of input data by statistically analyzing the input data distribution of neural spline flows, particularly within the data space of the LJSpeech dataset. Specifically, it statistically analyzes the nonlinear transformation component x2 in the input x:[x1,x2] of the normalized flow model, achieving data distribution statistics by statistically analyzing the frequency of data occurrences in the interval [-5,5]. This involves dividing [-5,5] into 100 uniform intervals and statistically analyzing the frequency of data appearing in each interval. Using the LJSpeech dataset, the distribution of the input data is as follows: Figure 2 (a) and Figure 2 As shown in (b), the data is mainly concentrated in the interval [-3,3].

[0031] (2) Determine the number of blocks in the entire transformation interval for effective transformation of neural spline flow.

[0032] In the VITS model, the hyperparameter B is used to determine the transformation interval of the input data. First, B is set as the interval boundary. A suitable value for B is determined by analyzing the distribution of the input data. Then, the entire interval [-B, B] is divided into M blocks [-B, B1], [B1, B2], [B2, B3]…[B… ...…B…B…B…B……B…B…B……B…B……B……B M-1 [B], M≥2, through detailed data analysis, the boundaries B1, B2, ..., B of each block are determined. M-1 The value chosen should not be close to the region where the data distribution is concentrated, so that the coupling transformation within each block is more effective; if the data distribution is biased towards a uniform distribution, then take M=1, that is, a range block.

[0033] Neural spline flow determines the domain and range of the interpolation function by setting boundaries, and the transformation function is as follows: Figure 3 As shown, B is a hyperparameter, i.e., the interval boundary. By analyzing the distribution of the input data, there is almost no data outside the interval [-5,5], very little data in the interval [-5,-3]∪[3,5], and a lot of data in the interval [3,3]. Therefore, B can be set to 5, and the entire interval is divided into 3 blocks: [-5,-3], [3,3], and [3,5]. The original spline flow can be considered as having only one block. The boundary setting is the same as the neural spline flow used in VITS, with Bound set to 5. Since a complex transformation function is needed in the [-3,3] interval, the central interval boundary value is set to 3.

[0034] (3) Determine the allocation of the number of sub-blocks within each interval for the generation of effective interpolation functions for neural spline flows. Within each interval block, determine the number of sub-blocks based on actual data requirements, i.e., each interval is divided into blocks. M The number of interpolation points t in M The corresponding number of sub-blocks k M For t M +1; Determine the appropriate number of interpolation points based on the data distribution of each interval block to ensure sufficient transformation capability for each interval; Consider the special characteristics of the data in the edge region and adjust the number of sub-blocks to ensure the accuracy of the transformation.

[0035] like Figure 4 As shown, the original neural spline flow was divided into 10 sub-blocks, interpolated based on 11 points, with two edge points fixed. The model predicts the width, height, and first derivative values ​​of the 9 middle points of each sub-block. Therefore, after dividing the [-5,5] interval into 3 blocks, it is necessary to determine the number of sub-blocks in each interval block. If the number of sub-blocks in an interval block is 1, then the corresponding transformation is a linear transformation. Since there is some data in the [-5,-3]∪[3,5] interval, the linear transformation cannot meet its transformation requirements. Therefore, the number of sub-blocks in the edge interval blocks is set to 2. Thus, the number of sub-blocks in the three interval blocks are 2, 6, and 2 respectively. Figure 5 As shown.

[0036] (4) Accelerate the training process of the neural spline flow model by freezing the training model.

[0037] After loading the VITS pre-trained model based on the LJSpeech dataset, this invention does not load the parameters of the duration prediction module because the network of the duration prediction module has changed. After loading the parameters of other modules (decoder, prior encoder, posterior encoder), the parameters that are unrelated to the neural spline flow are frozen and do not participate in parameter optimization. Only the parameters of the duration prediction module are optimized. Due to the low coupling between the duration prediction module and other modules, this can greatly speed up the model training.

[0038] (5) The performance of neural spline flow is evaluated by using the duration prediction error analysis method.

[0039] After training, the accuracy of the model's duration prediction is verified by calculating the mean and variance of the error between the predicted duration and the true duration. The true duration is derived from the value calculated by the monotonic alignment search during the VITS training process.

[0040] Equation (1) is based on the predicted value The duration error l for a single sample is calculated from the true value d, where n is the dimension of the duration vector, and l dur It is the average value of the time error.

[0041]

[0042] Equations (2) and (3) respectively calculate the average error of all samples. and variance Where t is the total number of samples.

[0043]

[0044]

[0045] Meanwhile, by comparing the duration prediction errors under different boundary parameters B, we determined the specific value of the intermediate interval boundary that can achieve the best performance.

[0046] Table 1

[0047]

[0048] Table 1 presents the comparison results of duration prediction errors between block spline flow and neural spline flow with different boundaries. The experimental results show that the original duration prediction module has a high prediction error when B is too small. When the boundary of the middle interval is 3, the duration prediction error of the block neural spline flow is smaller than that of the original duration predictor. The mean error is reduced by about 0.02 and the variance is reduced by about 0.07. This indicates that the block spline flow has a stronger ability to perform regional nonlinear transformations in the dataset.

[0049] (6) The quality of audio generated by neural spline flow is comprehensively evaluated by using subjective opinion scores and other indicators.

[0050] This invention uses subjective opinion scores to evaluate the quality of the final generated audio. These subjective opinion scores are calculated using a deep learning-based speech quality assessment method, specifically the NISQA model, which evaluates subjective opinion scores as NISQA subjective opinion scores. Furthermore, this invention comprehensively considers the time and space complexity of the method through real-time factor and parameter calculations.

[0051] Table 2

[0052]

[0053] Table 2 presents a comparison of subjective opinion scores, real-time factors, and parameter counts after using block neural spline flow. The experimental data shows that the model is more accurate in duration prediction after using block neural spline flow, thus improving the final subjective opinion score.

[0054] The above description of the embodiments is provided to enable those skilled in the art to understand and apply the present invention. Those skilled in the art can readily make various modifications to the above embodiments and apply the general principles described herein to other embodiments without creative effort. Therefore, the present invention is not limited to the above embodiments, and any improvements and modifications made to the present invention by those skilled in the art based on the disclosure thereof should be within the scope of protection of the present invention.

Claims

1. A phoneme duration prediction method based on VITS neural spline flow improvement, comprising the following steps: (1) Statistically analyze the distribution of input data of neural spline flow. Specifically: during the training and inference phases of the VITS model, the frequency of data occurrence in the nonlinear transformation part of the neural spline flow input data in the transformation interval is statistically analyzed. The transformation interval is evenly divided into 100 sub-intervals, and then the frequency of data occurrence in each sub-interval is statistically analyzed. (2) Divide the transformation interval of the input data into multiple block intervals, determine the boundary of each block interval, and allocate the number of sub-blocks to each block interval. Specifically: First, determine the hyperparameter B in the VITS model and the number of blocks in the transformation interval. The hyperparameter B is used to determine the transformation interval of the input data, i.e., the interval boundary value. Then, divide the entire transformation interval into multiple block intervals according to the distribution of the input data, and then determine the boundary of each block interval through data analysis. (3) By freezing the training of the VITS model, the training process of the neural spline flow is accelerated, thereby using the neural spline flow to predict the phoneme duration and generating audio by the model; (4) Evaluate the predictive performance of neural spline flow through duration prediction error analysis; (5) Evaluate the quality of the audio generated by the model and the time and space complexity of the model algorithm.

2. The phoneme duration prediction method according to claim 1, characterized in that: The duration prediction module in the VITS model is based on neural spline flow, which is a normalized flow generation model. It uses spline interpolation to replace the affine transformation function with a monotone rational quadratic function to achieve more complex data transformation, thereby enhancing the flexibility of its coupled transformation function and improving the model's generation capability. Thus, the VITS model is applied to the duration prediction module in speech synthesis to achieve highly accurate phoneme duration prediction.

3. The phoneme duration prediction method according to claim 1, characterized in that: The entire transformation interval of the neural spline flow input data is divided into N sub-blocks with a total of N+1 nodes. Interpolation is performed based on these nodes, with the nodes at both ends of the edge fixed. The model predicts the width, height, and first-order derivative values ​​of the N-1 nodes in the middle of each sub-block. Thus, step (2) allocates the corresponding number of sub-blocks in each block interval according to the actual data requirements to meet the requirements of nonlinear transformation.

4. The phoneme duration prediction method according to claim 1, characterized in that: In step (3), the VITS pre-trained model is loaded first. Since the neural spline flow in the duration prediction module has changed, the parameters of the duration prediction module are not loaded. After loading the parameters of other modules, these parameters are frozen and not used for optimization. Then, the model is trained using the LJSpeech dataset, and only the parameters of the duration prediction module are optimized, thereby improving the training efficiency of the model.

5. The phoneme duration prediction method according to claim 1, characterized in that: The specific implementation of step (4) is as follows: After the model training is completed, the accuracy of the model’s duration prediction is verified by first calculating the mean and variance of the error between the predicted duration value and the true value, where the true duration value comes from the value calculated by the monotonic alignment search during the model training process; then, by comparing the duration prediction error under different hyperparameters B, the boundary value of the transformation interval that can achieve the best performance is determined.

6. The phoneme duration prediction method according to claim 1, characterized in that: In step (5), subjective opinion scores are used as a verification index to evaluate the quality of the audio generated by the model. The subjective opinion scores are calculated using the deep learning model NISQA. In addition, the time and space complexity of the entire model algorithm are comprehensively evaluated by calculating real-time factors and parameters.

Images