Speech synthesis method and system based on improved vits

Abstract

The application provides a speech synthesis method and system based on an improved VITS, wherein a large language model is introduced by optimizing a text encoder of a VITS model, so that the input text can capture the emotion, intention and speaking style when the text is encoded; a random disturbance term is introduced when a Q value is solved by dynamic programming, so that the alignment flexibility in the early training stage is improved, and the monotonicity constraint is strictly maintained to avoid early convergence to a suboptimal solution; a ConvNeXt module is used as a basic backbone network of a decoder, and ISTFT is used to efficiently reconstruct a time domain signal, so that the upsampling of the waveform is realized, the redundant calculation of a traditional transpose convolution is avoided, and the inference is accelerated. The application can effectively improve the inference speed, emotion expression ability and flexibility of style control of speech synthesis, provides a new solution for cross-language diversified speech synthesis, and provides a reference for the development of speech synthesis technology in a more efficient and intelligent direction.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of speech synthesis, in particular to a speech synthesis method and system based on VITS improvement. BACKGROUND

[0002] With the rapid development of computer technology, human-computer interaction is increasingly close, and speech synthesis technology is widely used in many fields, such as mobile phone voice assistants, smart speakers, audio novels, high-dee speech navigation, intelligent dubbing, virtual anchors, etc. In particular, it is worth noting that speech synthesis technology plays a key role in the field of barrier-free services, which can help visually impaired and dyslexic people to obtain e-books, mobile content and other text information through the auditory channel, significantly improving the convenience of information acquisition. In addition, this technology can also support multilingual learning scenarios, providing pronunciation accurate and tone natural voice demonstrations for language learners, effectively promoting the improvement of language ability. With the continuous growth of social demand for intelligent services, the importance of speech synthesis technology is increasingly prominent.

[0003] Currently, speech synthesis technology can be mainly divided into two categories, namely traditional synthesis scheme and synthesis scheme based on neural network model. The traditional synthesis scheme includes speech synthesis based on waveform splicing and statistical parameter speech synthesis. The speech synthesis scheme based on waveform splicing first constructs a speech library, and in the speech synthesis stage, the similar acoustic units to the synthesized speech are selected from the speech library according to the analysis of the text and certain standards, and then the acoustic units are fine-tuned to splice the synthesized speech. Since the speech is spliced by multiple waveform fragments, the quality of the synthesized speech is low, and a very large speech library is needed to support it. Statistical parameter-based speech synthesis uses statistical models to model the parameterized representation of speech, predicts the corresponding acoustic parameters, and finally synthesizes the output audio through a vocoder. However, the statistical parameter-based speech synthesis process is very tedious, and the entire model complexity is very high. Although the statistical parameter scheme synthesizes smooth speech, compared with human speech, the speech generated by these schemes usually sounds unnatural. In recent years, with the rapid development of deep learning technology, speech synthesis methods based on deep neural networks have attracted widespread attention. Among them, the generation adversarial network, variational autoencoder (VAE) and Transformer based on deep learning are applied to the speech synthesis task. These methods model directly from text to speech waveform in an end-to-end manner, which can generate more natural and smooth speech, and at the same time avoid a large amount of manual annotation work and reduce the demand for tedious feature engineering. These advantages mean that end-to-end speech synthesis systems can be trained on a large amount of rich and expressive but noisy data.

[0004] However, although existing models have made significant progress in the core task of speech synthesis, there are still the following key defects: existing models such as VITS usually rely on complex network structures (such as transposed convolution) in the inference stage, which has large computational overhead, resulting in slow speech synthesis inference speed, which is difficult to meet the real-time requirements, especially on resource-constrained devices; secondly, existing models perform poorly in multi-language and cross-language speech synthesis tasks, and the prosody of speech synthesis is single, especially in the consistency of timbre, emotion and language style. SUMMARY

[0005] In view of the above situation, the main purpose of the present application is to propose a speech synthesis method and system based on VITS improvement to solve the above technical problems.

[0006] The present application proposes a speech synthesis method based on VITS improvement, which comprises the following steps:

[0007] In the training stage:

[0008] Taking the VITS model as the basic model, and giving the input text and the original speech signal, input the input text and the original speech signal into the basic model;

[0009] Use the semantic encoder based on the large language model to perform phoneme coding to capture the emotion, intention and speaking style of the input text, and generate an enhanced text embedding containing deep semantic information;

[0010] The original speech signal is transformed by short-time Fourier transform to generate linear spectrum, and is input into the posterior encoder to obtain the speech posterior distribution and sample the latent variable;

[0011] The latent variable is input into the decoder to generate the frequency domain acoustic feature, and the frequency domain acoustic feature is inverse short-time Fourier transformed to reconstruct the time domain speech waveform to obtain the reconstructed speech signal;

[0012] The enhanced text embedding and the latent variable are input into the MAS module for monotonic alignment search, and in the monotonic alignment search process, a random disturbance with a scale gradually decaying with training is introduced to obtain an alignment path;

[0013] The enhanced text embedding is input into the random duration predictor to obtain the predicted duration of each phoneme;

[0014] The enhanced text embedding is aligned according to the alignment path and input into the prior encoder to generate the text prior distribution;

[0015] A total loss function is constructed based on the reconstructed speech signal, the original speech signal, the predicted duration, the text prior distribution and the speech posterior distribution, and the basic model is trained in an adversarial learning manner according to the total loss function, and after the training is completed, a speech synthesis model is obtained;

[0016] In the inference stage:

[0017] The verification text is input into the speech synthesis model, the predicted duration of each phoneme is aligned as an alignment benchmark of the enhanced text embedding and is input into the prior encoder to generate the text prior distribution;

[0018] The text prior distribution is sampled and input into the decoder to obtain a speech synthesis result.

[0019] The application also provides a speech synthesis system improved based on VITS, wherein the system applies the speech synthesis method improved based on VITS, and the system comprises:

[0020] The training module is configured to:

[0021] The VITS model is used as the basic model, and the input text and the original speech signal are input into the basic model;

[0022] The phonetic coding is performed by using the semantic encoder based on the large language model to capture the emotion, intention and speaking style of the input text, and the enhanced text embedding containing deep semantic information is generated;

[0023] The original speech signal is subjected to short-time Fourier transform to generate linear spectrum, and the linear spectrum is input into the posterior encoder to obtain the speech posterior distribution and sample the latent variable;

[0024] The latent variable is input into the decoder to generate the frequency domain acoustic feature, and the frequency domain acoustic feature is subjected to inverse short-time Fourier transform to reconstruct the time domain speech waveform to obtain the reconstructed speech signal;

[0025] The enhanced text embedding and the latent variable are input into the MAS module to perform monotonic alignment search, and in the monotonic alignment search process, the random disturbance gradually decaying with the training progress scale is introduced to obtain an alignment path;

[0026] The enhanced text embedding is input into the random duration predictor to obtain the predicted duration of each phoneme;

[0027] The enhanced text embedding is aligned according to the alignment path and is input into the prior encoder to generate the text prior distribution;

[0028] A total loss function is constructed based on the reconstructed speech signal, the original speech signal, a prediction duration, a text prior distribution and a speech posterior distribution, and the basic model is trained in an adversarial learning manner according to the total loss function, and after the training is completed, a speech synthesis model is obtained;

[0029] The inference module is configured to:

[0030] The verification text is input into the speech synthesis model to obtain the text prior distribution, the text prior distribution is sampled and input into the decoder to obtain a speech synthesis result.

[0031] Compared with the prior art, the application has the following advantages:

[0032] 1. Optimization of the decoder: the ConvNeXt module is used as the basic backbone network to generate Fourier time-frequency coefficients, and the GELU (Gaussian Error Linear Unit) activation function is introduced to enhance the non-linear expression ability of the model. At the same time, a global response normalization layer (GRN) is integrated in the ConvNeXt module, which realizes global feature aggregation by calculating the channel-level L2 norm, and dynamically calibrates the channel weight by means of the normalization competition mechanism. The improvement of this structure can significantly improve the acoustic quality and naturalness of the synthesized speech;

[0033] 2. Alignment search mechanism enhancement: based on the traditional monotonic alignment search (MAS) algorithm, Gaussian noise is injected into the alignment probability calculation process in the early stage of training. Specifically, a random disturbance term is introduced when solving the Q value (the maximum log-likelihood cumulative value in forward propagation), which effectively expands the alignment search space and improves the alignment flexibility in the early stage of training, while strictly maintaining the monotonicity constraint to avoid premature convergence to a suboptimal solution;

[0034] 3. Semantic encoder replacement optimization: a semantic encoder based on Llama2 is designed to replace the original VITS model structure, which obtains deep semantic representation through a large language model. This design makes the generated speech not only retain complete acoustic features, but also accurately understand and convey the semantic information and emotional color in the text, thereby realizing more expressive acoustic output.

[0035] 4. The decoder based on ConvNeXt and the Llama2 enhanced semantic understanding mechanism proposed in the application can effectively improve the inference speed, emotional expression ability and flexibility of style control of speech synthesis, providing a new solution for cross-language diversified speech synthesis and providing a reference for the development of speech synthesis technology in a more efficient and intelligent direction.

[0036] Additional aspects and advantages of the application will be in part apparent and in part pointed out hereinafter. BRIEF DESCRIPTION OF DRAWINGS

[0037] Figure 1 This is a network structure diagram of the semantic encoder based on a large language model in this invention;

[0038] Figure 2 This is a flowchart illustrating the generation of descriptive sentences using a large language model, as described in this invention.

[0039] Figure 3 This is a structural diagram of the decoder of the present invention;

[0040] Figure 4 This is an architecture diagram of the speech synthesis system based on VITS improvement of the present invention. Detailed Implementation

[0041] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0042] These and other aspects of the embodiments of the present invention will become clear from the following description and accompanying drawings. In these descriptions and drawings, some specific embodiments of the present invention are specifically disclosed to illustrate some ways of implementing the principles of the embodiments of the present invention; however, it should be understood that the scope of the embodiments of the present invention is not limited thereto.

[0043] This embodiment provides a speech synthesis method based on VITS improvement, the method including the following steps:

[0044] During the training phase:

[0045] Step 1: Using the VITS model as the base model, and given the input text and the original speech signal, input the input text and the original speech signal into the base model;

[0046] Step 2: Use a semantic encoder based on a large language model to perform phonemic encoding to capture the sentiment, intent and speaking style of the input text and generate an enhanced text embedding containing deep semantic information.

[0047] Please see Figure 1 As a preferred embodiment of the present invention, phonemic encoding is performed using a semantic encoder based on a large language model to capture the emotion, intent, and speaking style of the input text, generating an enhanced text embedding containing deep semantic information. Specifically, this includes the following steps:

[0048] The input text is fed into a large language model for semantic extraction, resulting in a global semantic embedding, such as... Figure 2 As shown;

[0049] Input the input text into the large language model for semantic extraction, and obtain the global semantic embedding, which includes the following steps:

[0050] Input the input text into the large language model, and guide it through the first task instruction to generate a descriptive sentence. The corresponding process has the following relationship:

[0051] ;

[0052] Wherein, represents the descriptive sentence, represents the Llama2 model, represents the first task prompt instruction, represents the input text;

[0053] Input the descriptive sentence into the large language model again, guide it through the second task instruction to extract all token representations of the final hidden layer of the large language model, and take the average of all token hidden vectors to obtain the global semantic embedding. The corresponding process has the following relationship:

[0054] ;

[0055] Wherein, represents the global semantic embedding, represents the total number of tokens, represents the i-th token in the descriptive sentence, represents the second task prompt instruction.

[0056] Encode the input text into acoustic embedding to obtain acoustic text embedding;

[0057] Align the dimensions of the global semantic embedding and the acoustic text embedding, and then fuse them to obtain an enhanced text embedding containing deep semantic information. The corresponding process has the following relationship:

[0058] ;

[0059] Wherein, represents the enhanced text embedding for speech synthesis, represents the original acoustic text embedding of VITS, represents the linear transformation matrix.

[0060] In the above scheme, the present embodiment selects Llama2 as the basic large language model (LLM), which is one of the largest publicly accessible LLMs and has a comprehensive performance comparable to large models such as GPT3.5 and PaLM (540B). Llama2 uses an improved autoregressive Transformer architecture, integrates advanced components such as pre-normalization (Pre-LN), SwiGLU activation function, and rotated position encoding (RoPE), and adds a grouped query attention mechanism (Grouped-Query Attention, GQA) mechanism to reduce memory usage by sharing key / value projection matrices while maintaining comparable performance to multi-head attention. In addition, Llama2 has strong multilingual processing capabilities, supporting the joint pre-training of more than 20 languages including Chinese and English, and its excellent cross-language alignment characteristics lay a solid foundation for building a unified multilingual speech synthesis system.

[0061] With the technical advantages and application potential of Llama2, the present application proposes to combine its semantic representation capability with the VITS speech synthesis model. By extracting deep semantic information from Llama2, the generated speech can accurately understand and express the semantic connotation and emotional color in the text while maintaining high-quality acoustic features. Specifically, we design a semantic encoder based on Llama2, which utilizes its excellent semantic understanding and expression capabilities to further improve the expressiveness and naturalness of speech synthesis while maintaining the acoustic performance of the baseline model.

[0062] Step 3, the original speech signal is subjected to short-time Fourier transform to generate linear spectrum, and is input into the posterior encoder to obtain the speech posterior distribution and sample the latent variable;

[0063] Step 4, input the latent variable into the decoder to generate frequency domain acoustic features, and then perform inverse short-time Fourier transform on the frequency domain acoustic features to reconstruct time domain speech waveform to obtain the reconstructed speech signal;

[0064] Please refer to Figure 3 As a preferred embodiment of the present application, the decoder includes a plurality of stacked ConvNeXt blocks as a basic backbone network for generating Fourier time-frequency coefficients, and an embedding layer and a normalization layer are connected in series at the input end of the basic backbone network, and an inverse short-time Fourier transform module is connected in series at the output end of the basic backbone network.

[0065] The ConvNeXt block comprises a 7x7 depth convolution, a 1x1 point convolution and a 1x1 depth convolution, the 7x7 depth convolution, the 1x1 point convolution and the 1x1 depth convolution are connected in series in an inverted bottleneck structure, and a Gaussian error linear unit and a global response normalization layer are sequentially arranged between the 1x1 point convolution and the 1x1 depth convolution in the data transmission direction, and a residual link is introduced at the output end of the 1x1 depth convolution.

[0066] The latent variable is input into the decoder to generate a frequency domain acoustic feature, and the frequency domain acoustic feature is inverse short-time Fourier transformed to reconstruct a time domain speech waveform to obtain a reconstructed speech signal, and the specific steps include the following steps:

[0067] The latent variable is adjusted in dimension by an embedding layer to match the FFN points of the ISTFT to obtain a feature sequence adjusted in dimension;

[0068] The feature sequence adjusted in dimension is normalized and input into a basic backbone network to obtain Fourier time-frequency coefficients, and the corresponding process has the following relationship:

[0069] ;

[0070] wherein, the feature sequence adjusted in dimension, the embedding layer, the integrated global response normalization layer, the stacked basic backbone network ConvNeXts block, respectively represent Fourier time-frequency coefficients and phases.

[0071] The Fourier time-frequency coefficients are used to construct complex Fourier coefficients, and the corresponding process has the following relationship:

[0072] ;

[0073] wherein, the complex Fourier coefficients, the imaginary unit,

[0074] The complex Fourier coefficients are input into a Hann window-based inverse short-time Fourier transform module to reconstruct a time domain speech waveform to obtain a reconstructed speech signal, and the corresponding process has the following relationship:

[0075] ;

[0076] wherein, the reconstructed speech signal, the Hann window-based inverse short-time Fourier transform module, the window length, Indicates frame shift;

[0077] In the above scheme, since the present invention is based on VITS, and the decoder in VITS is based on a HiFi-GAN vocoder, which relies on transposed convolution to synthesize and reconstruct the speech signal from latent variables. Although such decoders based on upsampling layers generally have powerful modeling capabilities, the time-consuming nature of the upsampling process makes inference speed its main bottleneck.

[0078] To this end, this invention employs a ConvNeXt block as the basic backbone network to generate Fourier time-frequency coefficients. The ConvNeXt module consists of a 7x7 depthwise convolution and an inverted bottleneck structure, where the inverted bottleneck structure expands the feature dimension through a 1x1 pointwise convolution. Within the bottleneck, a GELU (Gaussian Error Linear Unit) activation function is used to enhance nonlinear expressive power. Furthermore, to further optimize feature representation, this invention incorporates a Global Response Normalization (GRN) layer into the ConvNeXt block. The GRN achieves global feature aggregation by calculating the channel-level L2 norm and dynamically calibrates channel weights using a normalization competition mechanism, thereby suppressing redundant features and enhancing key information, improving the architecture of the ConvNeXt model, and enhancing the quality of synthesized speech.

[0079] Then, the speech signal is synthesized and reconstructed using the Inverse Short-Time Fourier Transform (ISTFT). Unlike traditional neural vocoder architectures that rely on transposed convolutions for upsampling, traditional time-domain vocoders (such as HiFi-GAN) upsample stepwise through multiple layers of transposed convolutions, expanding low-resolution features to the target waveform resolution (typically requiring a magnification of hundreds of times). In contrast, this invention maintains the same feature temporal resolution across all layers, ultimately converting the Fourier coefficients (amplitudes) into the target waveform resolution. M With phase P Input the ISTFT module and use ISTFT to efficiently reconstruct the time-domain signal. ISTFT achieves waveform upsampling, avoiding redundant calculations in traditional transpose convolution and accelerating inference.

[0080] Step 5: Input the enhanced text embedding and latent variables into the MAS module for monotonic alignment search. During the monotonic alignment search process, introduce random perturbations that gradually decay with the scale of training to obtain the alignment path.

[0081] In a preferred embodiment of the present invention, the enhanced text embedding and latent variables are input into the MAS module for monotonic alignment search. During the monotonic alignment search process, a random perturbation that gradually decays with the training progress is introduced to obtain the alignment path, specifically including the following steps:

[0082] The enhanced text embedding is aligned with the dimension of the latent variable space by a projection operation to obtain an aligned enhanced text embedding and the latent variable;

[0083] A log-likelihood score of the aligned enhanced text embedding and the latent variable is calculated to obtain an alignment score matrix;

[0084] A random disturbance that gradually decays with the scale of training progress is added to each element of the alignment score matrix, and dynamic programming is used to accumulate the maximum log-likelihood along a monotonic path to obtain a maximum log-likelihood matrix, and the corresponding process has the following relationship:

[0085] ;

[0086] wherein, represents the value of the maximum log-likelihood accumulation in the forward operation when the index positions of the input text sequence and the posterior speech feature are a and b , respectively, Q represents the value of the maximum log-likelihood accumulation in the forward operation when the index positions of the input text sequence and the posterior speech feature are and a-1 , respectively, b-1 represents the value of the maximum log-likelihood accumulation in the forward operation when the index positions of the input text sequence and the posterior speech feature are Q and , respectively, a represents the value of the maximum log-likelihood accumulation in the forward operation when the index positions of the input text sequence and the posterior speech feature are b-1 and Q , respectively, respectively represent the index positions of the input text sequence and the posterior speech feature, represents the log probability density of the speech latent feature under the distribution of the given text position a , respectively represent the index positions of the input text sequence and the posterior speech feature, represents the mean, represents the standard deviation, represents a random disturbance that gradually decays with the scale of training progress;

[0087] According to the maximum log-likelihood matrix, path backtracking is performed to obtain an alignment path.

[0088] wherein, the calculation process of the random disturbance that gradually decays with the scale of training progress has the following relationship:

[0089] ;

[0090] wherein, represents a noise scale factor, represents the standard deviation of the current alignment path probability distribution, represents a standard normal distribution sampling noise.

[0091] To ensure better results, the introduction of random perturbations that gradually decay with the scale as training progresses can be done in the early stages of model training.

[0092] In the above scheme, the alignment search in VITS uses the MAS algorithm, which estimates the alignment A between the input text and the target speech. This algorithm uses dynamic programming to select the alignment with the highest joint probability between the text and audio from all possible monotonic alignment paths, and trains the model based on variational inference to maximize this probability. The efficiency of the MAS algorithm stems from its dynamic programming strategy, but traditional implementations have limitations: once the model converges to a specific alignment path in the early stages of training, its ability to explore other potentially better paths is limited, potentially leading to suboptimal solutions.

[0093] To address the aforementioned issues, this invention proposes introducing Gaussian noise into the alignment probability calculation process during the initial training phase. Specifically, a random perturbation term is added when calculating the Q-value (i.e., the maximum log-likelihood accumulation value in the forward operation) during dynamic programming. Compared to the traditional MAS algorithm, this invention expands the alignment search space through controllable noise perturbation. The noise is introduced only in the early training phase because the MAS algorithm itself has the ability to quickly learn alignment, and premature noise interference may hinder convergence. As the number of training steps increases, the noise scale... The alignment path is gradually decayed to zero, ensuring stable optimization of the alignment path in the later stages of model training. This method significantly improves the flexibility of alignment in the early stages of training, avoids premature convergence to a suboptimal solution, and maintains the monotonicity constraint of MAS.

[0094] Step 6: Embed the enhanced text into the random duration predictor to obtain the predicted duration of each phoneme;

[0095] Step 7: Align the enhanced text embedding according to the alignment path and input it into the prior encoder to generate the text prior distribution;

[0096] Step 9: Construct a total loss function based on the reconstructed speech signal, the original speech signal, the predicted duration, the prior distribution of the text, and the posterior distribution of the speech. Based on the total loss function, train the basic model using adversarial learning. After training, the speech synthesis model is obtained.

[0097] In a preferred embodiment of the present invention, the total loss function has the following relationship:

[0098] ;

[0099] in, These represent reconstruction loss, KL loss, duration loss, adversarial training loss, feature matching loss, and multi-subband scale loss, respectively. Represents the total loss function;

[0100] The calculation process for the multi-subband scale loss is based on the following relationship:

[0101] ;

[0102] in, Indicates the total number of scales. This represents the loss weight at the i-th scale. Let represent the loss at the i-th scale. The calculation process for the loss at the i-th scale follows the following relationship:

[0103] ;

[0104] in, Let L1 represent the spectral convergence loss, logarithmic amplitude spectrum L1 loss, linear amplitude spectrum L1 loss, phase consistency loss, and complex spectrum L2 loss at the i-th scale, respectively. express The corresponding weights;

[0105] The calculation process for the spectral convergence loss follows the following formula:

[0106] ;

[0107] in, Represents the original speech signal. Let represent the complex spectrum at the i-th scale of the reconstructed speech signal. The first part representing the original speech signal i Complex spectrum at various scales Describing the Frobenius norm, Represents the numerical stability constant;

[0108] The calculation process for the logarithmic amplitude spectrum L1 loss follows the following formula:

[0109] ;

[0110] in, Indicates the first i Number of frequency bins at each scale Indicates the first i Number of time frames per scale The first character representing the reconstructed speech signal i Frequency index f and time frame at each scale t The amplitude value at that point, The first part representing the original speech signal i Frequency index at each scale f and time frame t The amplitude value at that point;

[0111] The calculation process of the linear amplitude spectrum L1 loss exists the following relationship:

[0112] ;

[0113] The calculation process of the phase consistency loss exists the following relationship:

[0114] ;

[0115] wherein, denotes the phase spectrum of the i-th dimension of the reconstructed speech signal, denotes the phase spectrum of the i-th dimension of the original speech signal;

[0116] The calculation process of the complex spectrum L2 loss exists the following relationship:

[0117] ;

[0118] wherein, denotes the complex spectrum of the i-th dimension of the reconstructed speech signal, i denotes the complex spectrum of the i-th dimension of the original speech signal. i In the inference stage:

[0119] Step 9, input the verification text into the speech synthesis model, align each phoneme's predicted duration as the alignment benchmark of the enhanced text embedding, and input it into the prior encoder to generate the text prior distribution;

[0120] Step 10, sample the text prior distribution and input it into the decoder to obtain the speech synthesis result.

[0121] Step 10, sample the text prior distribution and input it into the decoder to obtain the speech synthesis result.

[0122] Please refer to Figure 4 The embodiment also provides a speech synthesis system improved based on VITS, wherein the system applies the speech synthesis method improved based on VITS as described above, and the system comprises:

[0123] A training module is configured to:

[0124] take the VITS model as a basic model, input an input text and an original speech signal into the basic model, and give the input text and the original speech signal;

[0125] use a semantic encoder based on a large language model to perform phoneme coding to capture the emotion, intention and speaking style of the input text, and generate an enhanced text embedding containing deep semantic information;

[0126] ​The original speech signal is subjected to short-time Fourier transform to generate linear spectrum and is input into a posterior encoder to obtain a speech posterior distribution and sample a latent variable;

[0127] The latent variable is input into a decoder to generate a frequency domain acoustic feature, and the frequency domain acoustic feature is subjected to inverse short-time Fourier transform to reconstruct a time domain speech waveform to obtain a reconstructed speech signal;

[0128] The enhanced text embedding and the latent variable are input into a monotonic alignment search (MAS) module to perform monotonic alignment search, and a random disturbance that gradually decays with the training scale is introduced in the monotonic alignment search process to obtain an alignment path;

[0129] The enhanced text embedding is input into a random duration predictor to obtain a predicted duration of each phoneme;

[0130] The enhanced text embedding is aligned according to the alignment path and is input into a prior encoder to generate a text prior distribution;

[0131] A total loss function is constructed based on the reconstructed speech signal, the original speech signal, the predicted duration, the text prior distribution and the speech posterior distribution, and the basic model is trained in an adversarial learning manner according to the total loss function, and after the training is completed, a speech synthesis model is obtained;

[0132] The inference module is configured to:

[0133] The verification text is input into the speech synthesis model to obtain a text prior distribution, the text prior distribution is sampled and sent into the decoder to obtain a speech synthesis result.

[0134] In order to evaluate the effectiveness and performance of the present application, the synthesized speech effect is evaluated as follows:

[0135] The LJSpeech dataset is used in this embodiment. The LJSpeech dataset consists of 13100 short audio segments of a single speaker, with a total length of about 24 hours, and the audio format is 16-bit PCM with a sampling rate of 22kHz. The dataset is randomly divided into a training set (12,500 samples), a validation set (100 samples) and a test set (500 samples).

[0136] Before training, the dataset is first subjected to endpoint detection processing to remove the silent segments at the beginning and end of the audio file, and is subjected to preprocessing, framing and windowing and other preprocessing operations. The linear spectrum obtained from the reconstructed speech signal through short-time Fourier transform (STFT) is used as the input of the post-encoder. The specific parameters for extracting the features are shown in Table 1.

[0137] Table 1: Extraction parameters of features

[0138]

[0139] This embodiment is implemented based on PyTorch in Python language on Windows 10 64-bit operating system, the processor is Inter Core i9 10900K @ 3.70GHz, the graphics card is NVIDIA GeForce RTX 2080Ti, the graphics card memory is 12G, and the Pytorch version is 11.3. The hyperparameters in the experiment are shown in Table 2.

[0140] Table 2 Hyperparameters in the experiment

[0141]

[0142] To comprehensively evaluate the effect of synthesized speech, this embodiment analyzes the system from two dimensions of speech quality and synthesis speed.

[0143] (1) Speech quality evaluation

[0144] In terms of speech synthesis quality, the mean opinion score (MOS) is used as the subjective evaluation index, and the mel-cepstral distortion (MCD) is introduced as the objective evaluation standard.

[0145] In the subjective evaluation process, 10 evaluators with good English language sense were invited to participate in the MOS test. Different systems were used to synthesize speech for the test set text, each evaluator listened to randomly assigned audio samples, and evaluated the speech naturalness based on a 5-point rating scale. To ensure fairness, all audio samples were amplitude normalized, and each evaluator evaluated each sample only once. Finally, after removing the highest and lowest scores, the 20 speech samples generated by each system were averaged to obtain the overall score of the system.

[0146] In terms of objective evaluation, the mel-cepstral distortion value (MCD) is used to measure the spectral difference between synthesized speech and target speech. The lower the MCD value, the smaller the spectral distortion of the synthesized speech, and the higher the speech quality.

[0147] (2) Synthesis speed

[0148] To verify the improvement of the LC-VITS model in inference speed, the real-time factor RTF (Real-Time Factor) is used as the evaluation index, which is defined as the ratio of the processing time required to generate speech to the actual length of the speech, and the calculation formula is as follows:

[0149] ;

[0150] wherein, RTF represents the real-time factor, t represents the time spent generating speech, The RTF represents the duration of the generated speech. The lower the RTF, the faster the system generates speech. RTF < 1 means real-time or faster generation. The time required to generate the speech is less than the duration of the speech. For example, RTF = 0.5 means that it takes only 30 seconds to generate a minute of speech. RTF > 1 means non-real-time, and the time required to generate the speech is longer than the duration of the speech. For example, RTF = 2 means that it takes 2 minutes to generate a minute of speech. The evaluation is carried out in an Intel i7@2.3GHz single-thread environment, and 100 sentences in the test set are used to compare the original VITS model with the improved LC-VITS model, so as to evaluate the optimization effect of the inference speed.

[0151] The LC-VITS model improved based on VITS proposed in the embodiment can effectively improve the inference speed, emotional expression ability and flexibility of style control of speech synthesis, and provides a new solution for cross-language diversified speech synthesis and a reference for the development of speech synthesis technology in a more efficient and intelligent direction.

[0152] It should be understood that parts of the present application can be realized by hardware, software, firmware or a combination thereof. In the above-described embodiments, a plurality of steps or methods can be realized by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if realized by hardware, and as in another embodiment, it can be realized by any one or a combination of the following technologies known in the art: discrete logic circuit with logic gate circuit for implementing logic function on data signal, application specific integrated circuit with suitable combination logic gate circuit, programmable gate array (PGA), field programmable gate array (FPGA) and the like.

[0153] In the description of the present specification, the description of the terms "one embodiment", "some embodiments", "example", "specific example" or "some examples" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the present specification, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0154] The above-described embodiments only express several embodiments of the present application, and the description is more specific and detailed, but it cannot be understood as limiting the scope of the present patent. It should be noted that for ordinary skilled persons in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are within the scope of the present application. Therefore, the protection scope of the present patent should be subject to the appended claims.

Claims

1. A speech synthesis method based on VITS improvement, characterized in that, The method includes the following steps: During the training phase: Using the VITS model as the base model, and given the input text and the original speech signal, input the input text and the original speech signal into the base model; Phonetic encoding is performed using a semantic encoder based on a large language model to capture the sentiment, intent, and speaking style of the input text, generating an enhanced text embedding containing deep semantic information. The original speech signal is subjected to short-time Fourier transform to generate a linear spectrum, which is then input into the posterior encoder to obtain the posterior distribution of the speech, and latent variables are sampled. The latent variables are input into the decoder to generate frequency domain acoustic features. The frequency domain acoustic features are then subjected to inverse short-time Fourier transform to reconstruct the time domain speech waveform, thus obtaining the reconstructed speech signal. The enhanced text embedding and latent variables are input into the MAS module for monotonic alignment search. During the monotonic alignment search process, a random perturbation that gradually decays with the scale of training is introduced to obtain the alignment path. The enhanced text is embedded into a random duration predictor to obtain the predicted duration for each phoneme; The enhanced text embedding is aligned according to the alignment path and fed into the prior encoder to generate the text prior distribution; A total loss function is constructed based on the reconstructed speech signal, the original speech signal, the predicted duration, the prior distribution of the text, and the posterior distribution of the speech. Based on the total loss function, the basic model is trained using an adversarial learning approach. After training, a speech synthesis model is obtained. During the reasoning stage: The validation text is input into the speech synthesis model, and the predicted duration of each phoneme is used as the alignment benchmark for the enhanced text embedding. This is then input into the prior encoder to generate the text prior distribution. The prior distribution of the text is sampled and input into the decoder to obtain the speech synthesis result; The total loss function has the following relationship: ； in, These represent reconstruction loss, KL loss, duration loss, adversarial training loss, feature matching loss, and multi-subband scale loss, respectively. Represents the total loss function; The calculation process for the multi-subband scale loss is based on the following relationship: ； in, Indicates the total number of scales. This represents the loss weight at the i-th scale. Let represent the loss at the i-th scale. The calculation process for the loss at the i-th scale follows the following relationship: ； in, They represent the first i The spectral convergence loss, logarithmic amplitude spectrum L1 loss, linear amplitude spectrum L1 loss, phase consistency loss, and complex spectrum L2 loss are calculated at each scale. express The corresponding weights; The calculation process for the spectral convergence loss follows the following formula: ； in, Represents the original speech signal. Let represent the complex spectrum at the i-th scale of the reconstructed speech signal. Let represent the complex spectrum at the i-th scale of the original speech signal. Denotes the Frobenius norm. Represents the numerical stability constant; The calculation process for the logarithmic amplitude spectrum L1 loss follows the following formula: ； in, This represents the number of frequency bins at the i-th scale. This represents the number of time frames at the i-th scale. This represents the frequency index f and the amplitude value at time frame t at the i-th scale of the reconstructed speech signal. This represents the frequency index f and the amplitude value at time frame t at the i-th scale of the original speech signal; The calculation process for the linear amplitude spectrum L1 loss follows the following relationship: ； The calculation process for phase consistency loss is based on the following relationship: ； in, This represents the phase spectrum at the i-th scale of the reconstructed speech signal. This represents the phase spectrum at the i-th scale of the original speech signal; The calculation process for the complex spectrum L2 loss follows the following relationship: ； in, The first character representing the reconstructed speech signal i Complex spectrum at various scales Let represent the complex spectrum of the i-th scale of the original speech signal.

2. The speech synthesis method based on VITS improvement according to claim 1, characterized in that, Using a semantic encoder based on a large language model for phonemic encoding to capture the sentiment, intent, and speaking style of the input text, and generating enhanced text embeddings containing deep semantic information, specifically includes the following steps: The input text is fed into a large language model for semantic extraction, resulting in a global semantic embedding. The input text encoder is acoustically embedded to obtain the acoustic text embedding; The global semantic embedding and the acoustic text embedding are dimensionally aligned and then fused to obtain an enhanced text embedding containing deep semantic information.

3. The speech synthesis method based on VITS improvement according to claim 2, characterized in that, The input text is fed into a large language model for semantic extraction to obtain the global semantic embedding. This process includes the following steps: The input text is fed into the large language model, guided by the first task instruction to generate descriptive sentences. The corresponding process has the following relationship: ； in, This is a descriptive sentence. Represents the Llama2 model. This indicates the first task prompt instruction. Indicates the input text; The descriptive sentence is input again into the large language model. Guided by the second task instruction, all token representations of the final hidden layer of the large language model are extracted. The average of all token hidden vectors is then taken to obtain the global semantic embedding. The corresponding process has the following relationship: ； in, Indicates global semantic embedding. Indicates the total number of tokens. In a descriptive sentence, the first... i One token, This indicates the second task prompt command.

4. The speech synthesis method based on VITS improvement according to claim 3, characterized in that, The global semantic embedding and the acoustic text embedding are dimensionally aligned and then fused to obtain an enhanced text embedding containing deep semantic information. The corresponding process has the following relationship: ； in, This represents enhanced text embeddings used for speech synthesis. This represents the original acoustic text embedding of VITS. This represents a linear transformation matrix.

5. The speech synthesis method based on VITS improvement according to claim 4, characterized in that, The decoder includes several stacked ConvNeXt blocks as a basic backbone network for generating Fourier time-frequency coefficients. An embedding layer and a normalization layer are connected in series at the input of the basic backbone network, and an inverse short-time Fourier transform module is connected in series at the output of the basic backbone network.

6. The speech synthesis method based on VITS improvement according to claim 5, characterized in that, The ConvNeXt block includes a 7x7 depthwise convolution, a 1x1 dot convolution, and a 1x1 depthwise convolution. The 7x7 depthwise convolution, the 1x1 dot convolution, and the 1x1 depthwise convolution are connected in series using an inverted bottleneck structure. Gaussian error linear units and global response normalization layers are sequentially set between the 1x1 dot convolution and the 1x1 depthwise convolution in the direction of data transfer. A residual link is introduced at the output of the 1x1 depthwise convolution.

7. The speech synthesis method based on VITS improvement according to claim 6, characterized in that, The latent variables are input into the decoder to generate frequency domain acoustic features. Then, the frequency domain acoustic features are subjected to inverse short-time Fourier transform to reconstruct the time domain speech waveform, resulting in the reconstructed speech signal. The specific steps include the following: The latent variables are adjusted through an embedding layer to match the number of FFN points of ISTFT, resulting in a dimension-adjusted feature sequence. The dimension-adjusted feature sequence is normalized and input into the basic backbone network to obtain the Fourier time-frequency coefficients. The corresponding process has the following relationship: ； in, This represents the feature sequence after dimension adjustment. Indicates the embedding layer. This indicates the integration of a global response normalization layer. This indicates the use of stacked basic backbone network ConvNeXts blocks. These represent the Fourier time-frequency coefficients and phase, respectively; The process of constructing complex Fourier coefficients from Fourier time-frequency coefficients follows the following relationship: ； in, Represents the complex Fourier coefficients. Represents the imaginary unit; The complex Fourier coefficients are input into the inverse short-time Fourier transform module based on the Hanning window to reconstruct the time-domain speech waveform, thus obtaining the reconstructed speech signal. The corresponding process has the following relationship: ； in, This indicates the reconstruction of the speech signal. This represents the inverse short-time Fourier transform module based on the Hanning window. Indicates the length of the window. Indicates frame shift.

8. The speech synthesis method based on VITS improvement according to claim 7, characterized in that, The enhanced text embedding and latent variables are input into the MAS module for monotonic alignment search. During the monotonic alignment search process, a random perturbation that gradually decays with the scale of training is introduced to obtain the alignment path. The specific steps include the following: The enhanced text embedding is aligned with the latent variable space dimension using a projection operation to obtain the aligned enhanced text embedding and latent variables; The alignment score matrix is ​​obtained by calculating the log-likelihood scores of the aligned enhanced text embeddings and latent variables. Add a random perturbation that gradually decays with the training scale to each element of the alignment score matrix, and use dynamic programming to accumulate the maximum log-likelihood along the monotonic path to obtain the maximum log-likelihood matrix. The corresponding process has the following relationship: ； in, This indicates the index position of the input text sequence. a The index positions of the posterior speech features are b time Q The value, i.e., the maximum log-likelihood accumulation value during the forward operation, Indicates the corresponding position is a-1 and b-1 time Q value, Indicates the corresponding position is a and b-1 time Q value, These represent the index positions of the input text sequence and the posterior speech features, respectively. Indicates the position of the given text. a Under the distribution of , the log probability density of latent speech features, This indicates the index position of the input text sequence and the posterior speech features. This represents the mean. Indicates standard deviation, This represents a random perturbation that gradually decays with the scale of training. Based on the maximum log-likelihood matrix, path backtracking is performed to obtain the aligned path; The calculation process for the random perturbation, which gradually decays with the training scale, follows the following relationship: ； in, Represents the noise scaling factor. This represents the standard deviation of the probability distribution of the current alignment path. This represents the sampling noise of a standard normal distribution.

9. A speech synthesis system based on VITS improvement, characterized in that, The system applies the VITS-based improved speech synthesis method as described in any one of claims 1 to 8, and the system comprises: The training module is used for: Using the VITS model as the base model, and given the input text and the original speech signal, input the input text and the original speech signal into the base model; Phonetic encoding is performed using a semantic encoder based on a large language model to capture the sentiment, intent, and speaking style of the input text, generating an enhanced text embedding containing deep semantic information. The original speech signal is subjected to short-time Fourier transform to generate a linear spectrum, which is then input into the posterior encoder to obtain the posterior distribution of the speech, and latent variables are sampled. The latent variables are input into the decoder to generate frequency domain acoustic features. The frequency domain acoustic features are then subjected to inverse short-time Fourier transform to reconstruct the time domain speech waveform, thus obtaining the reconstructed speech signal. The enhanced text embedding and latent variables are input into the MAS module for monotonic alignment search. During the monotonic alignment search process, a random perturbation that gradually decays with the scale of training is introduced to obtain the alignment path. The enhanced text is embedded into a random duration predictor to obtain the predicted duration for each phoneme; The enhanced text embedding is aligned according to the alignment path and fed into the prior encoder to generate the text prior distribution; A total loss function is constructed based on the reconstructed speech signal, the original speech signal, the predicted duration, the prior distribution of the text, and the posterior distribution of the speech. Based on the total loss function, the basic model is trained using an adversarial learning approach. After training, a speech synthesis model is obtained. The inference module is used for: The validation text is input into the speech synthesis model to obtain the text prior distribution; the text prior distribution is sampled and sent to the decoder to obtain the speech synthesis result.

Images