Voice synthesis method and system based on VITS improvement

Abstract

The invention provides a voice synthesis method and system based on VITS improvement, and the method comprises the steps: optimizing a text encoder of a VITS model, introducing a large language model, and enabling the emotion, intention and speaking style of an input text to be captured when the text is encoded; a random disturbance item is introduced when the Q value is dynamically planned and solved, the alignment flexibility in the initial training stage is improved, meanwhile, monotonicity constraint is strictly kept, and it is avoided that a suboptimal solution is obtained through convergence too early; a ConvNeXt module is used as a basic backbone network of a decoder, and ISTFT is utilized to efficiently reconstruct a time domain signal, so that waveform up-sampling is realized, redundant calculation of traditional transpose convolution is avoided, and reasoning is accelerated. According to the method, the reasoning speed, the emotion expression ability and the style control flexibility of speech synthesis can be effectively improved, a new solution is provided for cross-language diversified speech synthesis, and a reference is provided for the more efficient and more intelligent development of the speech synthesis technology.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of speech synthesis, in particular to a speech synthesis method and system based on VITS improvement. BACKGROUND

[0002] With the rapid development of computer technology, human-computer interaction is increasingly close, and speech synthesis technology is widely used in many fields, such as mobile phone voice assistants, smart speakers, audio novels, high-dee speech navigation, intelligent dubbing, virtual anchors, etc. In particular, it is worth noting that speech synthesis technology plays a key role in the field of barrier-free services, which can help visually impaired and dyslexic people to obtain e-books, mobile content and other text information through the auditory channel, significantly improving the convenience of information acquisition. In addition, this technology can also support multilingual learning scenarios, providing pronunciation accurate and tone natural speech demonstrations for language learners, effectively promoting the improvement of language ability. With the continuous growth of social demand for intelligent services, the importance of speech synthesis technology is increasingly prominent.

[0003] Currently, speech synthesis technology can be mainly divided into two categories, namely traditional synthesis scheme and synthesis scheme based on neural network model. The traditional synthesis scheme includes speech synthesis based on waveform splicing and statistical parameter speech synthesis. The speech synthesis scheme based on waveform splicing first constructs a speech library, and in the speech synthesis stage, the similar acoustic units to the synthesized speech are selected from the speech library according to the analysis of the text and certain standards, and then the acoustic units are fine-tuned to splice the synthesized speech. Since the speech is spliced by multiple waveform fragments, the quality of the synthesized speech is low, and a very large speech library is needed to support it. Statistical parameter-based speech synthesis uses statistical models to model the parameterized representation of speech, predicts the corresponding acoustic parameters, and finally synthesizes the output audio through a vocoder. However, the statistical parameter-based speech synthesis process is very tedious, and the entire model complexity is very high. Although the statistical parameter scheme synthesizes smooth speech, compared with human speech, the speech generated by these schemes usually sounds unnatural. In recent years, with the rapid development of deep learning technology, speech synthesis methods based on deep neural networks have attracted widespread attention. Among them, the generation adversarial network, variational autoencoder (VAE) and Transformer based on deep learning are applied to the speech synthesis task. These methods model directly from text to speech waveform in an end-to-end manner, which can generate more natural and smooth speech, and at the same time avoid a large amount of manual annotation work and reduce the demand for tedious feature engineering. These advantages mean that end-to-end speech synthesis systems can be trained on a large amount of rich and expressive but noisy data.

[0004] However, although existing models have made significant progress in the core task of speech synthesis, there are still the following key defects: existing models such as VITS usually rely on complex network structures (such as transpose convolution) in the inference stage, which has a large computational overhead, resulting in slow speech synthesis inference speed, which is difficult to meet the real-time requirements, especially on resource-constrained devices; secondly, existing models perform poorly in multi-language and cross-language speech synthesis tasks, and the speech synthesis rhythm is single, especially in terms of consistency of timbre, emotion and language style. SUMMARY

[0005] In view of the above situation, the main purpose of the present application is to propose a speech synthesis method and system based on VITS improvement to solve the above technical problems.

[0006] The present application proposes a speech synthesis method based on VITS improvement, which comprises the following steps: In the training stage: Take the VITS model as the base model, and give the input text and the original speech signal, input the input text and the original speech signal into the base model; Use the semantic encoder based on the large language model to perform phoneme coding to capture the emotion, intention and speaking style of the input text, and generate an enhanced text embedding containing deep semantic information; The original speech signal is transformed by short-time Fourier transform to generate linear spectrum, and is input into the posterior encoder to obtain the speech posterior distribution and sample the latent variable; The latent variable is input into the decoder to generate the frequency domain acoustic feature, and the frequency domain acoustic feature is inversely transformed by the short-time Fourier transform to reconstruct the time domain speech waveform to obtain the reconstructed speech signal; The enhanced text embedding and the latent variable are input into the MAS module for monotonic alignment search, and in the monotonic alignment search process, a random disturbance that gradually decays with the training scale is introduced to obtain an alignment path; The enhanced text embedding is input into the random duration predictor to obtain the predicted duration of each phoneme; The enhanced text embedding is aligned according to the alignment path and input into the prior encoder to generate the text prior distribution; Based on the reconstructed speech signal, the original speech signal, the predicted duration, the text prior distribution and the speech posterior distribution, a total loss function is constructed, and the base model is trained in an adversarial learning manner according to the total loss function, and after training, a speech synthesis model is obtained; In the inference stage: The verification text is input into the speech synthesis model, the predicted duration of each phoneme is aligned as the alignment benchmark of the enhanced text embedding, and is input into the prior encoder to generate the text prior distribution; The text prior distribution is sampled and input into the decoder to obtain a speech synthesis result.

[0007] The application further provides a speech synthesis system based on the improved VITS, wherein the system applies the speech synthesis method based on the improved VITS, and the system comprises: The training module is configured to: Take the VITS model as a base model, input the input text and the original speech signal into the base model, and give the input text and the original speech signal; The semantic encoder based on the large language model is used for phoneme coding to capture the emotion, intention and speaking style of the input text, and generate an enhanced text embedding containing deep semantic information; The original speech signal is subjected to short-time Fourier transform to generate linear spectrum, and is input into the posterior encoder to obtain a speech posterior distribution and sample a latent variable; The latent variable is input into the decoder to generate frequency domain acoustic features, and the frequency domain acoustic features are subjected to inverse short-time Fourier transform to reconstruct time domain speech waveform to obtain a reconstructed speech signal; The enhanced text embedding and the latent variable are input into the MAS module for monotonic alignment search, and a random disturbance that gradually decays with the training scale is introduced in the monotonic alignment search process to obtain an alignment path; The enhanced text embedding is input into the random duration predictor to obtain the predicted duration of each phoneme; The enhanced text embedding is aligned according to the alignment path and input into the prior encoder to generate a text prior distribution; The total loss function is constructed based on the reconstructed speech signal, the original speech signal, the predicted duration, the text prior distribution and the speech posterior distribution, the base model is trained in an adversarial learning manner according to the total loss function, and after the training is completed, a speech synthesis model is obtained; The inference module is configured to: The verification text is input into the speech synthesis model to obtain a text prior distribution, the text prior distribution is sampled and input into the decoder to obtain a speech synthesis result.

[0008] Compared with the prior art, the application has the following advantages: 1. Decoder optimization: ConvNeXt modules are used as the basic backbone network to generate Fourier time-frequency coefficients, and GELU (Gaussian Error Linear Unit) activation functions are introduced to enhance the non-linear representation ability of the model. At the same time, a global response normalization layer (GRN) is integrated into the ConvNeXt module, which calculates the channel-level L2 norm to achieve global feature aggregation, and dynamically calibrates the channel weight through the normalization competition mechanism. The improvement of this structure can significantly improve the acoustic quality and naturalness of synthesized speech; 2. Alignment search mechanism enhancement: Based on the traditional monotonic alignment search (MAS) algorithm, Gaussian noise is injected into the alignment probability calculation process at the beginning of training. Specifically, a random disturbance term is introduced when solving the Q value (the maximum log-likelihood cumulative value in forward propagation), which effectively expands the alignment search space and improves the alignment flexibility at the beginning of training, while strictly maintaining the monotonicity constraint to avoid premature convergence to a suboptimal solution; 3. Semantic encoder replacement optimization: Design a semantic encoder based on Llama2 to replace the original VITS model structure, which obtains deep semantic representation through a large language model. This design makes the generated speech not only retain complete acoustic features, but also accurately understand and convey semantic information and emotional color in the text, thus realizing more expressive acoustic output.

[0009] 4. The decoder based on ConvNeXt and the Llama2 enhanced semantic understanding mechanism proposed in the present application can effectively improve the inference speed, emotional expression ability and flexibility of style control of speech synthesis, providing a new solution for cross-language diversified speech synthesis and providing a reference for the development of more efficient and intelligent speech synthesis technology.

[0010] Additional aspects and advantages of the present application will be partially given in the following description, partially apparent from the following description, or understood through embodiments of the present application. BRIEF DESCRIPTION OF DRAWINGS

[0011] Figure 1 The network structure diagram of the semantic encoder based on a large language model of the present application; Figure 2 The flowchart for generating a descriptive sentence through a large language model of the present application; Figure 3 The structure diagram of the decoder of the present application; Figure 4 The architecture diagram of the speech synthesis system improved based on VITS of the present application. DETAILED DESCRIPTION

[0012] Embodiments of the present application are described below in the following detailed description and illustrations. Like reference numerals are used to describe like elements throughout the disclosure. The embodiments described below are merely exemplary for the purposes of explanation and are not intended to limit the application in accordance with the present application.

[0013] These and other aspects of embodiments of the present application will become apparent from the following description, and the accompanying drawings, in which, by way of illustration and example:

[0014] The present embodiment provides a speech synthesis method based on VITS improvement, which comprises the following steps: In the training stage: Step 1, taking the VITS model as the base model, and giving the input text and the original speech signal, inputting the input text and the original speech signal into the base model; Step 2, using a semantic encoder based on a large language model to perform phoneme coding to capture the emotion, intent and speaking style of the input text, and generate an enhanced text embedding containing deep semantic information; Please refer to Figure 1 As a preferred embodiment of the present application, the semantic encoder based on the large language model is used for phoneme coding to capture the emotion, intent and speaking style of the input text, and generate an enhanced text embedding containing deep semantic information, which comprises the following steps: The input text is input into the large language model to extract the global semantic embedding, as shown in Figure 2 ; The input text is input into the large language model to extract the global semantic embedding, which comprises the following steps: The input text is input into the large language model, guided by the first task instruction to generate a descriptive sentence, and the corresponding process has the following relationship: ; Wherein, represents the descriptive sentence, represents the Llama2 model, represents the first task prompt instruction, represents the input text; The descriptive sentence is input into the large language model again, guided by the second task instruction to extract all token representations of the final hidden layer of the large language model, and the average of all token hidden vectors is taken to obtain the global semantic embedding, and the corresponding process has the following relationship: ; wherein, denotes a global semantic embedding, denotes the total number of tokens, denotes the i-th token in the descriptive sentence, denotes the second task prompt instruction.

[0015] The input text encoder is subjected to acoustic embedding to obtain an acoustic text embedding; The global semantic embedding and the acoustic text embedding are subjected to dimensional alignment and then fused to obtain an enhanced text embedding containing deep semantic information, and the corresponding process has the following relationship: ; wherein, denotes the enhanced text embedding for speech synthesis, denotes the original acoustic text embedding of VITS, denotes a linear transformation matrix.

[0016] In the above scheme, the embodiment selects Llama2 as a basic large language model (LLM), and Llama2 is one of the largest publicly accessible LLMs, and its comprehensive performance can be comparable to large models such as GPT3.5 and PaLM (540B). Llama2 uses an improved autoregressive Transformer architecture, integrates advanced components such as pre-normalization (Pre-LN), SwiGLU activation function and rotated position encoding (RoPE), and adds a grouped query attention mechanism (Grouped-Query Attention, GQA) mechanism, which reduces memory occupation by sharing key / value projection matrices while maintaining comparable performance to multi-head attention. In addition, Llama2 also has strong multilingual processing capability, supports joint pre-training of more than 20 languages including Chinese and English, and its excellent cross-language alignment characteristics lay a solid foundation for building a unified multilingual speech synthesis system.

[0017] With the technical advantages and application potential of Llama2, the application combines its semantic representation capability with the VITS speech synthesis model. By extracting deep semantic information from Llama2, the generated speech can accurately understand and express the semantic connotation and emotional color in the text while maintaining high-quality acoustic features. Specifically, we designed a semantic encoder based on Llama2, which utilizes its excellent semantic understanding and expression capabilities to further improve the expressiveness and naturalness of speech synthesis while maintaining the acoustic performance of the baseline model.

[0018] Step 3, the original speech signal is subjected to short-time Fourier transform to generate linear spectrum and is input into the posterior encoder to obtain a speech posterior distribution and sample to obtain a latent variable; Step 4, the latent variable is input into the decoder to generate frequency domain acoustic features, and the frequency domain acoustic features are subjected to inverse short-time Fourier transform to reconstruct time domain speech waveform to obtain a reconstructed speech signal; Please refer to Figure 3 As a preferred embodiment of the present application, the decoder comprises a plurality of stacked ConvNeXt blocks as a basic backbone network for generating Fourier time-frequency coefficients, and an embedding layer and a normalization layer are connected in series at the input end of the basic backbone network, and an inverse short-time Fourier transform module is connected in series at the output end of the basic backbone network.

[0019] Among them, the ConvNeXt block comprises a 7x7 deep convolution, a 1x1 point convolution and a 1x1 deep convolution, the 7x7 deep convolution, the 1x1 point convolution and the 1x1 deep convolution are connected in series in an inverted bottleneck structure, and a Gaussian error linear unit and a global response normalization layer are sequentially arranged between the 1x1 point convolution and the 1x1 deep convolution in the data transmission direction, and a residual link is introduced at the output end of the 1x1 deep convolution.

[0020] Among them, the latent variable is input into the decoder to generate frequency domain acoustic features, and the frequency domain acoustic features are subjected to inverse short-time Fourier transform to reconstruct time domain speech waveform to obtain a reconstructed speech signal, which specifically comprises the following steps: The latent variable is adjusted in dimension through the embedding layer to match the FFN points of the ISTFT to obtain a feature sequence after dimension adjustment; The feature sequence after dimension adjustment is normalized and input into the basic backbone network to obtain Fourier time-frequency coefficients, and the corresponding process exists the following relationship: ; Among them, The feature sequence after dimension adjustment is represented as The embedding layer is represented as The integrated global response normalization layer is represented as The basic backbone network ConvNeXts block is represented as The Fourier time-frequency coefficients and the phase are represented as According to the Fourier time-frequency coefficients, a complex Fourier coefficient is constructed, and the corresponding process exists the following relationship: ; Among them, The complex Fourier coefficient is represented as The imaginary unit is represented as The complex Fourier coefficients are input into a Hann window-based inverse short-time Fourier transform module to be reconstructed into a time-domain speech waveform to obtain a reconstructed speech signal, and the corresponding process has the following relationship: ; wherein, represents the reconstructed speech signal, represents the Hann window-based inverse short-time Fourier transform module, represents the window length, represents the frame shift; In the above scheme, since the present application is based on VITS, and the decoder in VITS is based on a HiFi-GAN vocoder, which relies on transposed convolution to synthesize a reconstructed speech signal from hidden variables. Although such a decoder based on an upsampling layer generally has strong modeling capability, due to the time-consuming nature of the upsampling process, inference speed is its main bottleneck.

[0021] To this end, the present application uses a ConvNeXt block as a basic backbone network to generate Fourier time-frequency coefficients. The ConvNeXt module is composed of a 7x7 depthwise convolution and an inverted bottleneck structure, where the inverted bottleneck structure expands the feature dimension through a 1x1 pointwise convolution. The use of a GELU (Gaussian Error Linear Unit) activation function within the bottleneck enhances the non-linear expression capability. In addition, to further optimize feature representation, the present application incorporates a global response normalization (GRN) layer into the ConvNeXt block. GRN achieves global feature aggregation by calculating the channel-level L2 norm, and dynamically calibrates the channel weight using a normalization competition mechanism, thereby suppressing redundant features and enhancing key information, improving the architecture of the ConvNeXt model, and enhancing the quality of synthesized speech.

[0022] Then, a reconstructed speech signal is synthesized through a fast inverse short-time Fourier transform (ISTFT). Unlike traditional neural vocoder architectures that rely on transposed convolution for upsampling, traditional time-domain vocoders (such as HiFi-GAN) use multiple layers of transposed convolution to progressively upsample low-resolution features to the target waveform resolution (often requiring amplification by hundreds of times). In contrast, the present application maintains the same feature temporal resolution across all layers, and finally inputs the Fourier coefficients (amplitude M and phase P ) into the ISTFT module to efficiently reconstruct the time-domain signal using ISTFT. Waveform upsampling is achieved through ISTFT, avoiding the redundant calculations of traditional transposed convolution and speeding up inference.

[0023] Step 5, input the enhanced text embedding and the latent variable into the MAS module for monotonic alignment search, and introduce a random disturbance which gradually decays with the training scale during the monotonic alignment search process to obtain an alignment path; As a preferred embodiment of the present application, the enhanced text embedding and the latent variable are input into the MAS module for monotonic alignment search, and a random disturbance which gradually decays with the training scale is introduced during the monotonic alignment search process to obtain an alignment path, which specifically comprises the following steps: The enhanced text embedding is aligned with the latent variable space dimension by projection operation to obtain an aligned enhanced text embedding and latent variable; The log-likelihood score of the aligned enhanced text embedding and the latent variable is calculated to obtain an alignment score matrix; A random disturbance which gradually decays with the training scale is added to each element of the alignment score matrix, and dynamic programming is used to accumulate the maximum log-likelihood along the monotonic path to obtain a maximum log-likelihood matrix, and the corresponding process has the following relationship: ; Wherein, represents the value of the maximum log-likelihood accumulation in the forward operation when the index positions of the input text sequence and the posterior speech feature are a and b , respectively, Q represents the value of the maximum log-likelihood accumulation in the forward operation when the index positions of the input text sequence and the posterior speech feature are and a-1 , respectively, b-1 represents the value of the maximum log-likelihood accumulation in the forward operation when the index positions of the input text sequence and the posterior speech feature are Q and , respectively, a represents the index positions of the input text sequence and the posterior speech feature, respectively, b-1 represents the log probability density of the speech latent feature under the distribution of the given text position Q , represents the index positions of the input text sequence and the posterior speech feature, represents the log probability density of the speech latent feature under the distribution of the given text position a , represents the index positions of the input text sequence and the posterior speech feature, represents the mean, represents the standard deviation, represents a random disturbance which gradually decays with the training scale; According to the maximum log-likelihood matrix, path backtracking is performed to obtain an alignment path.

[0024] Wherein, the calculation process of the random disturbance which gradually decays with the training scale has the following relationship: ; Wherein, represents a noise scale factor, a standard deviation representing a current alignment path probability distribution, a standard normal distribution sampling noise is represented.

[0025] In order to ensure better effect, the introduction of the random disturbance which gradually decays with the training progress can be carried out in the initial stage of model training.

[0026] In the above scheme, the alignment search in VITS adopts MAS algorithm, and the MAS algorithm is used to estimate the alignment A between the input text and the target voice. The algorithm selects the alignment with the highest joint probability between the text and the audio from all possible monotonic alignment paths through dynamic programming, and trains the model based on variational inference to maximize the probability. The efficiency of the MAS algorithm is derived from its dynamic programming strategy, but the traditional implementation has limitations: once the model converges to a certain specific alignment path in the initial stage of training, its ability to explore other potential better paths will be limited, which may lead to a suboptimal solution.

[0027] To solve the above problems, the present application introduces Gaussian noise into the calculation process of alignment probability in the initial stage of training. Specifically, when calculating the Q value (i.e. the maximum log likelihood cumulative value in the forward operation) through dynamic programming, a random disturbance term is added. Compared with the traditional MAS algorithm, the present application expands the alignment search space through controllable noise disturbance. The introduction of noise is only carried out in the initial stage of training, because the MAS algorithm itself has the ability to quickly learn alignment, and early noise disturbance may hinder convergence. As the number of training steps increases, the noise scale gradually decays to zero, ensuring stable optimization of the alignment path in the later stage of the model. This method significantly improves the flexibility of alignment in the initial stage of training, avoids premature convergence to a suboptimal solution, and at the same time maintains the monotonicity constraint of MAS.

[0028] Step 6, input the enhanced text embedding into the random duration predictor to obtain the predicted duration of each phoneme; Step 7, align the enhanced text embedding according to the alignment path and input it into the prior encoder to generate the text prior distribution; Step 9, based on the reconstructed speech signal, the original speech signal, the predicted duration, the text prior distribution and the speech posterior distribution, a total loss function is constructed, and the basic model is trained in an adversarial learning manner according to the total loss function, and after the training is completed, a speech synthesis model is obtained; As a preferred embodiment of the present application, the total loss function has the following relationship: ; Among them, respectively represent the reconstruction loss, the KL loss, the duration loss, the adversarial training loss, the feature matching loss and the multi-subband scale loss; represent the total loss function; The calculation process for the multi-subband scale loss is based on the following relationship: ; in, Indicates the total number of scales. This represents the loss weight at the i-th scale. Let represent the loss at the i-th scale. The calculation process for the loss at the i-th scale follows the following relationship: ; in, Let L1 represent the spectral convergence loss, logarithmic amplitude spectrum L1 loss, linear amplitude spectrum L1 loss, phase consistency loss, and complex spectrum L2 loss at the i-th scale, respectively. express The corresponding weights; The calculation process for the spectral convergence loss follows the following formula: ; in, Represents the original speech signal. Let represent the complex spectrum at the i-th scale of the reconstructed speech signal. The first part representing the original speech signal i Complex spectrum at various scales Denotes the Frobenius norm. Represents the numerical stability constant; The calculation process for the logarithmic amplitude spectrum L1 loss follows the following formula: ; in, Indicates the first i Number of frequency bins at each scale Indicates the first i Number of time frames per scale The first character representing the reconstructed speech signal i Frequency index f and time frame at each scale t The amplitude value at that point, The first part representing the original speech signal i Frequency index at each scale f and time frame t The amplitude value at that point; The calculation process for the linear amplitude spectrum L1 loss follows the following relationship: ; The calculation process for phase consistency loss is based on the following relationship: ; in, This represents the phase spectrum at the i-th scale of the reconstructed speech signal. This represents the phase spectrum at the i-th scale of the original speech signal; The calculation process for the complex spectrum L2 loss follows the following relationship: ; in, The first character representing the reconstructed speech signal i Complex spectrum at various scales The first part representing the original speech signal i Complex spectrum at multiple scales.

[0029] During the reasoning stage: Step 9: Input the verification text into the speech synthesis model, align the predicted duration of each phoneme as the alignment reference for the enhanced text embedding, and input it into the prior encoder to generate the text prior distribution. Step 10: Sample the prior distribution of the text and input it into the decoder to obtain the speech synthesis result.

[0030] Please refer to Figure 4 This embodiment also provides a speech synthesis system based on VITS improvement, wherein the system applies the VITS-based speech synthesis method described above, and the system includes: The training module is used for: Using the VITS model as the base model, and given the input text and the original speech signal, input the input text and the original speech signal into the base model; Phonetic encoding is performed using a semantic encoder based on a large language model to capture the sentiment, intent, and speaking style of the input text, generating an enhanced text embedding containing deep semantic information. The original speech signal is subjected to short-time Fourier transform to generate a linear spectrum, which is then input into the posterior encoder to obtain the posterior distribution of the speech, and latent variables are sampled. The latent variables are input into the decoder to generate frequency domain acoustic features. The frequency domain acoustic features are then subjected to inverse short-time Fourier transform to reconstruct the time domain speech waveform, thus obtaining the reconstructed speech signal. The enhanced text embedding and latent variables are input into the MAS module for monotonic alignment search. During the monotonic alignment search process, a random perturbation that gradually decays with the scale of training is introduced to obtain the alignment path. The enhanced text is embedded into a random duration predictor to obtain the predicted duration for each phoneme; The enhanced text embedding is aligned according to the alignment path and fed into the prior encoder to generate the text prior distribution; The total loss function is constructed based on the reconstructed speech signal, the original speech signal, the prediction duration, the text prior distribution and the speech posterior distribution, the basic model is trained in an adversarial learning manner according to the total loss function, and after the training is completed, a speech synthesis model is obtained; The inference module is configured to: The verification text is input into the speech synthesis model to obtain the text prior distribution, the text prior distribution is sampled and input into the decoder to obtain a speech synthesis result.

[0031] In order to evaluate the effectiveness and performance of the present application, the synthesized speech effect is evaluated as follows: The present embodiment uses the LJSpeech dataset. The LJSpeech dataset consists of 13100 short audio segments of a single speaker, with a total length of about 24 hours, and the audio format is 16-bit PCM with a sampling rate of 22kHz. The dataset is randomly divided into a training set (12,500 samples), a validation set (100 samples) and a test set (500 samples).

[0032] Before training, the dataset is first subjected to endpoint detection processing to remove the silent segments at the beginning and end of the audio file, and pre-processing, frame windowing and other pre-processing operations are performed. The linear spectrogram obtained from the reconstructed speech signal by short-time Fourier transform (STFT) is used as the input of the post-encoder. The specific parameters for extracting features are shown in Table 1.

[0033] Table 1: Feature extraction parameters

[0034] The present embodiment is implemented using Python language based on PyTorch on a Windows 10 64-bit operating system, the processor is Inter Core i9 10900K @ 3.70GHz, the graphics card is NVIDIA GeForce RTX 2080Ti, the graphics card memory is 12G, and the Pytorch version is 11.3. The hyperparameters during the experiment are shown in Table 2.

[0035] Table 2: Hyperparameters in the experiment

[0036] To comprehensively evaluate the synthesized speech effect, the present embodiment analyzes the system from two dimensions of speech quality and synthesis speed.

[0037] (1) Speech quality evaluation In terms of speech synthesis quality, the mean opinion score (MOS) is used as the subjective evaluation index, and the mel cepstral distortion (MCD) is introduced as the objective evaluation standard.

[0038] In the subjective evaluation process, 10 evaluators with good English language skills were invited to participate in the MOS test. The test set text was synthesized using different systems, each evaluator listened to randomly assigned audio samples, and evaluated the naturalness of the speech based on a 5-point rating scale. To ensure fairness, all audio samples were amplitude normalized, and each evaluator evaluated each sample only once. Finally, after removing the highest and lowest scores, the 20 speech samples generated by each system were averaged to obtain the system's overall score.

[0039] In the objective evaluation, the Mel Cepstral Distortion (MCD) value was used to measure the spectral difference between the synthesized speech and the target speech. The lower the MCD value, the smaller the spectral distortion of the synthesized speech, and the higher the speech quality.

[0040] (2) Synthesis speed To verify the improvement of the LC-VITS model in inference speed, the real-time factor RTF (Real-Time Factor) was used as an evaluation index, which is defined as the ratio of the processing time required to generate speech to the actual length of the speech, and the calculation formula is as follows: ; wherein, RTF represents the real-time factor, t represents the time spent generating speech, T represents the length of the generated speech. The lower the RTF, the faster the speed of the system to generate speech. RTF<1 represents real-time or faster generation. The time required to generate speech is less than the length of the speech. For example, RTF=0.5 means that it takes only 30 seconds to generate a minute of speech. RTF>1 represents non-real-time, and the time required to generate speech is longer than the length of the speech. For example, RTF=2 means that it takes 2 minutes to generate a minute of speech. The evaluation was conducted in an Intel i7@2.3GHz single-threaded environment, using the test set of 100 sentences to compare the original VITS model and the improved LC-VITS model, to evaluate the optimization effect of inference speed.

[0041] The LC-VITS model based on the improvement of VITS proposed in this embodiment can effectively improve the inference speed, emotional expression ability and flexibility of style control of speech synthesis, providing a new solution for cross-language diversified speech synthesis and providing a reference for the development of speech synthesis technology in a more efficient and intelligent direction.

[0042] It should be appreciated that parts of the present application can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, several steps or methods can be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented in hardware, and as in another embodiment, any of the following technologies, or a combination thereof, can be used: discrete logic circuitry having logic gates for implementing logic functions upon an application of data signals, application specific integrated circuits having appropriate combinational logic gates, programmable gate arrays (PGA), field programmable gate arrays (FPGA), and the like.

[0043] In the description of the present specification, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the present specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0044] The above-described embodiments only express several implementation manners of the present application, which are described in a more specific and detailed manner, but cannot be understood as a limitation on the patent scope of the present application. It should be noted that, for those skilled in the art, several modifications and improvements can be made without departing from the concept of the present application, which are all within the protection scope of the present application. Therefore, the patent protection scope of the present application should be subject to the appended claims.

Claims

1. A speech synthesis method based on VITS improvement, characterized in that, The method includes the following steps: During the training phase: Using the VITS model as the base model, and given the input text and the original speech signal, input the input text and the original speech signal into the base model; Phonetic encoding is performed using a semantic encoder based on a large language model to capture the sentiment, intent, and speaking style of the input text, generating an enhanced text embedding containing deep semantic information. The original speech signal is subjected to short-time Fourier transform to generate a linear spectrum, which is then input into the posterior encoder to obtain the posterior distribution of the speech, and latent variables are sampled. The latent variables are input into the decoder to generate frequency domain acoustic features. The frequency domain acoustic features are then subjected to inverse short-time Fourier transform to reconstruct the time domain speech waveform, thus obtaining the reconstructed speech signal. The enhanced text embedding and latent variables are input into the MAS module for monotonic alignment search. During the monotonic alignment search process, a random perturbation that gradually decays with the scale of training is introduced to obtain the alignment path. The enhanced text is embedded into a random duration predictor to obtain the predicted duration for each phoneme; The enhanced text embedding is aligned according to the alignment path and fed into the prior encoder to generate the text prior distribution; A total loss function is constructed based on the reconstructed speech signal, the original speech signal, the predicted duration, the prior distribution of the text, and the posterior distribution of the speech. Based on the total loss function, the basic model is trained using an adversarial learning approach. After training, a speech synthesis model is obtained. During the reasoning stage: The validation text is input into the speech synthesis model, and the predicted duration of each phoneme is used as the alignment benchmark for the enhanced text embedding. This is then input into the prior encoder to generate the text prior distribution. The prior distribution of the text is sampled and input into the decoder to obtain the speech synthesis result.

2. The speech synthesis method based on VITS improvement according to claim 1, characterized in that, Using a semantic encoder based on a large language model for phonemic encoding to capture the sentiment, intent, and speaking style of the input text, and generating enhanced text embeddings containing deep semantic information, specifically includes the following steps: The input text is fed into a large language model for semantic extraction, resulting in a global semantic embedding. The input text encoder is acoustically embedded to obtain the acoustic text embedding; The global semantic embedding and the acoustic text embedding are dimensionally aligned and then fused to obtain an enhanced text embedding containing deep semantic information.

3. The speech synthesis method based on VITS improvement according to claim 2, characterized in that, The input text is fed into a large language model for semantic extraction to obtain the global semantic embedding. This process includes the following steps: The input text is fed into the large language model, guided by the first task instruction to generate descriptive sentences. The corresponding process has the following relationship: ； in, This indicates a descriptive sentence. Represents the Llama2 model. This indicates the first task prompt instruction. Indicates the input text; The descriptive sentence is input again into the large language model. Guided by the second task instruction, all token representations of the final hidden layer of the large language model are extracted. The average of all token hidden vectors is then taken to obtain the global semantic embedding. The corresponding process has the following relationship: ； in, Indicates global semantic embedding. Indicates the total number of tokens. In a descriptive sentence, the first... i One token, This indicates the second task prompt command.

4. The speech synthesis method based on VITS improvement according to claim 3, characterized in that, The global semantic embedding and the acoustic text embedding are dimensionally aligned and then fused to obtain an enhanced text embedding containing deep semantic information. The corresponding process has the following relationship: ； in, This represents enhanced text embeddings used for speech synthesis. This represents the original acoustic text embedding of VITS. This represents a linear transformation matrix.

5. The speech synthesis method based on VITS improvement according to claim 4, characterized in that, The decoder includes several stacked ConvNeXt blocks as a basic backbone network for generating Fourier time-frequency coefficients. An embedding layer and a normalization layer are connected in series at the input of the basic backbone network, and an inverse short-time Fourier transform module is connected in series at the output of the basic backbone network.

6. The speech synthesis method based on VITS improvement according to claim 5, characterized in that, The ConvNeXt block includes a 7x7 depthwise convolution, a 1x1 dot convolution, and a 1x1 depthwise convolution. The 7x7 depthwise convolution, the 1x1 dot convolution, and the 1x1 depthwise convolution are connected in series using an inverted bottleneck structure. Gaussian error linear units and global response normalization layers are sequentially set between the 1x1 dot convolution and the 1x1 depthwise convolution in the direction of data transfer. A residual link is introduced at the output of the 1x1 depthwise convolution.

7. The speech synthesis method based on VITS improvement according to claim 6, characterized in that, The latent variables are input into the decoder to generate frequency domain acoustic features. Then, the frequency domain acoustic features are subjected to inverse short-time Fourier transform to reconstruct the time domain speech waveform, resulting in the reconstructed speech signal. The specific steps include the following: The latent variables are adjusted through an embedding layer to match the number of FFN points of ISTFT, resulting in a dimension-adjusted feature sequence. The dimension-adjusted feature sequence is normalized and input into the basic backbone network to obtain the Fourier time-frequency coefficients. The corresponding process has the following relationship: ； in, This represents the feature sequence after dimension adjustment. Indicates the embedding layer. This indicates the integration of a global response normalization layer. This indicates the use of stacked basic backbone network ConvNeXts blocks. These represent the Fourier time-frequency coefficients and phase, respectively; The process of constructing complex Fourier coefficients from Fourier time-frequency coefficients follows the following relationship: ； in, Represents the complex Fourier coefficients. Represents the imaginary unit; The complex Fourier coefficients are input into the inverse short-time Fourier transform module based on the Hanning window to reconstruct the time-domain speech waveform, thus obtaining the reconstructed speech signal. The corresponding process has the following relationship: ； in, This indicates the reconstruction of the speech signal. This represents the inverse short-time Fourier transform module based on the Hanning window. Indicates the length of the window. Indicates frame shift.

8. The speech synthesis method based on VITS improvement according to claim 7, characterized in that, The enhanced text embedding and latent variables are input into the MAS module for monotonic alignment search. During the monotonic alignment search process, a random perturbation that gradually decays with the scale of training is introduced to obtain the alignment path. The specific steps include the following: The enhanced text embedding is aligned with the latent variable space dimension using a projection operation to obtain the aligned enhanced text embedding and latent variables; The alignment score matrix is ​​obtained by calculating the log-likelihood scores of the aligned enhanced text embeddings and latent variables. Add a random perturbation that gradually decays with the training scale to each element of the alignment score matrix, and use dynamic programming to accumulate the maximum log-likelihood along the monotonic path to obtain the maximum log-likelihood matrix. The corresponding process has the following relationship: ； in, This indicates the index position of the input text sequence. a The index positions of the posterior speech features are b time Q The value, i.e., the maximum log-likelihood accumulation value during the forward operation, Indicates the corresponding position is a-1 and b-1 time Q value, Indicates the corresponding position is a and b-1 time Q value, These represent the index positions of the input text sequence and the posterior speech features, respectively. Indicates the position of the given text. a Under the distribution of , the log probability density of latent speech features, This indicates the index position of the input text sequence and the posterior speech features. This represents the mean. Indicates standard deviation, This represents a random perturbation that gradually decays in scale as training progresses; Based on the maximum log-likelihood matrix, path backtracking is performed to obtain the aligned path; The calculation process for the random perturbation, which gradually decays with the training scale, follows the following relationship: ； in, Represents the noise scaling factor. This represents the standard deviation of the probability distribution of the current alignment path. This represents the sampling noise of a standard normal distribution.

9. The speech synthesis method based on VITS improvement according to claim 8, characterized in that, The total loss function has the following relationship: ； in, These represent reconstruction loss, KL loss, duration loss, adversarial training loss, feature matching loss, and multi-subband scale loss, respectively. Represents the total loss function; The calculation process for the multi-subband scale loss is based on the following relationship: ； in, Indicates the total number of scales. This represents the loss weight at the i-th scale. Let represent the loss at the i-th scale. The calculation process for the loss at the i-th scale follows the following relationship: ； in, They represent the first i The spectral convergence loss, logarithmic amplitude spectrum L1 loss, linear amplitude spectrum L1 loss, phase consistency loss, and complex spectrum L2 loss are calculated at each scale. express The corresponding weights; The calculation process for the spectral convergence loss follows the following formula: ； in, Represents the original speech signal. Let represent the complex spectrum at the i-th scale of the reconstructed speech signal. Let represent the complex spectrum at the i-th scale of the original speech signal. Describing the Frobenius norm, Represents the numerical stability constant; The calculation process for the logarithmic amplitude spectrum L1 loss follows the following formula: ； in, This represents the number of frequency bins at the i-th scale. This represents the number of time frames at the i-th scale. This represents the frequency index f and the amplitude value at time frame t at the i-th scale of the reconstructed speech signal. This represents the frequency index f and the amplitude value at time frame t at the i-th scale of the original speech signal; The calculation process for the linear amplitude spectrum L1 loss follows the following relationship: ； The calculation process for phase consistency loss is based on the following relationship: ； in, This represents the phase spectrum at the i-th scale of the reconstructed speech signal. This represents the phase spectrum at the i-th scale of the original speech signal; The calculation process for the complex spectrum L2 loss follows the following relationship: ； in, The first character representing the reconstructed speech signal i Complex spectrum at various scales Let represent the complex spectrum of the i-th scale of the original speech signal.

10. A speech synthesis system based on VITS improvement, characterized in that, The system applies the VITS-based improved speech synthesis method as described in any one of claims 1 to 9, and the system comprises: The training module is used for: Using the VITS model as the base model, and given the input text and the original speech signal, input the input text and the original speech signal into the base model; Phonetic encoding is performed using a semantic encoder based on a large language model to capture the sentiment, intent, and speaking style of the input text, generating an enhanced text embedding containing deep semantic information. The original speech signal is subjected to short-time Fourier transform to generate a linear spectrum, which is then input into the posterior encoder to obtain the posterior distribution of the speech, and latent variables are sampled. The latent variables are input into the decoder to generate frequency domain acoustic features. The frequency domain acoustic features are then subjected to inverse short-time Fourier transform to reconstruct the time domain speech waveform, thus obtaining the reconstructed speech signal. The enhanced text embedding and latent variables are input into the MAS module for monotonic alignment search. During the monotonic alignment search process, a random perturbation that gradually decays with the scale of training is introduced to obtain the alignment path. The enhanced text is embedded into a random duration predictor to obtain the predicted duration for each phoneme; The enhanced text embedding is aligned according to the alignment path and fed into the prior encoder to generate the text prior distribution; A total loss function is constructed based on the reconstructed speech signal, the original speech signal, the predicted duration, the prior distribution of the text, and the posterior distribution of the speech. Based on the total loss function, the basic model is trained using an adversarial learning approach. After training, a speech synthesis model is obtained. The inference module is used for: The validation text is input into the speech synthesis model to obtain the text prior distribution; the text prior distribution is sampled and sent to the decoder to obtain the speech synthesis result.

Images