Feature fusion speech synthesis method and device based on VITS

Abstract

According to the feature fusion speech synthesis method and device based on the VITS, the expressive force and naturalness of speech can be improved, high-quality speech which is more natural, smoother and rich in expressive force is generated, and the expression is particularly outstanding in speech synthesis of multiple languages, multiple emotion expressions and complex contexts. The method comprises the following steps: (1) generating rhythm features matched with input text information according to the input text information; (2) establishing a deep learning model based on Transform; (3) comprehensively processing the rhythm features and the text features by adopting a multi-modal feature fusion method; (4) inputting the comprehensive feature vector into a speech synthesis network based on a variational auto-encoder for processing, wherein the speech synthesis network is trained by adopting a generative adversarial network and a training method based on maximum likelihood; and (5) adopting a neural network model of a random time length predictor, and performing input data transformation through a multi-layer normalized flow structure.

Description

Technical Field

[0001] The present invention relates to the technical field of speech synthesis, and in particular to a feature fusion speech synthesis method based on VITS (Conditional Variational Inference with adversarial learning for end-to-end Text-to-Speech), and a feature fusion speech synthesis device based on VITS. Background Art

[0002] With the rapid development of artificial intelligence and deep learning technologies, Text-to-Speech (TTS) technology has made significant progress, especially the application of end-to-end models based on deep neural networks. These models can directly convert text into natural and fluent speech, and have been widely used in voice assistants, intelligent customer service, automatic translation and other fields. Currently, the most common TTS models include autoregressive models (such as the Tacotron series) and non-autoregressive models (such as the Glow-TTS and FastSpeech series). These models can generate highly natural synthesized speech by learning a large amount of speech and text data.

[0003] However, although existing TTS systems have achieved good results in speech clarity and accuracy, they still have shortcomings in speech naturalness, rhythmic expression, and emotional expression. Traditional TTS systems mainly rely on text encoding and audio generation networks to achieve speech generation through the mapping of acoustic features and audio waveforms. Most of these systems focus on text-to-speech conversion, that is, how to accurately match the audio output of each word or phoneme to the text input. Although this approach can achieve good results in simple speech synthesis tasks, it still has limitations when dealing with complex contexts, rich emotions, and diverse language expressions.

[0004] 1. Prosodic modeling issues in speech synthesis

[0005] Prosody refers to the rhythm, stress, intonation, pauses and other features of a language, which plays a decisive role in the naturalness, expressiveness and emotional transmission of speech. In natural language, prosodic features are closely related to grammar, semantics, emotion, etc. For example, the rise in intonation in interrogative sentences, the change in stress when emphasizing words, and the adjustment of pauses and speech speed can significantly change the meaning and emotional color of a sentence. Although existing TTS systems have made some progress in prosodic modeling, most systems still rely mainly on predefined rules or simple feature extraction, lacking flexibility and adaptability to contextual situations, resulting in the lack of expressiveness and emotional richness of the generated speech in complex contexts.

[0006] Traditional prosody modeling methods mostly rely on manually designed rules or shallow feature extraction. This method lacks the understanding of the deep semantics and emotions of the language, resulting in inaccurate prosody control during speech generation. In addition, many existing systems have weak prosodic features when dealing with different languages, dialects, and emotional changes, and often cannot capture the subtle differences in speech well, which in turn affects the naturalness and fluency of speech.

[0007] 2. Challenges of multimodal feature fusion

[0008] In recent years, with the development of deep learning technology, end-to-end TTS models have gradually been able to learn richer features, including acoustic features of speech, semantic information of text, and information such as emotion and prosody. These models combine different types of input features through multimodal feature fusion to generate more natural and expressive speech. However, the fusion process of multimodal features still faces the following challenges:

[0009] 1) Complexity of feature selection and fusion: Text, prosody, and acoustic features usually have different scales and expressions. How to effectively fuse these features to ensure accurate semantic communication while avoiding over-smoothing or information loss in audio generation is an important challenge.

[0010] 2) Context dependency problem: In the process of speech generation, prosodic features are closely related to the context. How to dynamically adjust the prosody by combining context information during the multimodal feature fusion process remains an urgent problem to be solved.

[0011] 3) Poor adaptability to multiple languages ​​and styles: Existing models usually require a large amount of training data when dealing with multi-language and multi-style speech synthesis tasks, and it is difficult to achieve rhythmic adaptation between languages ​​and styles.

[0012] These challenges indicate that there is still room for improvement in the existing TTS systems in terms of multimodal feature fusion, especially in terms of the diversity and naturalness of rhythm modeling and generation. Summary of the invention

[0013] In order to overcome the defects of the prior art, the technical problem to be solved by the present invention is to provide a feature fusion speech synthesis method based on VITS, which can improve the expressiveness and naturalness of speech, generate more natural, fluent and expressive high-quality speech, especially in speech synthesis of multiple languages, multiple emotional expressions and complex contexts.

[0014] The technical solution of the present invention is: the feature fusion speech synthesis method based on VITS comprises the following steps:

[0015] (1) Generate matching prosodic features based on the input text information. Using an end-to-end training method, the prosodic generation model adjusts the generated prosodic features according to the content and context of the text during the training process to adapt to different contexts and emotional needs.

[0017] (2) Establish a Transformer-based deep learning model, adopt a bidirectional encoding mechanism, and consider the contextual information before and after text encoding to obtain text features;

[0018] (3) A multimodal feature fusion method is used to comprehensively process the prosodic features and text features to form a comprehensive feature vector containing semantic information and prosodic information. The text features and prosodic features are respectively transmitted to multiple independent attention heads through a multi-head attention mechanism, and the correlation between them is calculated. The respective contributions are adjusted in a weighted manner. The multi-head attention mechanism captures the complex relationship between text and prosodic features in a high dimension by processing multiple attention heads in parallel.

[0019] (4) The comprehensive feature vector is input into a speech synthesis network based on a variational autoencoder for processing, and the text, prosody, and other relevant features are mapped to the audio waveform in an end-to-end manner to achieve speech synthesis; the speech synthesis network is trained using a generative adversarial network and a maximum likelihood-based training method;

[0020] (5) A neural network model with a random duration predictor is used to transform the input data through a multi-layer normalized stream structure. Each stream consists of nonlinear operations, including convolutional layers and affine transformations, which adjust the scale and position of the data in order to model complex duration distributions.

[0021] The present invention significantly improves the naturalness and fluency of speech synthesis by introducing a prosody generator and multimodal feature fusion. This method can improve the expressiveness and naturalness of speech, and generate more natural, fluent and expressive high-quality speech, especially in speech synthesis of multiple languages, multiple emotional expressions and complex contexts.

[0022] A feature fusion speech synthesis device based on VITS is also provided, which includes:

[0023] The rhythm generation module is configured to generate rhythmic features that match the input text information. The end-to-end training method is adopted. During the training process, the rhythm generation model adjusts the generated rhythmic features according to the content and context information of the text to adapt to different contexts and emotional needs.

[0024] The deep learning module is configured to build a Transformer-based deep learning model, which uses a bidirectional encoding mechanism to consider the contextual information before and after text encoding to obtain text features;

[0025] The multimodal feature fusion module is configured to use a multimodal feature fusion method to comprehensively process the rhythmic features and text features to form a comprehensive feature vector containing semantic information and rhythmic information. The multi-head attention mechanism transmits the text features and rhythmic features to multiple independent attention heads respectively, calculates the correlation between them, and adjusts their respective contributions in a weighted manner. The multi-head attention mechanism processes multiple attention heads in parallel.

[0026] head,captures the complex relationship between text and prosodic features in high dimensions;

[0027] A speech synthesis module, which is configured so that the comprehensive feature vector is input into a speech synthesis network based on a variational autoencoder for processing, and the text, prosody and other relevant features are mapped to the audio waveform in an end-to-end manner to achieve speech synthesis; the speech synthesis network is trained using a generative adversarial network and a maximum likelihood-based training method;

[0028] The neural network module is configured to use a neural network model of a random duration predictor to transform the input data through a multi-layer normalized flow structure. Each flow consists of non-linear operations, including convolutional layers and affine transformations to adjust the scale and position of the data in order to model complex duration distributions. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 1 is a diagram showing the working principle of the feature fusion speech synthesis method based on VITS according to the present invention.

[0030] Figure 2 It is a schematic diagram of the rhythm generation module structure in a specific embodiment of the present invention.

[0031] Figure 3 This is a schematic diagram of the BERT feature extraction proposed in the present invention.

[0032] Figure 4 It is a schematic diagram of a multimodal feature fusion module in a specific embodiment of the present invention.

[0033] Figure 5 It is a schematic diagram of the training part of the random duration predictor in a specific embodiment of the present invention.

[0034] Figure 6 It is a schematic diagram of the reasoning part of the random duration predictor in a specific embodiment of the present invention.

[0035] Figure 7The figure is a flow chart of the feature fusion speech synthesis method based on VITS according to the present invention. DETAILED DESCRIPTION

[0036] like Figure 7 As shown, this feature fusion speech synthesis method based on VITS includes the following steps:

[0037] (1) Generate matching prosodic features based on the input text information. Using an end-to-end training method, the prosodic generation model adjusts the generated prosodic features according to the content and context of the text during the training process to adapt to different contexts and emotional needs.

[0038] (2) Establish a Transformer-based deep learning model, adopt a bidirectional encoding mechanism, and consider the contextual information before and after text encoding to obtain text features;

[0039] (3) A multimodal feature fusion method is used to comprehensively process the prosodic features and text features to form a comprehensive feature vector containing semantic information and prosodic information. The text features and prosodic features are respectively transmitted to multiple independent attention heads through a multi-head attention mechanism, and the correlation between them is calculated. The respective contributions are adjusted in a weighted manner. The multi-head attention mechanism captures the complex relationship between text and prosodic features in a high dimension by processing multiple attention heads in parallel.

[0040] (4) The comprehensive feature vector is input into a speech synthesis network based on a variational autoencoder for processing, and the text, prosody, and other relevant features are mapped to the audio waveform in an end-to-end manner to achieve speech synthesis; the speech synthesis network is trained using a generative adversarial network and a maximum likelihood-based training method;

[0041] (5) A neural network model using a random duration predictor transforms the input data through a multi-layer normalized stream structure. Each stream consists of nonlinear operations, including convolutional layers and affine transformations to adjust the scale and position of the data.

[0042] In order to model complex duration distributions.

[0043] The present invention significantly improves the naturalness and fluency of speech synthesis by introducing a prosody generator and multimodal feature fusion. This method can improve the expressiveness and naturalness of speech, and generate more natural, fluent and expressive high-quality speech, especially in speech synthesis of multiple languages, multiple emotional expressions and complex contexts.

[0044] Preferably, the step (1) comprises the following sub-steps:

[0045] (1.1) Model input

[0046] The output of the text encoder is a high-dimensional feature vector, which represents the semantic information of the input text. This vector is used as the input of the prosody generation model to capture the structural and semantic features of the text. The output of the text encoder is:

[0047] X=[x 1 , x 2 , x 3 , …, x n ]

[0048] Where X represents the feature matrix of the text, x i Indicates the representation of the i-th character or word;

[0049] (1.2) Prosodic feature prediction

[0050] Establish a prosody generation model, dynamically generate the prosody parameters of speech by learning the relationship between input features and target prosody features; realize mapping through mapping function f, h = f(x i ), the prosody generation model dynamically calculates these predicted prosody features based on the input text features and their contextual relationships:

[0051] Y=[y 1 ,y 2 ,y 3 , ..., y n ]

[0052] Prosodic features are directly generated through the output layer. These prosodic features include intonation, stress, pauses, and speaking speed. i represents the prosodic features of the i-th word;

[0053] (1.3) Training and Optimization

[0054] Using mean square error:

[0055]

[0056] in, is the prosodic feature of the i-th word predicted by the model, y i is the actual rhythmic feature, and the model parameters are updated through the back propagation algorithm. In order to consider the contextual relationship, the self-attention mechanism unit is introduced to capture the long-term dependency in the text. The model calculates the attention matrix and weights the features of each word:

[0057]

[0058] Where Q and K are query and key matrices respectively, d k is the dimension of the key, and then the final output is generated by weighted feature representation:

[0059] Y=AV

[0060] Among them, V is the value matrix, which is obtained by multiplying each element of the input sequence (such as a word embedding vector) by the weight matrix, and Y is the generated prosodic feature.

[0061] Preferably, in step (1.3), a text dataset with contextual information is used to train the prosody generation model, each training sample includes an input text and a corresponding prosody label, and through a back-propagation algorithm, the prosody generation model adjusts its parameters to minimize the error between the predicted and actual prosody features.

[0062] Preferably, the step (2) comprises the following sub-steps:

[0063] (2.1) Text segmentation: Use a tokenizer to split the input text into words or subwords;

[0064] (2.2) Bidirectional context modeling: Using the Transformer’s self-attention mechanism, combined with bidirectional encoding, context information is learned from left to right and from right to left simultaneously to generate a contextual representation of each word;

[0065] (2.3) Pre-training and fine-tuning: Pre-training is performed using large-scale text data, and then fine-tuning is performed based on downstream tasks to focus on feature learning for specific tasks.

[0066] Preferably, in step (3), the calculation formula for each attention head is:

[0067]

[0068] Where Q is the query matrix, K is the key matrix, and d k is the dimension of the key, and the calculated A is the attention weight matrix; the feature representation Y generated by each attention head is obtained by the following formula:

[0069] Y=AV

[0070] Among them, V is the value matrix, which is obtained by multiplying each element of the input sequence (such as word embedding vector) with the weight matrix, and Y is the fused prosodic features and text features.

[0071] Preferably, in step (4), the optimization goal of speech synthesis is:

[0072] L=L MSE +L GAN +L VAE

[0073] Among them, L MSEis the mean square error loss, which measures the difference between the generated speech and the real speech;

[0074] L GAN To generate adversarial loss, it is used to improve the naturalness of speech; L VAE is the loss of the variational autoencoder to ensure the quality of the generated speech.

[0075] Preferably, in step (5), the input of the posterior encoder is z, which is a sample value of a standard normal distribution. A sample point in a Gaussian distribution with a mean of 0 and a variance of 1 is transformed into a random distribution u through flow, and the sigmoid function makes u satisfy the following formula:

[0076]

[0077] d is the training cycle. There are two conditional encoders in the structure, which receive d and text h respectively. text , the output is added to get the condition c text , u is a normal distribution between 0 and 1, and du is the actual number of frames rounded up.

[0078] Preferably, in step (5), the continuous data du is modeled and the mathematical expectation of the likelihood value is maximized to satisfy the following formula:

[0079]

[0080] Those skilled in the art can understand that all or part of the steps in the above-mentioned embodiment method can be implemented by instructing the relevant hardware through a program, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes the steps of the above-mentioned embodiment method, and the storage medium can be: ROM / RAM, disk, CD, memory card, etc. Therefore, corresponding to the method of the present invention, the present invention also includes a feature fusion speech synthesis device based on VITS, which is usually represented in the form of functional modules corresponding to the steps of the method. The device includes:

[0081] The rhythm generation module is configured to generate rhythmic features that match the input text information. The end-to-end training method is adopted. During the training process, the rhythm generation model adjusts the generated rhythmic features according to the content and context information of the text to adapt to different contexts and emotional needs.

[0082] The deep learning module is configured to build a Transformer-based deep learning model, which uses a bidirectional encoding mechanism to consider the contextual information before and after text encoding to obtain text features;

[0083] The multimodal feature fusion module is configured to use a multimodal feature fusion method to comprehensively process the rhythmic features and text features to form a comprehensive feature vector containing semantic information and rhythmic information. The multi-head attention mechanism transmits the text features and rhythmic features to multiple independent attention heads respectively, calculates the correlation between them, and adjusts their respective contributions in a weighted manner. The multi-head attention mechanism processes multiple attention heads in parallel.

[0084] head,captures the complex relationship between text and prosodic features in high dimensions;

[0085] A speech synthesis module, which is configured so that the comprehensive feature vector is input into a speech synthesis network based on a variational autoencoder for processing, and the text, prosody and other relevant features are mapped to the audio waveform in an end-to-end manner to achieve speech synthesis; the speech synthesis network is trained using a generative adversarial network and a maximum likelihood-based training method;

[0086] The neural network module is configured to use a neural network model of a random duration predictor to transform the input data through a multi-layer normalized flow structure. Each flow consists of non-linear operations, including convolutional layers and affine transformations to adjust the scale and position of the data in order to model complex duration distributions.

[0087] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Any simple modification, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention are still within the protection scope of the technical solution of the present invention.

Claims

1. A feature fusion speech synthesis method based on VITS, characterized by: It includes the following steps: (1) Generate matching prosodic features based on the input text information. Using an end-to-end training method, the prosodic generation model adjusts the generated prosodic features according to the content and context of the text during the training process to adapt to different contexts and emotional needs. (2) Establish a Transformer-based deep learning model, adopt a bidirectional encoding mechanism, and consider the contextual information before and after text encoding to obtain text features; (3) A multimodal feature fusion method is used to comprehensively process the prosodic features and text features to form a comprehensive feature vector containing semantic information and prosodic information. The text features and prosodic features are respectively transmitted to multiple independent attention heads through a multi-head attention mechanism, and the correlation between them is calculated. The respective contributions are adjusted in a weighted manner. The multi-head attention mechanism captures the complex relationship between text and prosodic features in a high dimension by processing multiple attention heads in parallel. (4) The comprehensive feature vector is input into a speech synthesis network based on a variational autoencoder for processing, and the text, prosody, and other relevant features are mapped to the audio waveform in an end-to-end manner to achieve speech synthesis; the speech synthesis network is trained using a generative adversarial network and a maximum likelihood-based training method; (5) A neural network model with a random duration predictor is used to transform the input data through a multi-layer normalized stream structure. Each stream consists of nonlinear operations, including convolutional layers and affine transformations, which adjust the scale and position of the data in order to model complex duration distributions.

2. The feature fusion speech synthesis method based on VITS according to claim 1 is characterized in that: The step (1) comprises the following sub-steps: (1.1) Model input The output of the text encoder is a high-dimensional feature vector, which represents the semantic information of the input text. This vector is used as the input of the prosody generation model to capture the structural and semantic features of the text. The output of the text encoder is: X=[x1,x2,x3,...,x n ] Where X represents the feature matrix of the text, x i Indicates the representation of the i-th character or word; (1.2) Prosodic feature prediction Establish a prosody generation model, dynamically generate the prosody parameters of speech by learning the relationship between input features and target prosody features; realize mapping through mapping function f, h = f(x i ), the prosody generation model dynamically calculates these predicted prosody features based on the input text features and their contextual relationships: <h2 style=";text-align:left;direction:ltr">Y = [y1,y2,y3,...,y<h2 style=";text-align:left;direction:ltr"> n <h2 style=";text-align:left;direction:ltr"> ] Prosodic features are directly generated through the output layer. These prosodic features include intonation, stress, pauses, and speaking speed. i represents the prosodic features of the i-th word; (1.3) Training and Optimization Using mean square error: in, is the prosodic feature of the i-th word predicted by the model, y i is the actual rhythmic feature, and the model parameters are updated through the back propagation algorithm. In order to consider the contextual relationship, the self-attention mechanism unit is introduced to capture the long-term dependency in the text. The model calculates the attention matrix and weights the features of each word: Where Q and K are query and key matrices respectively, d k is the dimension of the key, and then the final output is generated by weighted feature representation: Y=AV Among them, V is the value matrix obtained by multiplying each element of the input sequence by the weight matrix, and Y is the generated prosodic feature.

3. The feature fusion speech synthesis method based on VITS according to claim 2 is characterized in that: In the step (1.3), a text dataset with contextual information is used to train a prosody generation model, where each training sample includes an input text and a corresponding prosody label. Through a back-propagation algorithm, the prosody generation model adjusts its parameters to minimize the error between the predicted and actual prosody features.

4. The feature fusion speech synthesis method based on VITS according to claim 3 is characterized in that: The step (2) comprises the following sub-steps: (2.1) Text segmentation: Use a tokenizer to split the input text into words or subwords; (2.2) Bidirectional context modeling: Using the Transformer’s self-attention mechanism, combined with bidirectional encoding, context information is learned from left to right and from right to left simultaneously to generate a contextual representation of each word; (2.3) Pre-training and fine-tuning: Pre-training is performed using large-scale text data, and then fine-tuning is performed based on downstream tasks to focus on feature learning for specific tasks.

5. The feature fusion speech synthesis method based on VITS according to claim 4 is characterized in that: In step (4), the optimization goal of speech synthesis is: L=L MSE +L GAN +L VAE Among them, L MSE is the mean square error loss, which measures the difference between the generated speech and the real speech; L GAN To generate adversarial loss, it is used to improve the naturalness of speech; L VAE is the loss of the variational autoencoder to ensure the quality of the generated speech.

6. The feature fusion speech synthesis method based on VITS according to claim 5 is characterized in that: In step (5), the input of the posterior encoder is z, which is a sample value of a standard normal distribution. A sample point in a Gaussian distribution with a mean of 0 and a variance of 1 is transformed into a random distribution u through flow, and the sigmoid function makes u satisfy the following formula: d is the training cycle. There are two conditional encoders in the structure, which receive d and text h respectively. text , the output is added to get the condition c text , u is a normal distribution between 0 and 1, and du is the actual number of frames rounded up.

7. The feature fusion speech synthesis method based on VITS according to claim 6 is characterized in that: In step (5), the continuous data du is modeled and the mathematical expectation of the likelihood value is maximized to satisfy the following formula:

8. A feature fusion speech synthesis device based on VITS, characterized in that: It includes: The rhythm generation module is configured to generate rhythmic features that match the input text information. The end-to-end training method is adopted. During the training process, the rhythm generation model adjusts the generated rhythmic features according to the content and context information of the text to adapt to different contexts and emotional needs. The deep learning module is configured to build a Transformer-based deep learning model, which uses a bidirectional encoding mechanism to consider the contextual information before and after text encoding to obtain text features; The multimodal feature fusion module is configured to use a multimodal feature fusion method to comprehensively process the prosodic features and text features to form a comprehensive feature vector containing semantic information and prosodic information. The multi-head attention mechanism transmits the text features and prosodic features to multiple independent attention heads respectively, calculates the correlation between them, and adjusts their respective contributions in a weighted manner. The multi-head attention mechanism captures the complex relationship between text and prosodic features in a high dimension by processing multiple attention heads in parallel; A speech synthesis module, which is configured so that the comprehensive feature vector is input into a speech synthesis network based on a variational autoencoder for processing, and the text, prosody and other relevant features are mapped to the audio waveform in an end-to-end manner to achieve speech synthesis; the speech synthesis network is trained using a generative adversarial network and a maximum likelihood-based training method; The neural network module is configured to use a neural network model of a random duration predictor to transform the input data through a multi-layer normalized flow structure. Each flow consists of non-linear operations, including convolutional layers and affine transformations to adjust the scale and position of the data in order to model complex duration distributions.

Images