A multi-attribute speech synthesis method based on a generative adversarial network
By using a multi-attribute speech synthesis method based on generative adversarial networks, the problem of insufficient personalization and diversification in existing speech synthesis technologies is solved, thereby improving the naturalness and diversity of speech synthesis and providing easily controllable speech feature editing capabilities.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- CHONGQING UNIV
- Filing Date
- 2022-09-08
- Publication Date
- 2026-07-24
AI Technical Summary
Existing speech synthesis technology is insufficient in synthesizing personalized and diverse speech, making it difficult to meet users' demand for high-quality speech.
A multi-attribute speech synthesis method based on generative adversarial networks is adopted. By training multiple attribute partitioning hyperplanes through a vocoder, a generator-encoder recurrent network structure and a support vector machine, the multi-attribute editing and synthesis of speech can be realized.
It improves the naturalness and diversity of speech synthesis, enables easy control and quantitative editing of speech features, and can generate personalized and diverse speech.
Smart Images
Figure CN116312455B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of artificial intelligence and speech signal processing technology, specifically to a multi-attribute speech synthesis method based on generative adversarial networks. Background Technology
[0002] As a key technology for human-computer voice interaction, speech synthesis has a wide range of applications, such as voice assistants, voice broadcasting, smart speakers, audiobooks, travel translators, AI-powered intelligent outbound calling, AI broadcasters, song synthesis, and map navigation. Applications relying on speech synthesis are increasingly penetrating people's daily lives. With changes in business models and application scenarios, users are placing higher demands on the quality of synthesized speech, especially in terms of naturalness and personalized voice. Therefore, speech synthesis has significant theoretical and practical value.
[0003] With the development of deep learning, although the quality and naturalness of speech synthesis have been greatly improved, there are still significant shortcomings in synthesizing personalized and diverse speech. Summary of the Invention
[0004] To address the shortcomings of existing technologies, this invention provides a multi-attribute speech synthesis method based on generative adversarial networks, which solves the problems mentioned in the background section.
[0005] To achieve the above objectives, the present invention provides the following technical solution: a multi-attribute speech synthesis method based on generative adversarial networks, comprising a GAN-based generator in the vocoder, a recurrent network structure of generator-encoder interaction, training multiple single attributes using support vector machines to obtain multiple attribute partitioning hyperplanes, a multi-attribute speech editing and synthesis method, and further comprising the following synthesis steps:
[0006] S1: Train an acoustic model to convert a text sequence into a Mel spectrum sequence;
[0007] S2: Train a vocoder to convert Mel spectrum sequences into corresponding waveform speech, and train a generator based on a generative adversarial network.
[0008] S3: Train an encoder to convert samples (speech) into implicit codes that control the generation of their patterns;
[0009] S4: Select an attribute that needs to be edited in speech, such as the gender of the voice (male or female), the emotion of the voice (happy or sad), the age of the voice (young or old), etc., and assign the corresponding binary labels to the corresponding dataset; use an encoder to convert the binary labeled dataset into the corresponding hidden code, and use a support vector machine to perform binary classification on each attribute, and obtain the hyperplane that divides each attribute into two values.
[0010] S5: Edit the attributes of the hidden code. First, we encode the target speech into the corresponding hidden code using the encoder in S3, and then we edit the hidden code according to the various attributes in S4.
[0011] S6: Use a generator to synthesize speech with corresponding attributes. Combine the hidden code containing feature information encoded in S5 with Mel spectrum data containing speech content information and use the vocoder trained in S2 to obtain the edited speech.
[0012] Preferably, the dataset selected in the S1 process should be rich and diverse, such as including both male and female voices, as well as voices expressing a wide range of emotions, such as happiness and sadness, so as to facilitate the synthesis of multi-attribute speech.
[0013] Preferably, the training of the S2 process should employ adversarial training of the generator and discriminator together, with the losses of the generator (G) and discriminator (D) as follows:
[0014]
[0015] Preferably, in process S3, when training the encoder, the hidden code is first obtained using the encoder. Then, the hidden code is used as the input to the generator to obtain the generated sample (speech). Backpropagation is then performed using gradient descent to update the parameters. It is important to note that during encoder training, it is not necessary to update the generator parameters; only the encoder parameters need to be updated. The desired outcome in this process is to maximize the mutual information between the hidden code c and G(z, c). The definition of mutual information is as follows, where H(·) represents entropy:
[0016] I(X;Y)=H(X)-H(X|Y)=H(Y)-H(Y|X)
[0017] In summary, the entire framework model is based on adversarial training using mutual information regularization, as part of the S2 process.
[0018]
[0019] Preferably, when selecting binarization labels in the S4 process, it is essential to choose data that clearly distinguishes the data and ensure sufficient sample data for the model to be fully trained. This allows for finding suitable hyperplanes to divide the samples according to the corresponding attributes. Assume there are m attributes on the hyperplane, and the unit normal vectors of each hyperplane are n1, n2, n3, and n... m If there exists a unit normal vector n, then we define the distance from a hidden sample z to this hyperplane as:
[0020] d(n, z) = nT z
[0021] Preferably, either single-attribute editing or multi-attribute editing can be used in the S5 process. If the normal vectors of the hyperplanes of each attribute are orthogonal, then we can directly use the following formula for implicit code editing:
[0022] z edit =z+αn
[0023] Here, z represents an implicit code, n is the unit normal vector of a certain attribute hyperplane, and α can be either a number greater than zero or less than zero. If the normal vectors of the hyperplanes of different attributes are not orthogonal, we need to use... Figure 3 The method shown removes the components of other attributes from the hyperplane normal vector of a certain attribute:
[0024]
[0025] Preferably, the GAN-based generator is designed to synthesize speech that is rich, diverse, and easily controlled.
[0026] Preferably, the generator-encoder interaction recurrent network structure is designed to maximize the mutual information between the hidden code and the generated samples. The purpose of this is to allow a hidden code to precisely control the generation of the corresponding feature sample speech.
[0027] Preferably, the support vector machine is trained with multiple single attributes to obtain multiple attribute-dividing hyperplanes, the purpose of which is to use one or more hyperplanes to edit the attributes of the hidden code.
[0028] This invention provides a multi-attribute speech synthesis method based on generative adversarial networks, which has the following beneficial effects:
[0029] 1. This multi-attribute speech synthesis method based on generative adversarial networks (GANs) utilizes a vocoder based on a GAN generator on the general speech synthesis framework, demonstrating significant advantages in the naturalness and diversity of speech synthesis. The recurrent generator-encoder network makes the generated speech features easier to control. Training multiple single-attribute support vector machines allows for qualitative and quantitative control of speech generation. By combining these three technologies, and improving upon mainstream neural speech synthesis models and training methods, this method aims to synthesize personalized, diverse, and multi-attribute speech. It allows for qualitative and quantitative control of speech synthesis across multiple attributes, exhibiting unique and significant advantages in the richness of synthesized speech samples and the controllability of synthesis. Attached Figure Description
[0030] Figure 1This is a schematic diagram illustrating the general process of neural speech synthesis in this invention;
[0031] Figure 2 This is a schematic diagram of an improved vocoder portion of the present invention.
[0032] Figure 3 This is a schematic diagram illustrating conditional editing in a subspace during implicit code editing, as explained in this invention.
[0033] Figure 4 This is a flowchart of the editable multi-attribute speech synthesis based on GAN of the present invention;
[0034] Figure 5 This is a schematic diagram of the structure of multiple support vector machines trained according to the present invention. Detailed Implementation
[0035] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.
[0036] Please see Figures 1 to 5 This invention provides a technical solution: a multi-attribute speech synthesis method based on generative adversarial networks, including a GAN-based generator in the vocoder, a recurrent network structure of generator-encoder interaction, training multiple single attributes using support vector machines to obtain multiple attribute partitioning hyperplanes, a multi-attribute speech editing and synthesis method, and further including the following steps:
[0037] S1: Train an acoustic model to convert a text sequence into a Mel spectrum sequence;
[0038] S2: Train a vocoder to convert Mel spectrum sequences into corresponding waveform speech, and train a generator based on a generative adversarial network.
[0039] S3: Train an encoder to convert samples (speech) into implicit codes that control the generation of their patterns;
[0040] S4: Select an attribute that needs to be edited in speech, such as the gender of the voice (male or female), the emotion of the voice (happy or sad), the age of the voice (young or old), etc., and assign the corresponding binary labels to the corresponding dataset; use an encoder to convert the binary labeled dataset into the corresponding hidden code, and use a support vector machine to perform binary classification on each attribute, and obtain the hyperplane that divides each attribute into two values.
[0041] S5: Edit the attributes of the hidden code. First, we encode the target speech into the corresponding hidden code using the encoder in S3, and then we edit the hidden code according to the various attributes in S4.
[0042] S6: Use a generator to synthesize speech with corresponding attributes. Combine the hidden code containing feature information encoded in S5 with Mel spectrum data containing speech content information and use the vocoder trained in S2 to obtain the edited speech.
[0043] The dataset selected in the S1 process should be rich and diverse, including both male and female voices, as well as voices expressing a wide range of emotions such as happiness and sadness. This is beneficial for the synthesis of multi-attribute speech.
[0044] The training of the S2 process should employ adversarial training of the generator and discriminator together, with the losses for the generator (G) and discriminator (D) as shown below:
[0045]
[0046] In the S3 process, the encoder is first trained to obtain the hidden code. This hidden code is then used as input to the generator to produce the generated sample (speech). Backpropagation using gradient descent is then performed to update the parameters. It's important to note that during encoder training, only the encoder parameters need to be updated, not the generator parameters. The desired outcome is to maximize the mutual information between the hidden code c and G(z, c). The mutual information is defined as follows, where H(·) represents entropy:
[0047] I(X;Y)=H(X)-H(X|Y)=H(Y)-H(Y|X)
[0048] In summary, the entire framework model is based on adversarial training using mutual information regularization, as part of the S2 process.
[0049]
[0050] When selecting binarization labels in the S4 process, it is crucial to choose data that clearly distinguishes the data and ensure sufficient sample data for adequate model training. Find a suitable hyperplane to separate the samples for each attribute. Assume there are m attributes on the hyperplane, and the unit normal vectors of each hyperplane are n1, n2, n3, and n... m If there exists a unit normal vector n, then we define the distance from a hidden sample z to this hyperplane as:
[0051] d(n, z) = n T z
[0052] In the S5 process, both single-attribute and multi-attribute editing can be used. If the normal vectors of the hyperplanes for each attribute are orthogonal, then we can directly use the following formula for implicit editing:
[0053] z edit =z+αn
[0054] Here, z represents an implicit code, n is the unit normal vector of a certain attribute hyperplane, and α can be either a number greater than zero or less than zero. If the normal vectors of the hyperplanes of different attributes are not orthogonal, we need to use... Figure 3 The method shown Figure 3 This explains conditional editing in a subspace during implicit editing, which refers to the direction of the new vector obtained by removing the component of vector n2 from vector n1, and removing the components of other attributes from the hyperplane normal vector of a certain attribute:
[0055]
[0056] GAN-based generators are designed to synthesize rich, diverse, and easily controllable speech.
[0057] The generator-encoder interaction recurrent network structure is designed to maximize the mutual information between the hidden code and the generated samples. The goal is to allow a hidden code to precisely control the generation of speech based on the corresponding feature samples. On the basic framework of general speech synthesis, a vocoder based on a GAN generator is used, which has significant advantages in the naturalness and diversity of speech synthesis. The generator-encoder recurrent network makes the generated speech features easier to control.
[0058] Support Vector Machines (SVMs) are trained with multiple single attributes to obtain multiple attribute-partitioning hyperplanes. The purpose is to use one or more hyperplanes to edit the attributes of the hidden code. Training SVMs with multiple single attributes allows for qualitative and quantitative control over speech generation, and qualitative and quantitative control over speech synthesis across multiple attributes. This provides unique and significant advantages in the richness of synthesized speech samples and the controllability of synthesis.
[0059] In summary, this multi-attribute speech synthesis method based on generative adversarial networks (GANs) first trains an acoustic model to convert text sequences into Mel-spectrum sequences; then, it trains a vocoder to convert the Mel-spectrum sequences into corresponding waveform speech (in this invention, it trains a generator based on a GAN); next, it trains an encoder to convert samples (speech) into hidden codes that control their generation patterns; then, it selects an attribute to be edited, such as gender (male or female), emotion (happy or sad), or age (young or old), and assigns corresponding binary labels to the dataset; the encoder converts the binary-labeled dataset into corresponding hidden codes, and performs binary classification on each attribute using a support vector machine to obtain a hyperplane that divides each attribute; finally, it edits the attributes of the hidden codes. First, the encoder in S3 encodes the target speech into corresponding hidden codes, then edits the hidden codes according to the various attributes in S4, and finally uses the generator to synthesize speech with the corresponding attributes. Figure 1 The speech synthesis process demonstrated in the paper combines the hidden code containing feature information encoded in process S5 with Mel spectrum data containing speech content information, and uses the vocoder trained in S2 to obtain the edited speech. By using a GAN-based vocoder on the basic framework of general speech synthesis, it has significant advantages in the naturalness and diversity of speech synthesis. The recurrent network of generator-encoder makes the generated speech features easier to control. Training multiple single-attribute support vector machines can qualitatively and quantitatively control the generation of speech. Based on mainstream neural speech synthesis, its model and training method are improved, aiming to synthesize personalized, diverse, and multi-attribute speech, and to qualitatively and quantitatively control the synthesis of speech on multiple attributes.
[0060] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.
Claims
1. A multi-attribute speech synthesis method based on generative adversarial networks, characterized in that: This includes GAN-based generators in vocoders, recurrent network structures with generator-encoder interactions, methods for training multiple single attributes using support vector machines to obtain multi-attribute partitioning hyperplanes, and multi-attribute speech editing and synthesis methods. It also includes the following synthesis steps: S1: Train an acoustic model to convert a text sequence into a Mel spectrum sequence; S2: Train a vocoder to convert Mel spectrum sequences into corresponding waveform speech, and train a generator based on a generative adversarial network. S3: Train an encoder to convert samples into implicit codes that control the generation of their patterns; During training, the latent code output by the encoder is input to the generator to form a generator-encoder loop structure to achieve feature mapping between the latent code and speech samples; S4: Select an attribute that needs to be edited in the speech, including the gender of the voice, the emotion of the voice, and the age of the voice, and assign the corresponding binary label to the dataset. The encoder is used to convert the binary labeled dataset into the corresponding hidden code, and a support vector machine is used to perform binary classification on each attribute to obtain the hyperplane that divides each attribute into two values. S5: Edit the attributes of the hidden code. First, we encode the target speech into the corresponding hidden code using the encoder in S3, and then we edit the hidden code according to the various attributes in S4. S6: Use a generator to synthesize speech with corresponding attributes. Combine the hidden code containing feature information encoded in S5 with the Mel spectrum data containing speech content information and use the vocoder trained in S2 to obtain the edited speech.
2. The multi-attribute speech synthesis method based on generative adversarial networks according to claim 1, characterized in that: The dataset selected in the S1 process should be rich and diverse, including both male and female voices, as well as both happy and sad voices, in order to facilitate the synthesis of multi-attribute speech.
3. The multi-attribute speech synthesis method based on generative adversarial networks according to claim 1, characterized in that: The training of the S2 process should employ adversarial training of the generator and discriminator together, with the losses of the generator and discriminator as shown below: 。 4. The multi-attribute speech synthesis method based on generative adversarial networks according to claim 1, characterized in that: In process S3, when training the encoder, the hidden code is first obtained using the encoder. Then, the hidden code is used as input to the generator to obtain generated samples. Backpropagation is then performed using gradient descent to update the parameters. It's important to note that during encoder training, only the encoder parameters need to be updated; the generator parameters do not need to be updated. The desired outcome in this process is to maximize the hidden code. and Mutual information between them, the definition of mutual information is as follows, where Representing entropy: ， In summary, the entire framework model is based on adversarial training using mutual information regularization, as part of the S2 process. 。 5. The multi-attribute speech synthesis method based on generative adversarial networks according to claim 1, characterized in that: When selecting binarization labels in the S4 process, it is essential to choose data that clearly distinguishes the data and ensure sufficient sample data for the model to be adequately trained, finding a suitable hyperplane to separate the samples of the corresponding attributes. Assume there is... A hyperplane with n attributes, where the unit normal vector of each hyperplane is... If there is a unit normal vector So let's define a hidden code sample. The distance to this hyperplane is: 。 6. The multi-attribute speech synthesis method based on generative adversarial networks according to claim 1, characterized in that: In the S5 process, both single-attribute editing and multi-attribute editing can be used. If the normal vectors of the hyperplanes of each attribute are orthogonal to each other, then we can directly use the following formula for implicit code editing: Where z represents an implicit code, n is the unit normal vector of a certain attribute hyperplane, and α can be either a number greater than zero or a number less than zero. If the normal vectors of the hyperplanes of different attributes are not orthogonal, according to the conditional editing method in the subspace during implicit code editing, the components of other attributes are removed from the hyperplane normal vector of a certain attribute. 。 7. The multi-attribute speech synthesis method based on generative adversarial networks according to claim 1, characterized in that: The GAN-based generator is designed to synthesize rich, diverse, and easily controllable speech.
8. The multi-attribute speech synthesis method based on generative adversarial networks according to claim 1, characterized in that: The generator-encoder interaction recurrent network structure is designed to maximize the mutual information between the hidden code and the generated samples. The purpose of this is to allow a hidden code to precisely control the generation of speech from the corresponding feature sample.
9. A multi-attribute speech synthesis method based on generative adversarial networks according to claim 1, characterized in that: The support vector machine is trained with multiple single attributes to obtain multiple attribute-dividing hyperplanes. The purpose is to use one or more hyperplanes to edit the attributes of the hidden code.