A text expression driven based speaker generation method

Abstract

The application discloses a kind of speaker generation methods based on text expression driving, it is related to speech synthesis field, including the following steps: text expression is input to description encoder, the text expression is set to describe the timbre characteristics of speaker;Description encoder uses discriminative method and / or generative method, and text expression is encoded as timbre feature vector, and text expression and the timbre characteristics of speaker are mapped to the same space;Timbre feature vector and text content are input to multi-speaker TTS system, and multi-speaker TTS system generates corresponding timbre voice;The timbre of the voice generated by multi-speaker TTS system is consistent with the timbre encoded in timbre feature vector, and the content of voice is consistent with the text content input.The application can use any text description, generate the representation of corresponding timbre speaker, which can be used to drive multi-speaker system to generate corresponding timbre voice, and can achieve trade-off in maintaining text description and generation quality.

Description

Technical Field

[0001] This invention relates to the field of speech synthesis, and more particularly to a speaker generation method based on text representation. Background Technology

[0002] In multi-speaker speech synthesis technology, when aiming to synthesize speech with a specific speaker's timbre, a reference speech segment for that speaker is often provided to the system to extract their timbre information. Based on this limitation, the synthesizable timbre is restricted to the timbre corresponding to the speaker whose reference speech can be found. Furthermore, in some gaming scenarios, using someone else's timbre to assign voices to game characters may raise legal issues. Therefore, research is needed on how to represent a speaker's timbre without a reference speech, or to generate the desired timbre from a non-existent speaker, and synthesize speech with the desired timbre.

[0003] Therefore, those skilled in the art are dedicated to developing a speaker generation method based on text representation. Summary of the Invention

[0004] In view of the above-mentioned deficiencies of the prior art, the technical problem to be solved by the present invention is how to characterize a speaker’s timbre without reference speech, or to generate the desired timbre of a speaker that does not exist.

[0005] To achieve the above objectives, the present invention provides a speaker generation method based on text representation-driven approach, characterized in that the method includes the following steps:

[0006] S101: Input the text description into the description encoder, wherein the text description is set to describe the speaker's vocal characteristics;

[0007] S103: The description encoder uses a discriminative method and / or a generative method to encode the text representation into a timbre feature vector, and maps the text representation and the speaker's timbre features into the same space;

[0008] S105: Input the timbre feature vector and text content into the multi-speaker TTS system, and the multi-speaker TTS system generates speech with the corresponding timbre;

[0009] S107: The timbre of the speech generated by the multi-speaker TTS system is consistent with the timbre encoded in the timbre feature vector, and the content of the speech is consistent with the content of the input text.

[0010] Furthermore, the discriminative method includes a language model RoBERTa, a fast adaptive module LoRA, and a mapping layer. The language model RoBERTa is a pre-trained language model capable of understanding the semantics of text. The fast adaptive module LoRA enables the language model RoBERTa to adapt quickly. The mapping layer includes multiple linear layers used to map the output of the language model RoBERTa to the same dimension as the speaker's representation.

[0011] Furthermore, the language model RoBERTa is pre-trained on a large amount of text data, and the weight parameters of the language model RoBERTa are fixed during the training process.

[0012] Furthermore, the language model RoBERTa incorporates the fast adaptive module LoRA after pre-training, enabling RoBERTa to adapt to the multi-speaker TTS system.

[0013] Furthermore, the fast adaptive module LoRA is configured as a lightweight, low-quality adaptive module.

[0014] Furthermore, the distinguishability method includes the following steps:

[0015] S1031: Receive the text description and concatenate it with the text description to obtain the CLS token;

[0016] S1032: Input the text representation and the CLS token into the pre-trained language model RoBERTa, and the language model RoBERTa outputs the output corresponding to the position of the CLS token;

[0017] S1033: The fast adaptive module LoRA receives the output at the position corresponding to the CLS token and maps the output at the position corresponding to the CLS token to the corresponding timbre feature vector.

[0018] Furthermore, the generative method uses a stream matching generative model to generate a distribution of the timbre feature vector based on the text representation, the distribution being obtained by transforming a Gaussian distribution.

[0019] Furthermore, the flow matching generative model uses a flow matching algorithm to obtain the target distribution by integrating the Gaussian distribution from time t=0 to time t=1 by integrating an ordinary differential equation.

[0020] Furthermore, the ordinary differential equation is:

[0021]

[0022] Where, φ t(x) represents the distribution of the ordinary differential equation as the integral is performed over time t, where x is a variable that changes with time. CLS The output for the CLS token position, Let θ be the timbre feature vector, θ be the parameters of the neural network, and v be the timbre feature vector. t For the distribution φ t (x) is the derivative of the equation with respect to time, φ0(x) is the initial Gaussian distribution, x0 is the value of the ordinary differential equation at time 0, and N(0,I) is a Gaussian distribution with variance of 1 and mean of 0 in all dimensions.

[0023] Furthermore, the discriminative method and the generative method can be used in a cascaded manner, and the output of the discriminative method can be used as the input of the generative method.

[0024] In a preferred embodiment of the present invention, compared with the prior art, the present invention has the following beneficial effects:

[0025] 1. The present invention proposes to control speaker generation through text, and combines generative and discriminative methods to make the generated representation more consistent with the text description, and the generated results more diverse. The two methods are complementary and further improve performance.

[0026] 2. This invention can use any text description to generate a representation of the corresponding speaker's timbre, which can be used to drive a multi-speaker system to generate speech with the corresponding timbre.

[0027] The following will further explain the concept, specific structure, and technical effects of the present invention in conjunction with the accompanying drawings, so as to fully understand the purpose, features, and effects of the present invention. Attached Figure Description

[0028] Figure 1 This is a schematic diagram illustrating the steps of a text-based speaker generation method according to an embodiment of the present invention.

[0029] Figure 2 This is a schematic diagram of a pre-trained TTS system in the existing technology;

[0030] Figure 3 This is a schematic diagram of the overall system structure according to an embodiment of the present invention;

[0031] Figure 4 This is a schematic diagram illustrating the generation of speaker representations from text descriptions according to an embodiment of the present invention;

[0032] Figure 5 This is a schematic diagram illustrating the correlation between the generated speech and the text description in terms of timbre in an embodiment of the present invention;

[0033] Figure 6 This is a schematic diagram of the voice quality evaluation results according to an embodiment of the present invention. Detailed Implementation

[0034] The following description, with reference to the accompanying drawings, illustrates several preferred embodiments of the present invention to make its technical content clearer and easier to understand. The present invention can be embodied in many different forms, and the scope of protection of the present invention is not limited to the embodiments mentioned herein.

[0035] In the accompanying drawings, components with the same structure are indicated by the same numerical designation, and components with similar structures or functions are indicated by similar numerical designations. The dimensions and thicknesses of each component shown in the drawings are arbitrary, and the present invention does not limit the dimensions and thicknesses of each component. To make the illustrations clearer, the thickness of some components has been appropriately exaggerated in the drawings.

[0036] like Figure 1 As shown, to characterize a speaker's timbre in the absence of reference speech, or to generate the desired timbre of a speaker that does not exist, this embodiment of the invention proposes a text-based speaker generation method. This method controls speaker generation through text and specifically includes the following steps:

[0037] Step 1: Input the text description into the description encoder. The text description is set to describe the speaker's vocal characteristics.

[0038] In this embodiment, a descriptive encoder is used to replace the speaker encoder in existing multi-speaker TTS (Text-to-Speech, TTS) systems. Specifically, by inputting a description of a person's vocal characteristics, such as "a person with a hoarse voice," this descriptive encoder can directly encode this textual description into a vocal timbre vector. This allows the previously trained multi-speaker TTS system to be used directly. To enable the generation of speaker speech using text descriptions, [the following is a separate, unrelated step]. Figure 2 The speaker encoder in the middle is replaced with Figure 3 The description encoder (PromptEncoder) in the program can accept descriptions of a speaker's timbre and generate corresponding timbre-coded features.

[0039] Step 2: Describe the encoder's use of discriminative and / or generative methods to encode text representations into timbre feature vectors and map text representations and speaker timbre features into the same space.

[0040] The speaker generation method proposed in this invention combines a discriminative method and a generative method, which are complementary and further improve performance.

[0041] The discriminative method provided in this embodiment includes multiple modules such as the language model RoBERTa, the fast adaptive module LoRA, and the mapping layer, wherein:

[0042] RoBERTa is a pre-trained language model capable of understanding the semantics of text. It is pre-trained on a large amount of text data, with its weight parameters fixed during training. After pre-training, a fast adaptive module (LoRA) is added to enhance its adaptability to multi-speaker TTS systems.

[0043] LoRA (Loose Adaptive Module): Enables the language model RoBERTa to adapt quickly. LoRA is set as a lightweight, low-quality adaptation module.

[0044] The mapping layer consists of multiple linear layers used to map the output of the language model RoBERTa to the same dimension as the speaker's representation.

[0045] The distinguishing method includes the following steps:

[0046] 1) Receive the text description and concatenate it with the text description to obtain the CLS token;

[0047] 2) Input the text representation and CLS token into the pre-trained language model RoBERTa, and the language model RoBERTa outputs the output corresponding to the position of the CLS token;

[0048] 3) The fast adaptive module LoRA receives the output at the position corresponding to the CLS token and maps the output at the position corresponding to the CLS token to the corresponding timbre feature vector.

[0049] The generative method provided in this embodiment uses a stream matching generative model to generate a distribution of timbre feature vectors based on the text description. This distribution can be obtained by transforming a Gaussian distribution.

[0050] The flow matching generative model uses the flow matching algorithm to obtain the target distribution by integrating the Gaussian distribution from time t=0 to time t=1 by integrating an ordinary differential equation.

[0051] The ordinary differential equation is as follows:

[0052]

[0053] Where, φ t (x) represents the distribution of the variable x at time t when integrating the ordinary differential equation, φ t (x) is not always a Gaussian distribution; φ0(x) is the initial Gaussian distribution. CLSThe output for the CLS token position, Let θ be the timbre feature vector, θ be the parameters of the neural network, and v be the timbre feature vector. t For the distribution φ t The derivative of (x) with respect to time is also called the vector field of the ordinary differential equation. We determine this based on v. t The value is integrated over the ordinary differential equation; N(0,I) is a Gaussian distribution with variance of 1 and mean of 0 in all dimensions; x is a variable that changes with time, x0 is the value of the ordinary differential equation at time point 0. At time point 0, x follows a Gaussian distribution, and at time point 1, x follows a different distribution. This is the final desired result.

[0054] The discriminative method and generative method provided in this embodiment can be used in a cascaded manner, and the output of the discriminative method can be used as the input of the generative method.

[0055] Step 3: Input the timbre feature vector and text content into the multi-speaker TTS system, and the multi-speaker TTS system generates speech with the corresponding timbre.

[0056] Step 4: The timbre of the speech generated by the multi-speaker TTS system is consistent with the timbre encoded in the timbre feature vector, and the content of the speech is consistent with the content of the input text.

[0057] Compared with existing technologies, the speaker generation method based on text representation-driven approach provided in this invention has the following specific effects:

[0058] 1. To address the problem of representing a speaker's timbre without reference speech, or generating the desired timbre of a speaker that does not exist, this invention proposes a method for generating corresponding timbre representations using text descriptions. This involves mapping the text descriptions and corresponding speaker features to the same space using generative or discriminative methods. This allows for the generation of a speaker's timbre representation using any text description during use. This representation can then drive a multi-speaker system to generate speech with the corresponding timbre.

[0059] 2. How to ensure that the generated speaker representation conforms to the text description, and how to generate diverse representations for the same description? This invention combines discriminative and generative methods. The discriminative method makes the generated representation more consistent with the text description, while the generative method makes the generated results more diverse. By cascading the discriminative and generative methods, the generated speaker representation maintains the characteristics described in the text and also maintains the diversity of the generated representation in different sampling processes.

[0060] 3. Regarding the question of how to map text descriptions to timbre representation space, this embodiment of the invention uses a pre-trained language model and the low-quality adaptation module LoRA for adaptation. The pre-trained language model can map text content to vector space, but it may not be generalized to specific tasks if used directly. By using LoRA, it can be effectively adapted to specific tasks, and can better map text descriptions to timbre representation space.

[0061] The present invention will now be described in detail with reference to preferred embodiments.

[0062] like Figure 2 The image shows a prior art multi-speaker text-to-speech (TTS) system. This system has two inputs: a reference speech and text. The reference speech is processed by a pre-trained speaker encoder to obtain timbre encoding features. The multi-speaker TTS system then synthesizes speech with corresponding timbre based on the timbre encoding features or the text content. Figure 2 In a multi-user TTS system, the speaker encoder receives speech input and encodes the timbre information from the speech input into a vector e. This multi-user TTS system accepts the speaker encoder e and text as input and synthesizes new speech. The timbre of this synthesized speech is consistent with the timbre encoded in e, and the content of the synthesized speech is consistent with the input text.

[0063] This invention proposes a method for generating corresponding timbre representations using text descriptions, addressing the problem of representing a speaker's timbre when there is no reference speech, or generating the desired timbre of a speaker that does not exist. Figure 3 As shown, a description encoder is used to replace the speaker encoder. In this embodiment of the invention, by inputting a description of a person's vocal characteristics, such as "a person with a hoarse voice," the description encoder can directly encode this textual description into a vocal timbre vector e. This allows the previously trained multi-person TTS system to be used directly. To enable the generation of speaker speech using textual descriptions, [the following is omitted as it's not part of the main text]. Figure 2 The speaker encoder in the middle has been replaced with Figure 3 The description encoder in the text takes a description of the speaker's timbre and generates corresponding timbre-coded features.

[0064] like Figure 3 As shown, the discriminative method provided in this embodiment of the invention can identify specific timbre encoding features based on text input characteristics. This method includes three modules:

[0065] (1) Pre-trained language model RoBERTa: This is a model pre-trained on a large amount of text data that can understand the meaning of text;

[0066] (2) Fast Adaptive Module LoRA: Although the pre-trained model RoBERTa can understand the semantics of the text, since the input text is mainly a description of the speaker's characteristics, these texts may not match the training text of RoBERTa to some extent. Therefore, in the discriminative method, LoRA is used to quickly adapt the system's capabilities.

[0067] (3) Projection Layers: These are a series of linear layers used to map the RoBERTa output of the pre-trained model to the same dimension as the speaker representation.

[0068] In this discriminative approach, the model accepts text descriptions as input and concatenates a [CLS] token into the text input. The text and [CLS] are then fed into a pre-trained language model, RoBERTa. During training, the weights of this language model are fixed, and a lightweight Low Rank Adaptation (LoRA) module is added to better adapt the pre-trained model to the task. Finally, the output corresponding to the [CLS] token is extracted and mapped to the corresponding timbre-encoded feature e. Figure 4 The discriminative method in (a) outputs fixed timbre encoding features for a given text input. This is somewhat inconsistent with practical applications. For example, a description of a person's timbre as "a man with a deep voice" could apply to many people. Therefore, a different approach was designed... Figure 4 (b) shows a generative model that generates a distribution of timbre-encoded features based on the text description, rather than a single timbre-encoded feature. To address this, a flow-matching generative model is proposed to transform a Gaussian distribution into the desired distribution. In the flow-matching algorithm, the Gaussian distribution is integrated from time t=0 to time t=1 using an ordinary differential equation to obtain the target distribution. This ordinary differential equation can be expressed as:

[0069]

[0070] In this ordinary differential equation, we can see that the model input, in addition to the distribution x, also includes o. CLS or This o CLS Actually it is Figure 4 Output of the RoBERTa[CLS] token position in (a); This is the output of the entire discriminative model. If using... In other words, the output of the entire discriminative model is used as the input of the generative model, which is then considered as a cascade of two models.

[0071] To achieve this functionality, a flow matching algorithm is used to map the output corresponding to the [CLS] token, or the output corresponding to the discriminative method, to the corresponding distribution. It is noted here that if... Figure 4 (b) Accepts the output of the discriminative method, which will be a cascade of discriminative and generative methods.

[0072] The system combining generative and discriminative methods proposed in this invention achieves a balance between maintaining text description and generation quality, yielding good results in both dimensions. Figure 5 and Figure 6 The effect evaluation is shown.

[0073] exist Figure 5 The correlation between speech generated by discriminative and flow-matching methods and the described timbre was compared, with higher correlation values ​​indicating higher correlation. Regarding... Figure 5 As a result, a large audience was gathered to score the generated speech on various dimensions, such as... Figure 5 The scores for age and pitch were limited to a range of 1-5. The correlation between the generated speech score and the reference speech was then calculated; a higher correlation indicates that the system's generated speech better matches the text's description of the speaker's characteristics.

[0074] exist Figure 5 The system includes the following four systems:

[0075] (1) Discriminative (w / o LoRA): This is a system that uses a discriminative approach, but without the LoRA adaptive module;

[0076] (2) Discriminative (w / LoRA): This is also a system that uses the discriminative method, but with the addition of the LoRA adaptive module;

[0077] (3) Flow-Matching (w / LoRA): This is a generative method system with the LoRA module added;

[0078] (4) Discriminative + Flow-Matching: This is a cascade of discriminative and generative methods. Through analysis, it was found that discriminative methods perform better than generative methods in this respect, and the combination of the two (Discriminative + Flow-Matching) also works very well.

[0079] like Figure 6 As shown, the speech quality corresponding to discriminative and generative methods is illustrated. In speech quality evaluation, a smaller FAD and a larger MOS are better. It can be observed that the generative method produces better speech quality, while the combination of discriminative and generative methods also achieves good speech quality. Figure 5 and Figure 6 The conclusion shows that combining generative and discriminative methods not only better preserves the timbre of the description but also generates higher-quality speech. Furthermore, since this invention was trained on only a small dataset, larger-scale data collection and training are needed for real-world application.

[0080] The method for generating corresponding timbre representations using text descriptions provided in this invention has the following advantages:

[0081] 1. Technical advantages: The method for generating speakers by controlling them through text proposed in this embodiment of the invention combines generative and discriminative methods, which are complementary and further improve performance.

[0082] 2. Performance metrics: The system combining generative and discriminative methods proposed in this embodiment of the invention can achieve a balance between maintaining text description and generation quality, and achieve good results in both dimensions.

[0083] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. A text utterance expression driven based speaker generation method, characterized in that, The method comprises the following steps: S101: input a text description into a description encoder, the text description being configured to describe the timbre characteristics of a speaker; S103: the description encoder uses a discriminative method and / or a generative method to encode the text description into a timbre feature vector, and maps the text description and the timbre characteristics of the speaker to the same space; S105: input the timbre feature vector and the text content into a multi-speaker TTS system, the multi-speaker TTS system generating voice corresponding to the timbre; S107: the timbre of the voice generated by the multi-speaker TTS system is consistent with the timbre encoded in the timbre feature vector, and the content of the voice is consistent with the input text content; wherein, The discriminative method comprises a language model RoBERTa, a fast adaptation module LoRA, and a mapping layer, the language model RoBERTa being a pre-trained language model capable of understanding the semantics of a text, the fast adaptation module LoRA enabling the language model RoBERTa to quickly adapt, and the mapping layer comprising multiple linear layers for mapping the output of the language model RoBERTa to the same dimension as the speaker representation; The discriminative method comprises the following steps: S1031: receive a text description and splice a CLS token from the text description; S1032: input the text description and the CLS token into the pre-trained language model RoBERTa, the language model RoBERTa outputting the output at the position corresponding to the CLS token; S1033: the fast adaptation module LoRA receives the output at the position corresponding to the CLS token and maps the output at the position corresponding to the CLS token to the corresponding timbre feature vector; The generative method uses a flow matching generative model to generate a distribution about the timbre feature vector according to the text description, the distribution being obtained by converting a Gaussian distribution.

2. The method of claim 1, wherein, The language model RoBERTa is pre-trained on a large amount of text data, and the weight parameters of the language model RoBERTa are fixed during training.

3. The method of claim 2, wherein, The language model RoBERTa is added with the fast adaptation module LoRA after pre-training, so that the language model RoBERTa adapts to the multi-speaker TTS system.

4. The method of claim 3, wherein, The fast adaptation module LoRA is configured as a lightweight low-quality adaptation module.

5. The method of claim 4, wherein, The flow matching generative model uses a flow matching algorithm to obtain a target distribution by integrating a common differential equation from time t=0 to time t=1.

6. The method of claim 5, wherein, The common differential equation is: in, For ordinary differential equations in time t When accumulating points x The distribution corresponding to the variable For variables that change over time, for CLS Output of token position, For timbre feature vectors, For the parameters of the neural network, For distribution The derivative with respect to time, Given an initial Gaussian distribution, This represents the value of the ordinary differential equation at time point 0. It is a Gaussian distribution with variance of 1 in all dimensions and mean of 0.

7. The method of claim 6, wherein, The discriminative method and the generative method are used in cascade, and the output of the discriminative method is used as the input of the generative method.

Images