Multi-dialect TTS generation method based on large model

Abstract

The invention relates to a multi-dialect TTS generation method based on a large model, and the method comprises the following steps: S1, collecting the data of each dialect and mandarin, and forming a data set; s2, gradually training from coarse to fine by using the data set, so that the large model has an accurate TTS generation capability; s3, training the large model by adopting a data set with dialect text and voice pairs to generate a multi-dialect TTS large model; s4, optimizing the multi-dialect TTS large model by adopting a transform network structure; and S5, enhancing the expression ability of the multi-dialect TTS large model by adopting hierarchical reinforcement learning. The dialect voice of a specific region can be generated, and the user experience is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of TTS large models, and in particular to a multi-dialect TTS generation method based on a large model. Background Art

[0002] The only existing technology is a large Mandarin TTS (Text To Speech) model that can perform zero-shot speech synthesis. However, there is little research on speech synthesis for China's vast dialects. Training a large model for speech synthesis in various Chinese dialects is an extremely challenging task.

[0003] Therefore, it is necessary to provide a multi-dialect TTS generation method based on a large model to generate dialect speech in a specific region and improve user experience. Summary of the invention

[0004] The purpose of the present invention is to provide a multi-dialect TTS generation method based on a large model to generate dialect speech in a specific region and improve user experience.

[0005] In order to solve the problems existing in the prior art, the present invention provides a multi-dialect TTS generation method based on a large model, comprising the following steps:

[0006] S1: Collect data from various dialects and Mandarin to form a data set;

[0007] S2: Use the data set to conduct step-by-step training from coarse to fine, so that the large model has accurate TTS generation capabilities;

[0008] S3: Use a dataset with dialect text and speech pairs to train a large model to generate a multi-dialect TTS large model;

[0009] S4: Using transformer network structure to optimize multi-dialect TTS large model;

[0010] S5: Use hierarchical reinforcement learning to enhance the expressive power of multi-dialect TTS large models.

[0011] Optionally, in the multi-dialect TTS generation method based on a large model, the data of each dialect is collected as follows:

[0012] The tokens of various dialects are standardized and some overlap is made between Mandarin and the tokens of various dialects, so that Mandarin can provide basic pronunciation capabilities.

[0013] Optionally, in the multi-dialect TTS generation method based on the big model, in the big model, token refers to the smallest unit into which text data is divided before or during processing by the big model.

[0014] Optionally, in the multi-dialect TTS generation method based on a large model, a step-by-step training from coarse to fine is performed in the following manner:

[0015] Use data sets for training to give large models basic TTS capabilities;

[0016] The use of precisely labeled data for training enables the large model to have accurate TTS generation capabilities.

[0017] Optionally, in the multi-dialect TTS generation method based on a large model, the data is marked in advance to form data with precise markings.

[0018] Optionally, in the multi-dialect TTS generation method based on a large model, a transformer network structure is used to optimize the multi-dialect TTS large model in the following manner:

[0019] S41: For dialects that are similar in pronunciation to Mandarin, the Mandarin scheme should be used, but the dialect label should be added;

[0020] S42: For dialects that are completely different from Mandarin pronunciation, mark them as dialects that need to be learned separately;

[0021] S43: A cross-attention-based fusion mechanism is used to inject dialect tokens into different layers of the multi-dialect TTS large model.

[0022] Optionally, in the multi-dialect TTS generation method based on a large model, indicators and a separately trained small dialect TTS model are used together as a reward model to enhance the expressiveness of the multi-dialect TTS large model, wherein the indicators include MOS and WER.

[0023] Compared with the prior art, the present invention has the following advantages:

[0024] (1) The multi-dialect TTS large model of the present invention can not only obtain Mandarin speech content, but also specify a specific dialect for synthesis;

[0025] (2) The present invention also uses a transformer network structure to optimize the multi-dialect TTS large model to make the generated dialect more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 The overall structure diagram of the large model provided by the embodiment of the present invention;

[0027] Figure 2 This is a diagram of the dialect MOE structure provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0028] The specific implementation of the present invention will be described in more detail below in conjunction with the schematic diagram. The advantages and features of the present invention will become clearer based on the following description. It should be noted that the drawings are all in a very simplified form and are not in exact proportions, and are only used to facilitate and clearly assist in explaining the purpose of the embodiments of the present invention.

[0029] Hereinafter, if the method described herein includes a series of steps, the order in which the steps are presented herein is not necessarily the only order in which the steps may be performed, and some of the steps described may be omitted and / or some other steps not described herein may be added to the method.

[0030] The only existing technology is a large Mandarin TTS (Text To Speech) model that can perform zero-shot speech synthesis. However, there is little research on speech synthesis for China's vast dialects. Training a large model for speech synthesis in various Chinese dialects is an extremely challenging task.

[0031] In order to solve the problems existing in the prior art, the present invention provides a multi-dialect TTS generation method based on a large model, such as Figure 1 and 2 As shown, the method comprises the following steps:

[0032] S1: Collecting data of various dialects and Mandarin to form a data set, so that the present invention can achieve high-quality dialect speech synthesis under limited data conditions;

[0033] Specifically, the method of collecting data from various dialects is as follows: standardize and unify the tokens of various dialects, and make Mandarin and the tokens of various dialects partially overlap, so as to use Mandarin to provide basic pronunciation capabilities. Among them, token refers to the smallest unit into which text data is divided before or during processing by the large model.

[0034] S2: Mainly based on the refined token-wise alignment of large-scale multimodal pre-training, a multi-stage, multimodal (text and audio) pre-training learning framework is proposed for text-speech token alignment.

[0035] Specifically, the data set is used to conduct step-by-step training from coarse to fine, so that the large model has accurate TTS generation capabilities; the step-by-step training from coarse to fine is conducted in the following manner: the data set is used for training, so that the large model has basic TTS capabilities; the data with precise labels is used for training, so that the large model has accurate TTS generation capabilities. Preferably, the data is labeled in advance to form data with precise labels.

[0036] S3: A large model is trained using a data set containing dialect text and speech pairs to generate a multi-dialect TTS large model; the large model of the present invention learns the text-speech alignment relationship at a token fine-grained level to promote refined alignment between the two modalities.

[0037] S4: Use the transformer network structure to optimize the multi-dialect TTS large model; the method is as follows:

[0038] S41: For dialects that are similar in pronunciation to Mandarin, the Mandarin scheme should be used, but the dialect label should be added;

[0039] S42: For dialects that are completely different from Mandarin pronunciation, mark them as dialects that need to be learned separately;

[0040] S43: A cross-attention-based fusion mechanism is used to inject dialect tokens into different layers of the multi-dialect TTS large model.

[0041] S5: Adopt hierarchical reinforcement learning (RL) to enhance the expressive power of the multi-dialect TTS large model. Specifically, the indicators and the separately trained small dialect TTS model are used as reward models to enhance the expressive power of the multi-dialect TTS large model, wherein the indicators include MOS and WER. The present invention establishes an RL hierarchical training strategy in order to allow the basic training strategy to support the exploration of high-quality dialect speech expression, while allowing the advanced training strategy to strengthen the speech characteristics of different dialects on this basis.

[0042] In summary, compared with the prior art, the present invention has the following advantages:

[0043] (1) The multi-dialect TTS large model of the present invention can not only obtain Mandarin speech content, but also specify a specific dialect for synthesis;

[0044] (2) The present invention also uses a transformer network structure to optimize the multi-dialect TTS large model to make the generated dialect more accurate.

[0045] The above is only a preferred embodiment of the present invention and does not limit the present invention in any way. Any technician in the relevant technical field, without departing from the scope of the technical solution of the present invention, makes any form of equivalent replacement or modification to the technical solution and technical content disclosed in the present invention, which does not depart from the content of the technical solution of the present invention and still falls within the protection scope of the present invention.

Claims

1. A multi-dialect TTS generation method based on a large model, characterized in that: The following steps are involved: S1: Collect data from various dialects and Mandarin to form a data set; S2: Use the data set to conduct step-by-step training from coarse to fine, so that the large model has accurate TTS generation capabilities; S3: Use a dataset with dialect text and speech pairs to train a large model to generate a multi-dialect TTS large model; S4: Using transformer network structure to optimize multi-dialect TTS large model; S5: Use hierarchical reinforcement learning to enhance the expressive power of multi-dialect TTS large models.

2. The multi-dialect TTS generation method based on a large model as claimed in claim 1, characterized in that: The data for each dialect is collected as follows: The tokens of various dialects are standardized and some overlap is made between Mandarin and the tokens of various dialects, so that Mandarin can provide basic pronunciation capabilities.

3. The multi-dialect TTS generation method based on a large model as claimed in claim 2, characterized in that: In the big model, token refers to the smallest unit into which text data is divided before or during processing by the big model.

4. The multi-dialect TTS generation method based on a large model as claimed in claim 1, characterized in that: Conduct training from coarse to fine step by step as follows: Use data sets for training to give large models basic TTS capabilities; The use of precisely labeled data for training enables the large model to have accurate TTS generation capabilities.

5. The multi-dialect TTS generation method based on a large model as claimed in claim 4, characterized in that: Mark the data in advance to form data with precise labels.

6. The multi-dialect TTS generation method based on a large model as claimed in claim 1, characterized in that: The transformer network structure is used to optimize the multi-dialect TTS large model as follows: S41: For dialects that are similar in pronunciation to Mandarin, the Mandarin scheme should be used, but the dialect label should be added; S42: For dialects that are completely different from Mandarin pronunciation, mark them as dialects that need to be learned separately; S43: A cross-attention-based fusion mechanism is used to inject dialect tokens into different layers of the multi-dialect TTS large model.

7. The multi-dialect TTS generation method based on a large model as claimed in claim 1, characterized in that: Indicators and separately trained small dialect TTS models are used together as reward models to enhance the expressiveness of the multi-dialect TTS large model, where the indicators include MOS and WER.

Images