Speech synthesis model training method, speech synthesis method, device and equipment
By extending multilingual features in the language embedding layer of the pre-trained pronunciation synthesis model and further training using the training samples of the second language, the problems of low training efficiency and poor effect of multilingual pronunciation synthesis model in the prior art are solved, and fast and efficient multilingual pronunciation synthesis model training is achieved.
Patent Information
- Application Number
- CN202510255721.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-03-05
- Publication Date
- 2025-06-06
AI Technical Summary
When training multilingual pronunciation synthesis models, existing pronunciation synthesis systems face the problems of data acquisition and the problem of inadequate model structure to new languages, resulting in low training efficiency and poor results.
By obtaining the pre-trained first language base model and extending the language dictionary in its language embedding layer, it contains language features of multiple languages. Then, the model is further trained based on the training samples of the second language, and the loss calculation is performed using the predicted audio features and the target audio features, and the model parameters are optimized to obtain the trained speech synthesis model.
It realizes the rapid and efficient training of multilingual pronunciation synthesis models without modifying the model structure, saving training time and computing resources, and avoiding the difficulty of training models from scratch.
Smart Images
Figure CN120108372A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of speech synthesis technology, and in particular to a training method for a speech synthesis model, a speech synthesis method, a device and equipment. Background Art
[0002] With the development of speech synthesis technology, the ability of speech synthesis is no longer limited to a single language, and the ability of multi-language synthesis is becoming more and more concerned and challenging.
[0003] Although current speech synthesis systems can produce relatively high-quality synthesized speech, such speech synthesis systems rely on a large amount of training data. For languages with fewer speech resources, it is very difficult to train speech synthesis models for these languages from scratch. Summary of the invention
[0004] In order to solve the above technical problems or at least partially solve the above technical problems, at least one embodiment of the present disclosure provides a training method for a speech synthesis model, a speech synthesis method, an apparatus and a device.
[0005] In a first aspect, the present disclosure provides a method for training a speech synthesis model, comprising:
[0006] Obtain a pre-trained first language base model, wherein the language embedding layer of the first language base model includes a language dictionary, wherein the language dictionary includes language features corresponding to a plurality of languages, respectively, and the first language is at least one of Chinese and English;
[0007] Acquire a training sample in a second language, wherein the training sample is marked with a language identifier of the second language and includes a text sample and a corresponding audio sample in the second language, wherein the second language includes another language different from the first language;
[0008] Continue training based on the first language base model to obtain predicted audio features corresponding to the training sample, wherein the language embedding layer of the first language base model determines a target language feature corresponding to the language identifier based on the language dictionary, and the target language feature is used to determine the predicted audio feature;
[0009] Performing loss calculation based on the predicted audio feature and the target audio feature corresponding to the audio sample to obtain a current loss value;
[0010] Taking minimizing the current loss value as an optimization goal, parameter optimization is continued based on the parameters of the first language base model to obtain a trained speech synthesis model.
[0011] In a second aspect, the present disclosure provides a speech synthesis method, comprising:
[0012] Obtain the text to be processed and reference audio information;
[0013] Inputting the text to be processed and the reference audio information into a speech synthesis model for speech synthesis prediction to obtain audio features corresponding to the text to be processed, wherein the speech synthesis model is trained by the speech synthesis model training method as described in the first aspect;
[0014] The audio features corresponding to the text to be processed are input into a vocoder for processing to obtain the audio corresponding to the text to be processed.
[0015] In a third aspect, the present disclosure provides a training device for a speech synthesis model, comprising:
[0016] A first acquisition module is used to acquire a pre-trained first language base model, wherein the language embedding layer of the first language base model includes a language dictionary, wherein the language dictionary includes language features corresponding to a plurality of languages, respectively, and the first language is at least one of Chinese and English;
[0017] A second acquisition module is used to acquire training samples in a second language, wherein the training samples are marked with a language identifier of the language and include a text sample and a corresponding audio sample in the second language, wherein the second language includes other languages different from the first language;
[0018] A training module is used to continue training based on the first language base model to obtain predicted audio features corresponding to the training samples, wherein the language embedding layer of the first language base model determines the target language features corresponding to the language identifier based on the language dictionary, and the target language features are used to determine the predicted audio features; based on the predicted audio features and the target audio features corresponding to the audio samples, loss calculation is performed to obtain a current loss value; with minimizing the current loss value as an optimization goal, parameter optimization is continued based on the parameters of the first language base model to obtain a trained speech synthesis model.
[0019] In a fourth aspect, the present disclosure provides a speech synthesis device, comprising:
[0020] A text acquisition module is used to acquire the text to be processed and reference audio information;
[0021] A prediction module, used for inputting the text to be processed and the reference audio information into a speech synthesis model for speech synthesis prediction, and obtaining audio features corresponding to the text to be processed, wherein the speech synthesis model is trained by the training method of the speech synthesis model as described in the first aspect;
[0022] The speech synthesis module is used to input the audio features corresponding to the text to be processed into the vocoder for processing to obtain the audio corresponding to the text to be processed.
[0023] In a fifth aspect, the present disclosure provides an electronic device, including: a processor and a memory;
[0024] The processor is used to execute the training method of the speech synthesis model as described in the first aspect, or to execute the speech synthesis method as described in the second aspect by calling the program or instruction stored in the memory.
[0025] In a sixth aspect, the present disclosure provides a computer-readable storage medium, which stores a program or instruction, and the program or instruction enables a computer to execute the training method of the speech synthesis model as described in the first aspect, or execute the speech synthesis method as described in the second aspect.
[0026] In a seventh aspect, the present disclosure provides a computer program product, which is used to execute the training method of the speech synthesis model as described in the first aspect, or to execute the speech synthesis method as described in the second aspect.
[0027] Compared with the prior art, the technical solution provided by the embodiments of the present disclosure has at least the following advantages:
[0028] In an embodiment of the present disclosure, a pre-trained first language base model is obtained, wherein the language embedding layer of the first language base model includes a language dictionary, wherein the language dictionary includes language features corresponding to a plurality of languages, and the first language is at least one of Chinese and English; a training sample of a second language is obtained, wherein the training sample is annotated with a language identifier of the second language and includes a text sample and a corresponding audio sample of the second language, and the second language includes other languages different from the first language; training is continued based on the first language base model to obtain predicted audio features corresponding to the training samples, wherein the language embedding layer of the first language base model determines a target language feature corresponding to the language identifier based on the language dictionary, and the target language feature is used to determine the predicted audio feature; loss calculation is performed based on the predicted audio feature and the target audio feature corresponding to the audio sample to obtain a current loss value; and parameter optimization is continued based on the parameters of the first language base model with minimizing the current loss value as an optimization target to obtain a trained speech synthesis model. By adopting the above technical scheme, by expanding the language dictionary including language features corresponding to multiple languages in the language embedding layer of the pre-trained first language base model, it is achieved that positions for multiple languages are reserved in the language embedding layer, which provides conditions for the subsequent addition of new languages. By adding the language features of the languages, the model can better fit the data of the corresponding languages during training, so that when training speech synthesis models of other languages based on the first language base model, there is no need to modify the model structure, and the first language base model can be directly used for training to obtain a usable speech synthesis model, thereby achieving fast and efficient model training; and, by training speech synthesis models of other languages on the basis of the pre-trained first language base model, the speech synthesis model of the language can be fine-tuned using fewer training samples of other languages, thereby saving training time and computing power resources, and avoiding the problem of long time and high computing power requirements for training models from scratch. BRIEF DESCRIPTION OF THE DRAWINGS
[0029] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.
[0030] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.
[0031] Figure 1 A flowchart of a method for training a speech synthesis model provided by an exemplary embodiment of the present disclosure;
[0032] Figure 2A flowchart of a method for training a speech synthesis model provided by another exemplary embodiment of the present disclosure;
[0033] Figure 3 is a structural schematic diagram of a speech synthesis model of an exemplary embodiment of the present disclosure;
[0034] Figure 4 A flowchart of a speech synthesis method provided by an exemplary embodiment of the present disclosure;
[0035] Figure 5 A schematic diagram of the structure of a training device for a speech synthesis model provided in one embodiment of the present disclosure;
[0036] Figure 6 A schematic diagram of the structure of a speech synthesis device provided in one embodiment of the present disclosure. DETAILED DESCRIPTION
[0037] In order to more clearly understand the above-mentioned purposes, features and advantages of the present disclosure, the present disclosure is further described in detail below in conjunction with the accompanying drawings and embodiments. It is understandable that the described embodiments are part of the embodiments of the present disclosure, rather than all of the embodiments, and the specific embodiments described herein are only used to explain the present disclosure, rather than to limit the present disclosure. In the absence of conflict, the embodiments of the present disclosure and the features in the embodiments can be combined with each other. Based on the described embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in the field belong to the scope of protection of the present disclosure.
[0038] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.
[0039] With the development of text-to-speech (TTS) technology, the ability of speech synthesis is no longer limited to a single language, and the ability of multi-language synthesis has attracted more and more attention and become a challenge. Although the current TTS system can produce high-quality synthesized speech, such TTS models rely on a large amount of speech data for training. The production cost of this data is relatively high, and it is difficult to expand to all existing languages, especially for languages with less speech resources. Therefore, there are two problems in training multi-language speech synthesis models. On the one hand, due to data limitations, it is difficult to obtain multi-language data, which makes it difficult to train a multi-language base model from scratch. It requires a lot of computing power and data, resulting in poor results for the trained base model. On the other hand, even if there is a pre-trained base model, the structure of the base model is often not suitable for the expansion of speech synthesis capabilities for new languages, and the structure needs to be readjusted for training.
[0040] In response to the above problems, the present disclosure provides a training scheme for a speech synthesis model. This scheme does not have the ability to train a multilingual model from scratch, but is based on the first language base model and continues to use multilingual data for training to obtain a new model with multilingual capabilities. Among them, in the training model, the language embedding layer reserves positions for multiple languages through the language dictionary, expands the dimension of the language type elements in the language embedding layer, and only uses part of the positions when training Chinese and English data. The remaining positions can be used by adding other language data. This method does not need to modify the model structure, and directly uses the model trained in both Chinese and English. The reusable model continues to train multilingual data to achieve a usable state. Generally speaking, language modeling is divided into explicit modeling and implicit modeling. This solution adopts explicit modeling. The embedded representation (language embedding) of traditional explicit modeling uses several languages to represent several positions. If the language classification is not explicitly modeled, the language type identification is not required. The explicit modeling of this solution reserves the language features corresponding to all languages through the language dictionary. The explicit modeling of the language can assist the model in adaptively learning the pronunciation styles of different languages. When training the first language base model (first language speech synthesis model), this solution reserves different multilingual positions through the language dictionary when training the first language capability model in order to better expand business needs. This makes it convenient to add new languages later without modifying the network structure of the first language base model for training.
[0041] The disclosed solution can be seamlessly connected to any language without changing the original model design. It can dynamically increase the types of training languages in real time according to changes in business needs, realize the expansion capability of multi-language training, expand the model's capabilities to the language synthesis capabilities of any business needs, and quickly and efficiently train a model with multi-language effects. This solution can quickly achieve the transformation from the first language speech synthesis model to other languages (such as Chinese, English, Japanese, Korean, German, Italian, French, Polish, Spanish, Portuguese, Dutch, Malay, etc.) by expanding the multi-language data training of the already trained first language base model.
[0042] The specific implementation methods of the speech synthesis model training method, speech synthesis method, device and equipment disclosed in the present invention are explained in detail below with reference to the accompanying drawings.
[0043] Figure 1 A flow chart of a method for training a speech synthesis model provided for an exemplary embodiment of the present disclosure. The method can be executed by a training device for a speech synthesis model provided for an embodiment of the present disclosure. The training device for a speech synthesis model can be implemented using software and / or hardware and can be integrated in an electronic device.
[0044] like Figure 1As shown, the training method of the speech synthesis model may include the following steps:
[0045] Step 101, obtaining a pre-trained first language base model, wherein the language embedding layer of the first language base model includes a language dictionary, the language dictionary includes language features corresponding to a plurality of languages, and the first language is at least one of Chinese and English.
[0046] Among them, the first language base model is pre-trained using speech data of the first language (including at least one of Chinese and English). Since Chinese and English are currently the most commonly used languages, it is easy to collect a large number of speech resources and text resources. Therefore, the first language base model pre-trained using Chinese and / or English as the first language can ensure the effect of the first language base model. In addition, when training the first language base model, the language embedding layer is expanded. The language embedding layer expansion includes a language dictionary of language features corresponding to multiple languages, so as to dynamically increase or change the training language types according to changes in business needs in real time, and realize the expansion capability of multi-language training.
[0047] In this embodiment, the dimensions of the language embedding layer are no longer a few dimensions for a few languages, but a dimension that covers enough languages and can be activated when needed. Therefore, only the language encoding needs to be readjusted, that is, there can be a unified encoder that can handle multilingual data. No matter how many types of language data there are, they can be unified in this way, and the structure of the first language base model and the model parameters of the trained first language base model can be used to continue training speech synthesis models for other languages, avoiding the problem of long time and high computing power requirements for optimizing the model from scratch.
[0048] For example, the dimension of the language embedding layer can be expressed as nn.Embedding(num_embeddings=50,embedding_dim=128), where num_embeddings represents the size of the language dictionary, that is, the number of different languages; embedding_dim represents the dimension of the vector representation (i.e., language features) of each language, so the dimension of the language embedding layer is directly set to 50. The language dictionary includes language features corresponding to 50 languages, while the number of languages currently required by actual business is far less than 50. The language embedding layer can map the input language identifier into a dense vector representation (i.e., language features) to participate in the subsequent network speech synthesis prediction, and assist the model in adaptively learning the pronunciation styles of different languages.
[0049] Step 102: obtaining training samples in a second language. The training samples are marked with a language identifier of the second language and include a text sample and a corresponding audio sample in the second language. The second language includes other languages different from the first language.
[0050] The second language can be determined according to actual business needs, including other languages different from the first language, such as French, Arabic, etc. It can include only at least one other language, or it can include at least one other language and the first language, so that the speech synthesis model finally trained has the synthesis capabilities of multiple languages of the first language and other languages. When the second language includes Chinese, English and other languages, the speech synthesis model finally trained using the training samples of these languages can maintain the language capabilities of the previous first language base model while adding the language capabilities of the new language.
[0051] In this embodiment, the acquired training samples include audio samples and corresponding text samples, as well as corresponding language identifiers. A training sample is composed of a text sample of a marked language and a corresponding audio sample, and the audio sample is the audio of the text sample being read aloud.
[0052] Step 103, continue training based on the first language base model to obtain predicted audio features corresponding to the training samples, wherein the language embedding layer of the first language base model determines the target language features corresponding to the language identifier based on the language dictionary, and the target language features are used to determine the predicted audio features.
[0053] In this embodiment, the training sample is input into the first language base model, and the training is continued based on the first language base model to obtain the predicted audio features corresponding to the training sample, wherein the text sample and the audio sample of the training sample are respectively encoded based on the first language base model to obtain the corresponding features, and then combined with the language embedding layer based on the language identifier of the language marked by the training sample and the target language features corresponding to the language identifier determined by the language dictionary, the audio features corresponding to the training sample are calculated (for the convenience of description and distinction, referred to as predicted audio features), and the predicted audio features are output.
[0054] Step 104: Perform loss calculation based on the predicted audio features and the target audio features corresponding to the audio sample to obtain a current loss value.
[0055] In this embodiment, during the model training process, the predicted audio features output by the first language base model are obtained, and the loss calculation is performed based on the obtained predicted audio features and the target audio features corresponding to the audio sample to obtain the current loss value. It should be noted that in this embodiment, when performing the loss calculation, the loss value can be calculated using the currently commonly used loss function, and this disclosure does not limit this.
[0056] Step 105 , taking minimizing the current loss value as the optimization goal, continue to optimize the parameters based on the parameters of the first language base model to obtain a trained speech synthesis model.
[0057] In this embodiment, after calculating the current loss value, the current loss value can be minimized as the optimization goal, and parameter optimization is continued based on the parameters of the first language base model. Gradient feedback is performed according to the current loss value, and the model parameters of the first language base model are adjusted to achieve the optimization purpose. When the number of iterative training is sufficient (for example, the number of iterative training reaches the iteration threshold), the training is completed and a trained speech synthesis model is obtained.
[0058] The training method of the speech synthesis model of the embodiment of the present disclosure obtains a pre-trained first language base model, wherein the language embedding layer of the first language base model includes a language dictionary, the language dictionary includes language features corresponding to multiple languages, and the first language is at least one of Chinese and English; obtains a training sample of a second language, wherein a training sample is marked with a language identifier of the second language and includes a text sample and a corresponding audio sample of the second language, and the second language includes other languages different from the first language; continues training based on the first language base model to obtain predicted audio features corresponding to the training samples, wherein the language embedding layer of the first language base model determines a target language feature corresponding to the language identifier based on the language dictionary, and the target language feature is used to determine the predicted audio feature; performs loss calculation based on the predicted audio feature and the target audio feature corresponding to the audio sample to obtain a current loss value; continues to optimize parameters based on the parameters of the first language base model with minimizing the current loss value as the optimization goal to obtain a trained speech synthesis model. By adopting the above technical scheme, by expanding the language dictionary including language features corresponding to multiple languages in the language embedding layer of the pre-trained first language base model, it is achieved that positions for multiple languages are reserved in the language embedding layer, which provides conditions for the subsequent addition of new languages. By adding the language features of the languages, the model can better fit the data of the corresponding languages during training, so that when training speech synthesis models of other languages based on the first language base model, there is no need to modify the model structure, and the first language base model can be directly used for training to obtain a usable speech synthesis model, thereby achieving fast and efficient model training; and, by training speech synthesis models of other languages on the basis of the pre-trained first language base model, the speech synthesis model of the language can be fine-tuned using fewer training samples of other languages, thereby saving training time and computing power resources, and avoiding the problem of long time and high computing power requirements for training models from scratch.
[0059] In an optional implementation of the present disclosure, the first language base model further includes a text encoder, an audio encoder and an acoustic model, wherein the text encoder is used to encode the input text sample, the audio encoder is used to encode the input audio sample, and the acoustic model is used to perform speech synthesis prediction based on the input, such as Figure 2 As shown, based on the above embodiment, step 103 may include the following sub-steps:
[0060] Step 201 : input the language identifier into the language embedding layer to obtain the target language feature corresponding to the language identifier determined by the language embedding layer based on the language dictionary.
[0061] In this embodiment, the training samples are input into the first language base model, and the language identifier marked by the training samples is used as the input of the language embedding layer. After receiving the input language identifier, the language embedding layer determines the language feature corresponding to the received language identifier according to the language dictionary (for the convenience of description and distinction, it is called the target language feature), and the target language feature output by the language embedding layer is input into the acoustic model.
[0062] Step 202: Input the text sample into a text encoder for encoding to obtain target text features corresponding to the text sample, wherein the text encoder is pre-trained based on multilingual text data.
[0063] In this embodiment, the training samples are input into the first language base model, and the text samples in the training samples are input into the text encoder for encoding to obtain text features corresponding to the text samples (referred to as target text features for ease of description and distinction), and the target text features output by the text encoder are input into the acoustic model.
[0064] Among them, the text encoder is pre-trained based on multilingual text data. Multilingual text data refers to text data corresponding to multiple languages. Multilingual can cover as many languages as possible. For example, the language dictionary includes language features of 50 languages (including Chinese, English, French, Arabic, etc.). Accordingly, the text encoder is trained using the text data of these 50 languages.
[0065] In an optional embodiment of the present disclosure, when training a text encoder, text data of multiple languages (for ease of description and distinction, referred to as first text data) can be obtained, and an initial text encoder can be constructed. For example, the initial text encoder can be a tokenizer; and then the initial text encoder is trained using the first text data. After the training is completed, a text encoder is obtained. Among them, the first text data covers as many languages as possible, including not only text data of the first language, but also text data of other languages. When training the initial text encoder, the input first text data is first converted into a smaller unit (such as the International Phonetic Alphabet IPA) format for representation and then trained. The purpose of the training is to learn the relationship between each IPA combination, and then use it to convert the first text data into a digital form of encoding. By training a text encoder using the first text data of multiple languages, a text encoder that encodes texts of all languages can be obtained, and then the text encoder is used to encode the text samples in the training samples, so that multilingual data can be quickly trained based on the first language base model, and the usable state of multilingual synthesis capability can be achieved.
[0066] In addition, in an optional embodiment of the present disclosure, a unified phonetic dictionary can also be constructed for data of multiple languages, so that users can check the output results of the text encoder according to the phonetic dictionary to see whether the output results are appropriate, whether there are errors, etc. When constructing a phonetic dictionary, the modeling unit of the International Phonetic Alphabet can be adopted, and the words and texts in the text data of multiple languages can be simply phonemed through the open source code library Phonemizer, and then the phonetic symbols obtained by phonemicizing all language data are counted, and then de-duplication and merging are performed to construct a unified multilingual phonetic dictionary. Among them, Phonemizer is a powerful open source Python library for converting words and texts into International Phonetic Alphabet (IPA) or other speech representations. This library supports the phonemicization of texts in multiple languages, that is, the conversion of text to phonemes, which is an important step in the text-to-speech (TTS) task. The present disclosure uses the backend engine of Espeak, which supports more than 100 languages. By converting text into International Phonetic Alphabet, it can retain information at the word, sentence, and even syllable level, which is convenient for further speech processing. At the same time, users can customize phoneme mapping files to adapt to specific language environments. The present invention constructs a unified multilingual phonetic dictionary by counting the phonetic symbols of its own multilingual text data, which provides data support for accuracy checking in the process of training a text encoder.
[0067] Step 203: Input the audio sample into an audio encoder for encoding to obtain target audio features corresponding to the audio sample.
[0068] In this embodiment, the training samples are input into the first language base model, and the audio samples in the training samples are input into the audio encoder for encoding to obtain audio features corresponding to the audio samples (referred to as target audio features for ease of description and distinction), and the target audio features of the audio encoder are input into the acoustic model.
[0069] Step 204: Input the target language features, target text features, and target audio features into the acoustic model for speech synthesis prediction to obtain predicted audio features.
[0070] In this embodiment, the target language features, target text features and target audio features are input into the acoustic model, and the acoustic model learns these features and performs speech synthesis prediction, and outputs predicted audio features.
[0071] The training method of the speech synthesis model of this embodiment is as follows: the language identifier is input into the language embedding layer to obtain the target language features corresponding to the language identifier determined by the language embedding layer based on the language dictionary, the text sample is input into the text encoder for encoding to obtain the target text features corresponding to the text sample, wherein the text encoder is pre-trained based on multilingual text data, the audio sample is input into the audio encoder for encoding to obtain the target audio features corresponding to the audio sample, and then the target language features, target text features and target audio features are input into the acoustic model for speech synthesis prediction to obtain the predicted audio features, thereby realizing speech synthesis prediction by combining text features, audio features and language features, so that the acoustic model can better adapt to the pronunciation style of the learning language and improve the model training speed and effect.
[0072] Further, in an optional embodiment of the present disclosure, in Figure 2Based on the illustrated embodiment, the first language base model can be trained in the following manner: obtaining a pre-training sample of the first language, the pre-training sample is marked with a language identifier of the target language and includes a pre-training text sample of the target language and a corresponding pre-training audio sample, the target language is Chinese and / or English, that is, one pre-training sample is marked with a language identifier of a language (English or Chinese) and includes a pre-training text sample of the language and a corresponding pre-training audio sample; inputting the language identifier of the target language into the language embedding layer to obtain sample language features corresponding to the target language; inputting the pre-training text sample of the target language into the text encoder to obtain sample text features corresponding to the pre-training text sample; and, inputting the pre-training audio sample of the target language into the audio encoder to obtain sample audio features corresponding to the pre-training audio sample; further, inputting the sample language features, sample text features and sample audio features into the acoustic model for speech synthesis prediction to obtain an audio feature prediction result; calculating a loss value based on the audio feature prediction result and the sample audio features; optimizing the parameters of the text encoder, the audio encoder and the acoustic model with minimizing the loss value as the optimization target to obtain a trained first language base model.
[0073] Figure 3 FIG. 1 is a structural diagram of a speech synthesis model according to an exemplary embodiment of the present disclosure. Figure 3 As shown, the speech synthesis model includes a language embedding layer, a text encoder, an audio encoder and an acoustic model. The text encoder is a pre-trained unified text encoder for multiple languages, which can encode the input text data to obtain text features. The language embedding layer expands the language dictionary including language features of multiple languages, and can convert the input language identifier into the corresponding language features. It can be understood that the structure of the first language base model is consistent with the structure of the speech synthesis model. When the speech synthesis model is trained based on the first language base model, the audio encoder encodes the input audio sample to obtain the corresponding audio features, and inputs the acoustic model for speech synthesis prediction. The acoustic model performs speech synthesis prediction based on the output of the audio encoder, the text encoder and the language embedding layer, and outputs the predicted discrete audio features. The present disclosure only needs to perform simple adaptation on the text encoder, and can be trained and inferred for multilingual data. It is a simple, fast and efficient solution for expanding from a single language synthesis model to a multilingual speech synthesis model.
[0074] Corresponding to the above embodiments, the present disclosure also provides a speech synthesis method implemented based on the speech synthesis model trained in the above embodiments.
[0075] Figure 4A flowchart of a speech synthesis method provided for an exemplary embodiment of the present disclosure is provided. The method can be performed by a speech synthesis device provided for an embodiment of the present disclosure. The speech synthesis device can be implemented by software and / or hardware and can be integrated in an electronic device.
[0076] like Figure 4 As shown, the speech synthesis method may include the following steps:
[0077] Step 301, obtaining text to be processed and reference audio information.
[0078] The reference audio information includes the reference audio, the language of the reference audio and the reference text corresponding to the reference audio. The reference audio information is used to instruct the speech synthesis model to output discrete audio features that conform to the sound characteristics of the reference audio.
[0079] Step 302: Input the text to be processed and the reference audio information into a speech synthesis model to perform speech synthesis prediction, and obtain audio features corresponding to the text to be processed.
[0080] Among them, the speech synthesis model is trained by the speech synthesis model training method described in the aforementioned embodiment, and the language applicable to the speech synthesis model includes the language of the text to be processed. For example, the speech synthesis model is applicable to Chinese, English and French, and the language of the text to be processed can be any one of Chinese, English and French.
[0081] In this embodiment, the acquired text to be processed and reference audio information are input into a pre-trained speech synthesis model, the text encoder of the speech synthesis model encodes the text to be processed to obtain text features, the language embedding layer of the speech synthesis model outputs corresponding language features according to the language of the text to be processed, and then the acoustic model of the speech synthesis model performs speech synthesis prediction based on the language features and text features, and under the guidance of the input reference audio information, outputs audio features that correspond to the text to be processed and conform to the sound characteristics of the reference audio.
[0082] Step 303: Input the audio features corresponding to the text to be processed into a vocoder for processing to obtain the audio corresponding to the text to be processed.
[0083] In this embodiment, after obtaining the audio features output by the speech synthesis model, the audio features can be input into a vocoder, which converts the audio features into audio, thereby obtaining audio that corresponds to the text to be processed and conforms to the sound characteristics of the reference audio.
[0084] The speech synthesis method of the embodiment of the present disclosure obtains the text to be processed and the reference audio information, inputs the text to be processed and the reference audio information into a speech synthesis model for speech synthesis prediction, obtains the audio features corresponding to the text to be processed, and then inputs the audio features corresponding to the text to be processed into a vocoder for processing to obtain the audio corresponding to the text to be processed, thereby realizing text-to-speech conversion.
[0085] In order to implement the above embodiments, the present disclosure also provides a training device for a speech synthesis model. The training device for the speech synthesis model can be implemented by software and / or hardware and can be integrated in an electronic device.
[0086] Figure 5 A structural diagram of a training device for a speech synthesis model provided in an embodiment of the present disclosure is shown in FIG. Figure 5 As shown, the training device 40 of the speech synthesis model may include: a first acquisition module 410, a second acquisition module 420 and a training module 430.
[0087] The first acquisition module 410 is used to acquire a pre-trained first language base model, wherein the language embedding layer of the first language base model includes a language dictionary, the language dictionary includes language features corresponding to a plurality of languages, and the first language is at least one of Chinese and English;
[0088] A second acquisition module 420 is used to acquire a training sample in a second language, the training sample is marked with a language identifier of the second language and includes a text sample and a corresponding audio sample in the second language, the second language including other languages different from the first language;
[0089] The training module 430 is used to continue training based on the first language base model to obtain predicted audio features corresponding to the training samples, wherein the language embedding layer of the first language base model determines the target language features corresponding to the language identifier based on the language dictionary, and the target language features are used to determine the predicted audio features; based on the predicted audio features and the target audio features corresponding to the audio samples, a loss is calculated to obtain a current loss value; with minimizing the current loss value as the optimization goal, parameter optimization is continued based on the parameters of the first language base model to obtain a trained speech synthesis model.
[0090] Optionally, the first language base model further includes a text encoder, an audio encoder and an acoustic model; the training module 430 is further used for:
[0091] Inputting the language identifier into the language embedding layer to obtain the target language features corresponding to the language identifier determined by the language embedding layer based on the language dictionary;
[0092] Inputting the text sample into a text encoder for encoding to obtain target text features corresponding to the text sample, wherein the text encoder is pre-trained based on multilingual text data;
[0093] Input the audio sample into the audio encoder for encoding to obtain the target audio feature corresponding to the audio sample;
[0094] The target language features, target text features, and target audio features are input into the acoustic model for speech synthesis prediction to obtain predicted audio features.
[0095] Further optionally, the text encoder is pre-trained in the following manner:
[0096] Acquire first text data in multiple languages;
[0097] Build the initial text encoder;
[0098] The initial text encoder is trained using the first text data to obtain a text encoder.
[0099] Further optionally, the first language base model is obtained by:
[0100] Obtaining a pre-training sample in a first language, wherein the pre-training sample is marked with a language identifier of a target language and includes a pre-training text sample and a corresponding pre-training audio sample in the target language, wherein the target language is Chinese and / or English;
[0101] Input the language identifier of the target language into the language embedding layer to obtain sample language features corresponding to the target language;
[0102] Input the pre-trained text sample of the target language into the text encoder to obtain the sample text features corresponding to the pre-trained text sample;
[0103] Inputting the pre-trained audio sample of the target language into the audio encoder to obtain the sample audio features corresponding to the pre-trained audio sample;
[0104] Inputting sample language features, sample text features, and sample audio features into an acoustic model for speech synthesis prediction to obtain audio feature prediction results;
[0105] The loss value is calculated based on the audio feature prediction result and the sample audio feature;
[0106] Taking minimizing the loss value as the optimization goal, the parameters of the text encoder, audio encoder and acoustic model are optimized to obtain a trained first language base model.
[0107] Optionally, the second language also includes Chinese and English.
[0108] The training device for the speech synthesis model applicable to the electronic device provided in the embodiment of the present disclosure can execute the training method for the speech synthesis model provided in the embodiment of the present disclosure, and has the functional modules and beneficial effects corresponding to the execution method. The contents not described in detail in the embodiment of the device of the present disclosure can refer to the description in any method embodiment of the present disclosure.
[0109] In order to implement the above embodiments, the present disclosure further provides a speech synthesis device, which can be implemented by software and / or hardware and can be integrated in an electronic device.
[0110] Figure 6 A schematic diagram of the structure of a speech synthesis device provided in an embodiment of the present disclosure is shown in FIG. Figure 6 As shown, the speech synthesis device 50 may include: a text acquisition module 510 , a prediction module 520 and a speech synthesis module 530 .
[0111] Wherein, the text acquisition module 510 is used to acquire the text to be processed and the reference audio information;
[0112] A prediction module 520 is used to input the text to be processed and the reference audio information into a speech synthesis model for speech synthesis prediction, and obtain audio features corresponding to the text to be processed, wherein the speech synthesis model is trained by the training method of the speech synthesis model described in the above embodiment;
[0113] The speech synthesis module 530 is used to input the audio features corresponding to the text to be processed into the vocoder for processing to obtain the audio corresponding to the text to be processed.
[0114] The speech synthesis device applicable to electronic devices provided in the embodiments of the present disclosure can execute the speech synthesis method provided in the embodiments of the present disclosure, and has the corresponding functional modules and beneficial effects of the execution method. For the contents not described in detail in the embodiments of the apparatus of the present disclosure, reference can be made to the description in any method embodiment of the present disclosure.
[0115] The embodiments of the present disclosure also provide an electronic device, including a processor and a memory; the processor calls the program or instructions stored in the memory to execute the steps of the various embodiments of the training method of the speech synthesis model as described above, or executes the steps of the various embodiments of the speech synthesis method as described above. To avoid repeated description, they will not be repeated here.
[0116] The embodiments of the present disclosure also provide a computer-readable storage medium, which is non-transitory and stores programs or instructions. The programs or instructions enable a computer to execute the steps of the embodiments of the training method of the speech synthesis model as described above, or to execute the steps of the embodiments of the speech synthesis method as described above. To avoid repeated description, they will not be repeated here.
[0117] The embodiments of the present disclosure also provide a computer program product, which is used to execute the steps of the various embodiments of the training method of the speech synthesis model as described above, or to execute the steps of the various embodiments of the speech synthesis method as described above. To avoid repeated description, they are not repeated here.
[0118] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.
[0119] The above description is only a specific embodiment of the present disclosure, so that those skilled in the art can understand or implement the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to the embodiments described herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A method for training a speech synthesis model, characterized in that: The method comprises: Obtain a pre-trained first language base model, wherein the language embedding layer of the first language base model includes a language dictionary, wherein the language dictionary includes language features corresponding to a plurality of languages, respectively, and the first language is at least one of Chinese and English; Acquire a training sample in a second language, wherein the training sample is marked with a language identifier of the second language and includes a text sample and a corresponding audio sample in the second language, wherein the second language includes another language different from the first language; Continue training based on the first language base model to obtain predicted audio features corresponding to the training sample, wherein the language embedding layer of the first language base model determines a target language feature corresponding to the language identifier based on the language dictionary, and the target language feature is used to determine the predicted audio feature; Performing loss calculation based on the predicted audio feature and the target audio feature corresponding to the audio sample to obtain a current loss value; Taking minimizing the current loss value as an optimization goal, parameter optimization is continued based on the parameters of the first language base model to obtain a trained speech synthesis model.
2. The method according to claim 1, characterized in that The first language base model also includes a text encoder, an audio encoder and an acoustic model; The continuing training based on the first language base model to obtain the predicted audio features corresponding to the training samples includes: Inputting the language identifier into the language embedding layer to obtain the target language feature corresponding to the language identifier determined by the language embedding layer based on the language dictionary; Inputting the text sample into the text encoder for encoding to obtain target text features corresponding to the text sample, wherein the text encoder is pre-trained based on multilingual text data; Inputting the audio sample into the audio encoder for encoding to obtain a target audio feature corresponding to the audio sample; The target language feature, the target text feature and the target audio feature are input into the acoustic model for speech synthesis prediction to obtain the predicted audio feature.
3. The method according to claim 2, characterized in that The text encoder is pre-trained in the following way: Acquire first text data in the multiple languages; Build the initial text encoder; The initial text encoder is trained using the first text data to obtain the text encoder.
4. The method according to claim 3, characterized in that The first language base model is obtained in the following way: Obtaining a pre-training sample in the first language, wherein the pre-training sample is marked with a language identifier of a target language and includes a pre-training text sample and a corresponding pre-training audio sample in the target language, wherein the target language is Chinese and / or English; Inputting the language identifier of the target language into the language embedding layer to obtain sample language features corresponding to the target language; Inputting the pre-trained text sample of the target language into the text encoder to obtain sample text features corresponding to the pre-trained text sample; Inputting the pre-trained audio sample of the target language into the audio encoder to obtain sample audio features corresponding to the pre-trained audio sample; Inputting the sample language feature, the sample text feature and the sample audio feature into an acoustic model for speech synthesis prediction to obtain an audio feature prediction result; Calculate a loss value based on the audio feature prediction result and the sample audio feature; Taking minimizing the loss value as an optimization goal, parameters of the text encoder, the audio encoder and the acoustic model are optimized to obtain a trained first language base model.
5. The method according to any one of claims 1 to 4, characterized in that: The second language also includes Chinese and English.
6. A speech synthesis method, characterized in that: The method comprises: Obtain the text to be processed and reference audio information; Inputting the text to be processed and the reference audio information into a speech synthesis model for speech synthesis prediction to obtain audio features corresponding to the text to be processed, wherein the speech synthesis model is trained by the training method for a speech synthesis model according to any one of claims 1 to 5; The audio features corresponding to the text to be processed are input into a vocoder for processing to obtain the audio corresponding to the text to be processed.
7. A training device for a speech synthesis model, characterized in that: include: A first acquisition module is used to acquire a pre-trained first language base model, wherein the language embedding layer of the first language base model includes a language dictionary, wherein the language dictionary includes language features corresponding to a plurality of languages, respectively, and the first language is at least one of Chinese and English; A second acquisition module is used to acquire training samples in a second language, wherein the training samples are marked with a language identifier of the language and include a text sample and a corresponding audio sample in the second language, wherein the second language includes other languages different from the first language; A training module, configured to continue training based on the first language base model to obtain predicted audio features corresponding to the training samples, wherein the language embedding layer of the first language base model determines a target language feature corresponding to the language identifier based on the language dictionary, and the target language feature is used to determine the predicted audio feature; A loss calculation is performed based on the predicted audio features and the target audio features corresponding to the audio samples to obtain a current loss value; and parameter optimization is continued based on the parameters of the first language base model with minimizing the current loss value as an optimization goal to obtain a trained speech synthesis model.
8. A speech synthesis device, characterized in that: include: A text acquisition module is used to acquire the text to be processed and reference audio information; A prediction module, used for inputting the text to be processed and the reference audio information into a speech synthesis model for speech synthesis prediction, and obtaining audio features corresponding to the text to be processed, wherein the speech synthesis model is trained by the training method of the speech synthesis model according to any one of claims 1 to 5; The speech synthesis module is used to input the audio features corresponding to the text to be processed into the vocoder for processing to obtain the audio corresponding to the text to be processed.
9. An electronic device, characterized in that: include: Processor and memory; The processor is used to execute the training method of the speech synthesis model as described in any one of claims 1 to 5, or to execute the speech synthesis method as described in claim 6 by calling the program or instruction stored in the memory.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a program or instruction, which enables a computer to execute the training method for a speech synthesis model as described in any one of claims 1 to 5, or to execute the speech synthesis method as described in claim 6.