Speech synthesis model construction method and system, storage medium and program product

Abstract

The invention provides a speech synthesis model construction method and system, a storage medium and a program product. The method comprises the following steps: acquiring audio, characters and pinyin; setting an audio coding module to extract audio codes from the audio, setting a character token to convert the characters into character codes, and setting a pinyin coding module to extract pinyin codes from the pinyin; a feature fusion module is arranged to fuse the audio codes and the character codes or the pinyin codes to obtain fused features; processing the fused features through a large language model to obtain a final audio vector; an acoustic decoder is adopted to decode the final audio vector to obtain a final audio; and performing reverse iteration on the speech synthesis model according to the loss. The speech synthesis model can accurately control pronunciation of a single character, and solves the problem of poor pronunciation accurate control capability in speech synthesis in an existing speech synthesis model.

Description

Technical Field

[0001] The present invention relates to the field of intelligent speech technology, and in particular to a speech synthesis model construction method, system, storage medium and program product. Background Art

[0002] Traditional speech synthesis models often synthesize sound waveform files through text, phonemes, intermediate features, and Mel-spectrograms. Phonemes can be generally understood as pinyin. Phonemes are used as the smallest pronunciation units, and the synthesis of audio is controlled by predicting the pronunciation duration and pitch of syllables.

[0003] In recent years, speech synthesis models have gradually begun to use large language models (LLMs) to generate special semantic tokens based on input text and audio information, and then use pre-trained decoders to decode these semantic tokens into sound waveforms to complete speech synthesis. Since phonemes are no longer explicitly used as input, a tokenization method that is more suitable for large language models, such as BPE, is adopted. Although such a model design can achieve better speech naturalness, it also loses the ability to accurately control the pronunciation in speech synthesis to a certain extent.

[0004] Since polyphonic characters often appear in Chinese, and considering dialects and other factors, customers often have the need to define the pronunciation of a certain character. Therefore, it is urgent to invent a speech synthesis model that can accurately control the pronunciation of a single word to solve the problem of poor ability of existing speech synthesis models to accurately control the pronunciation in speech synthesis. Summary of the invention

[0005] In view of the shortcomings of the prior art, the present invention provides a method, system, storage medium and program product for constructing a speech synthesis model, aiming to introduce a pinyin encoding module and use a specific training method to enable the model to have the ability to accurately control the pronunciation of single words after training.

[0006] In a first aspect, the present invention provides a method for constructing a speech synthesis model, comprising the following steps:

[0007] Get audio, text, and pinyin;

[0008] Set an audio encoding module to extract audio encoding from the audio, set a text tokenizer to convert the text into text encoding, and set a pinyin encoding module to extract pinyin encoding from the pinyin;

[0009] Setting a feature fusion module to fuse the audio code and the text code or the pinyin code to obtain a fused feature;

[0010] Passing the fused features through a large language model to obtain a final audio vector;

[0011] Using an acoustic decoder to decode the final audio vector to obtain a final audio;

[0012] Iterate backward through the loss for the speech synthesis model.

[0013] As a further improvement of the present invention, the obtaining of audio, text, and pinyin includes:

[0014] Get audio;

[0015] Classifying the audio according to different speakers to obtain classified audio;

[0016] Cutting the classified audio into a number of short sentence audios;

[0017] Using an ASR automatic speech recognition model to obtain text corresponding to the short sentence audio;

[0018] The G2P model is used to obtain the pinyin corresponding to the text.

[0019] As a further improvement of the present invention, the feature fusion module has a 50% probability of fusing the audio code with the text code, and has a 50% probability of fusing the audio code with the pinyin code.

[0020] As a further improvement of the present invention, the reverse iteration of the speech synthesis model according to the loss includes: using KL divergence loss to constrain the distribution of the pinyin code and the text code in the hidden space, so that the pinyin code and the text code distribution are closer, and the loss function Loss_new in each iteration is:

[0021] Loss_new=Loss_old+KL(P1(phone1),T1(text1)),

[0022] Among them, Loss_old is the original model loss function, KL() is the KL divergence loss, P1(phone1) is the pinyin encoding, and T1(text1) is the text encoding.

[0023] As a further improvement of the present invention, the KL divergence loss is calculated as follows:

[0024] KL(y_pred,y_true)=y_true*log(y_true / y_pred),

[0025] Among them, KL() is KL divergence loss, y_true=T1(text1), y_pred=P1(phone1), P1(phone1) is the pinyin code, and T1(text1) is the text code.

[0026] In a second aspect, the present invention provides a computer system comprising a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method described in the first aspect.

[0027] In a third aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in the first aspect.

[0028] In a fourth aspect, the present invention provides a computer program product, comprising a computer program, which implements the steps of the method described in the first aspect when executed by a processor.

[0029] Compared with the prior art, the present invention has the following beneficial effects:

[0030] The present invention provides a speech synthesis model construction method, system, storage medium and program product, which can accurately control the pronunciation of single words in the speech synthesis model, and solve the problem that the existing speech synthesis model has poor ability to accurately control the pronunciation in speech synthesis. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 A schematic diagram of a speech synthesis model provided for one embodiment of the present invention. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the present invention will combine the accompanying drawings to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments, wherein steps S1, S2... in the embodiments described in the present invention do not limit the only execution steps of the present invention; the various models described in the present invention are not the only limiting methods of the present invention. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0033] In the present invention, computer devices / equipment / systems refer to related entities applied to computers, such as hardware, a combination of hardware and software, software or software in execution, etc. In detail, for example, software includes but is not limited to a process running on a processor, a processor, an object, executable software, an execution thread, a program and / or a computer. In addition, an application or script program running on a server, a server can also be software. One or more software can be in an execution process and / or thread, and the software can be localized on one computer and / or distributed between two or more computers, and can be run by various computer-readable media.

[0034] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application may be combined with each other.

[0035] In a first aspect, an embodiment of the present invention provides a method for constructing a speech synthesis model, such as Figure 1 As shown, the specific steps can be as follows:

[0036] S1: Get audio, text, and pinyin;

[0037] Preferably, the obtaining of audio, text, and pinyin includes:

[0038] Get audio;

[0039] Classifying the audio according to different speakers to obtain classified audio;

[0040] Cutting the classified audio into a number of short sentence audios;

[0041] Using an ASR automatic speech recognition model to obtain text corresponding to the short sentence audio;

[0042] The G2P model is used to obtain the pinyin corresponding to the text.

[0043] Among them, the G2P (Grapheme-to-Phoneme) model is a grapheme-to-phoneme model, which can be used to convert text into corresponding pinyin.

[0044] S2: setting an audio encoding module to extract audio encoding from the audio, setting a text tokenizer to convert the text into text encoding, and setting a pinyin encoding module to extract pinyin encoding from the pinyin;

[0045] S3: Setting a feature fusion module to fuse the audio code and the text code or the pinyin code to obtain a fused feature;

[0046] Preferably, the feature fusion module has a 50% probability of fusing the audio code with the text code, and a 50% probability of fusing the audio code with the pinyin code.

[0047] S4: Pass the fused features through a large language model to obtain a final audio vector;

[0048] S5: using an acoustic decoder to decode the final audio vector to obtain a final audio;

[0049] S6: Iterate the speech synthesis model backward based on the loss.

[0050] Preferably, the reverse iteration of the speech synthesis model according to the loss includes: using KL divergence loss to constrain the distribution of the phonetic code and the text code in the hidden space, so that the distribution of the phonetic code and the text code is closer, and the loss function Loss_new in each iteration is:

[0051] Loss_new=Loss_old+KL(P1(phone1),T1(text1)),

[0052] Among them, Loss_old is the original model loss function, KL() is the KL divergence loss, P1(phone1) is the pinyin encoding, and T1(text1) is the text encoding.

[0053] Preferably, the KL divergence loss is calculated as follows:

[0054] KL(y_pred,y_true)=y_true*log(y_true / y_pred),

[0055] Among them, KL() is KL divergence loss, y_true=T1(text1), y_pred=P1(phone1), P1(phone1) is the pinyin code, and T1(text1) is the text code.

[0056] In one embodiment of the present invention, illustratively, 50,000 to 100,000 hours of audio are collected, the audio is classified according to different speakers to obtain classified audio, the classified audio is cut into several short sentence audios of 5 to 20 seconds, an ASR automatic speech recognition model is used to obtain the text corresponding to the short sentence audio, and a G2P model is used to obtain the pinyin corresponding to the text, to obtain data pairs for input (text1, audio1, phone1), which represent text, audio, and pinyin, respectively.

[0057] Set the audio encoding module to C1, extract the audio code C1(audio1) from the audio audio1; set the text tokenizer to T1, convert the text text1 into the text code T1(text1); set the pinyin encoding module to P1, extract the pinyin code P1(phone1) from the pinyin phone1.

[0058] Set the feature fusion module to F1 and obtain the fused feature Feature_LLM.

[0059] Among them, there is a 50% chance that the audio encoding and text encoding will be merged, namely:

[0060] Feature_LLM=F1(T1(text1),C1(audio1)),

[0061] There is a 50% chance that the audio encoding and the pinyin encoding will be merged, that is:

[0062] Feature_LLM=F1(P1(phone1),C1(audio1)).

[0063] Set up the LLM module and pass the fused feature Feature_LLM through the large language model to obtain the final audio vector;

[0064] Using an acoustic decoder to decode the final audio vector to obtain the final audio;

[0065] The speech synthesis model is iterated in reverse according to the loss, and the KL divergence loss is used to constrain the distribution of the pinyin code and the text code in the hidden space, so that the distribution of the two is closer. For example, the KL divergence loss here is implemented by KLdivLoss in the pytorch package, that is:

[0066] KL(y_pred,y_true)=y_true*log(y_true / y_pred),

[0067] Among them, KL() is KL divergence loss, y_true=T1(text1), y_pred=P1(phone1), P1(phone1) is the pinyin code, and T1(text1) is the text code.

[0068] The loss function Loss_new at each iteration is:

[0069] Loss_new=Loss_old+KL(P1(phone1),T1(text1)),

[0070] Among them, Loss_old is the original model loss function, KL() is the KL divergence loss, P1(phone1) is the pinyin encoding, and T1(text1) is the text encoding.

[0071] To facilitate understanding of Loss_old, for example, taking fish speech as an example, its input sequence is {t_1, t_2, ...., v_1, v_2, ...}, where t represents the text token sequence and v represents the audio token sequence, that is, the text sequence is concatenated with the audio sequence as input. The output sequence is {t_1_pred, t_2_pred, ...., v_1_pred, v_2_pred, ....}. The loss function is divided into two parts, one part is called base_loss = CE_loss (t_seq, t_seq_pred), that is, the cross entropy loss is taken with the text part sequence, and the other part is semantic_loss = CE_loss (v_se q, v_seq_pred), that is, the cross entropy loss is taken with the audio token sequence. Finally, Loss_old = base_loss + semantic_loss.

[0072] It should be noted that Loss_old may be different in different LLM-based TTS speech synthesis models, and no matter what kind of Loss_old is, it will not affect the implementation of the present invention. Therefore, the exemplary description of Loss_old in the present invention should not be regarded as limiting the scope of protection of the present invention. All other embodiments obtained based on the embodiments of the present invention by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0073] In a second aspect, an embodiment of the present invention provides a computer system, comprising a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method described in the first aspect.

[0074] In a third aspect, an embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the method described in the first aspect are implemented.

[0075] In a fourth aspect, an embodiment of the present invention provides a computer program product, comprising a computer program, which implements the steps of the method described in the first aspect when executed by a processor.

[0076] The present invention adds a pinyin encoding module, randomly uses different feature fusions as LLM input during training, and uses KL divergence as a training constraint to make the output of the pinyin encoding and the original text token more similar in distribution in the feature space, thereby ensuring that the naturalness of the original LLM-based speech synthesis model is not affected. Without affecting the original speech synthesis quality, a new module is introduced, and the word granularity pronunciation of the synthesized speech can be finely controlled through the new module.

Claims

1. A method for constructing a speech synthesis model, characterized in that: The following steps are involved: Get audio, text, and pinyin; Set an audio encoding module to extract audio encoding from the audio, set a text tokenizer to convert the text into text encoding, and set a pinyin encoding module to extract pinyin encoding from the pinyin; Setting a feature fusion module to fuse the audio code and the text code or the pinyin code to obtain a fused feature; Passing the fused features through a large language model to obtain a final audio vector; Using an acoustic decoder to decode the final audio vector to obtain a final audio; Iterate backward through the loss for the speech synthesis model.

2. The method for constructing a speech synthesis model according to claim 1, characterized in that: The obtaining of audio, text, and pinyin includes: Get audio; Classifying the audio according to different speakers to obtain classified audio; Cutting the classified audio into a number of short sentence audios; Using an ASR automatic speech recognition model to obtain text corresponding to the short sentence audio; The G2P model is used to obtain the pinyin corresponding to the text.

3. The method for constructing a speech synthesis model according to claim 1, characterized in that: The feature fusion module has a 50% chance of fusing the audio code with the text code, and a 50% chance of fusing the audio code with the pinyin code.

4. The method for constructing a speech synthesis model according to claim 1, characterized in that: The reverse iteration of the speech synthesis model according to the loss includes: using KL divergence loss to constrain the distribution of the pinyin code and the text code in the hidden space, so that the pinyin code and the text code distribution are closer, and the loss function Loss_new in each iteration is: Loss_new=Loss_old+KL(P1(phone1),T1(text1)), Among them, Loss_old is the original model loss function, KL() is the KL divergence loss, P1(phone1) is the pinyin encoding, and T1(text1) is the text encoding.

5. The method for constructing a speech synthesis model according to claim 4, characterized in that: The KL divergence loss is calculated as follows: KL(y_pred,y_true)=y_true*log(y_true / y_pred), Among them, KL() is KL divergence loss, y_true=T1(text1), y_pred=P1(phone1), P1(phone1) is the pinyin code, and T1(text1) is the text code.

6. A computer system comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 5.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

8. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images