Customizable speech synthesis method and device and electronic equipment

Abstract

The invention discloses a customizable speech synthesis method and apparatus, and an electronic device. The method comprises the steps of generating a training text set, a test text set, and cloned speech corresponding to the training text set; filtering the generated clone voice and obtaining a recognition text, and refiltering the recognition text according to a real training text to obtain a training set; training a speech synthesis model by using the training set, generating speech of the test text set by using the speech synthesis model, and counting the error rate and error phonemes of the speech; and generating an enhanced training text set containing the error phonemes to obtain an enhanced training set, and finely tuning the speech synthesis model by using the enhanced training set until the error rate is reduced to a threshold value or a stable state. Through the method, the stability of the speech synthesis model is enhanced, the data cost of the customized speech synthesis model is reduced, the model iteration period is shortened, the accuracy of customized speech synthesis is improved, and the user experience is improved.

Description

Technical Field

[0001] The present invention relates to the field of speech synthesis technology, and in particular to a customizable speech synthesis method, device and electronic equipment. Background Art

[0002] Speech synthesis, also known as text-to-speech synthesis (TTS), is a technology widely used in text readers and human-computer interaction. As a result, speech synthesis technology has attracted more and more attention from researchers. The methods of speech synthesis have evolved from the earliest concatenated speech synthesis and statistical parameter speech synthesis to the end-to-end speech synthesis system that directly inputs text or phonetic characters to guide speech synthesis in recent years, which has continuously improved the quality of synthesized speech and is almost close to the quality of human speech. Despite this, the synthesized speech currently generally lacks emotional color and tone information. In daily conversations, tone expresses the speaker's views and attitudes towards a certain behavior or phenomenon. It can bring a wide range of emotional colors to the discourse information without changing the facts of the content stated in the sentence. Chinese and English tone can be roughly divided into declarative sentences, interrogative sentences, exclamatory sentences, and imperative sentences. Of course, each tone can be subdivided into smaller granularity.

[0003] At present, the research on speech synthesis focuses on emotion, rhythm, pitch, and tone. Although it can effectively synthesize speech with style to achieve customization, customized speech synthesis has problems such as high recording cost and long recording cycle. In recent years, due to the development of speech cloning technology, the use of large speech models to clone the target speech is very effective. However, large speech models face poor stability, such as unstable timbre, unstable emotions, typos and omissions, which leads to poor stability of customized synthesized speech. Summary of the invention

[0004] The object of the present invention is to provide a customizable speech synthesis method, device and electronic device, so as to solve the technical problem of poor stability of customized synthesized speech in the prior art.

[0005] The present invention solves the above problems through the following technical solutions:

[0006] A customizable speech synthesis method comprises the following steps:

[0007] S1. Generate training text set and test text set using large language model;

[0008] S2. Generate cloned speech corresponding to the training text set using the large speech model and the target speech set;

[0009] S3. Using the voiceprint recognition model and the emotion recognition model, the generated cloned voice is first filtered;

[0010] S4. Using the speech recognition model, obtain the recognized text of the cloned speech after the first filtering, and perform a second filtering on the recognized text according to the real training text in the training text set to obtain a training set;

[0011] S5. Use the training set of S4 to train the speech synthesis model;

[0012] S6. Generate speech of the test text set using the trained speech synthesis model, and use the speech recognition model to count the error rate and incorrect phonemes of the speech;

[0013] S7. Use the large language model to generate an enhanced training text set containing the incorrect phonemes in S6, and then repeat steps S2-S4 from the enhanced training text set to obtain an enhanced training set;

[0014] S8. Fine-tune the speech synthesis model trained in S5 using the enhanced training set;

[0015] S9. Repeat steps S6-S8 until the error rate drops to a threshold or a stable state.

[0016] As a further improvement, the training text set and the test text set cover all pronunciations.

[0017] As a further improvement, the total duration of the target speech set is not less than one minute.

[0018] As a further improvement, the voiceprint recognition model adopts the ECAPA voiceprint recognition model; the emotion recognition model adopts the large version of the emotion recognition model of emotion2vec.

[0019] As a further improvement, the generated cloned voice is first filtered, and the specific method includes:

[0020] The voiceprint recognition model and emotion recognition model are used to obtain the voiceprint and emotion embedding vector of each target voice and the corresponding cloned voice. The distance between the voiceprint and emotion embedding vector of the cloned voice and the target voice is calculated. The distance measurement includes Euclidean distance and cosine similarity, and the cloned voices with a longer distance are filtered out.

[0021] As a further improvement, the speech recognition model converts the obtained recognition text into phonemes to facilitate comparison and filtering with real training text phonemes in the training text set.

[0022] As a further improvement, the speech recognition model is used to count the error rate and erroneous phonemes of the speech, and the specific method includes:

[0023] The speech recognition model is used to recognize the speech of the test text set generated by the speech synthesis model, and the insertion errors, replacement errors, and deletion errors of the recognized text compared with the real test text in the test text set are counted, and the word error rate is obtained; at the same time, the erroneous phonemes and the corresponding error frequencies are recorded, and the erroneous phoneme set is output after sorting by error frequency.

[0024] As a further improvement, the trained speech synthesis model is fine-tuned, and the number of fine-tuning iterations is determined according to the loss curve during training and the sampling test of the inference audio.

[0025] At the same time, the present invention also solves the above problems through the following technical solutions:

[0026] A customizable speech synthesis device, characterized by comprising:

[0027] The text generation module uses a large language model to generate training text sets and test text sets;

[0028] The voice cloning module uses a large voice model to clone the timbre and emotion of the target voice and obtain the cloned voice corresponding to the training text set;

[0029] The data screening module performs the first filtering and speech recognition on the cloned voice according to the voiceprint and emotion distance between the cloned voice and the target voice, and performs the second filtering on the cloned voice with pronunciation errors through the speech recognition results to obtain the training set;

[0030] Model training module, which uses the training set to train the speech synthesis model;

[0031] The enhancement module uses the speech recognition model to count the error rate and erroneous phonemes of the speech synthesis model inference results, and fine-tunes the speech synthesis model by constructing enhanced training sets multiple times to make the speech synthesis model stable.

[0032] In addition, the present invention also solves the above-mentioned problem through the following technical solutions:

[0033] An electronic device includes: a processor and a memory, wherein the processor uses a computer program stored in the memory to implement a customizable speech synthesis method as described above.

[0034] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0035] The present invention uses a large language model, a large speech model, a voiceprint recognition model, an emotion recognition model, and a speech recognition model to obtain a training set to train a speech synthesis model, and uses the speech output of the trained speech synthesis model, and again uses the speech recognition model to perform speech statistics of error rate and erroneous phonemes, and uses the large language model to obtain an enhanced training set for fine-tuning the trained speech synthesis model, and repeats the fine-tuning to reduce the error rate of the finally generated speech synthesis model to a threshold or a stable state; the method enhances the stability of the speech synthesis model, reduces the data cost of the customized speech synthesis model, shortens the model iteration cycle, improves the accuracy of customized speech synthesis, and improves user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 A flowchart of a customizable speech synthesis method of the present invention;

[0037] Figure 2 The structure block diagram of a customizable speech synthesis device of the present invention. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical solutions and advantages of the present application clearer, the technical solutions in the embodiments of the present application will be described in more detail below in conjunction with the drawings in the preferred embodiments of the present application. In the drawings, the same or similar reference numerals throughout represent the same or similar parts or parts with the same or similar functions. The described embodiments are part of the embodiments of the present application, not all of the embodiments. The embodiments described below with reference to the drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limitations on the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0039] In the description of the present application, it should be noted that, unless expressly provided and limited otherwise, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or display comprising a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units that are not explicitly listed or inherent to these processes, methods, products or displays.

[0040] The following will be combined Figure 1-2 , a customizable speech synthesis method, device and electronic device involved in the embodiments of the present application are described in detail. It is worth noting that the following embodiments are only used to explain the present application and do not constitute a limitation of the present application.

[0041] Embodiment 1:

[0042] Refer to the attached Figure 1 A customizable speech synthesis method according to an embodiment of the present invention comprises the following steps:

[0043] S1. Generate training text set and test text set using large language model;

[0044] Optionally, the large language model includes but is not limited to ChatGPT, Wenxinyiyan, etc. The training text set and the test text set are required to cover all pronunciations, including but not limited to specific fields. This embodiment uses ChatGPT to generate texts for six emotions: happy, sad, angry, natural, afraid, and surprised. The generated texts are deduplicated and the phonemes of the generated texts are counted. ChatGPT is used again to generate texts for the uncovered phonemes. Finally, 60,000 training sets and 6,000 test sets are generated.

[0045] S2. Generate cloned speech corresponding to the S1 training text set using the large speech model and the target speech set;

[0046] Understandably, the large voice model is required to have good zero-sample or few-sample cloning capabilities, including but not limited to models such as CosyVoice and GPT-So-VITS. In this embodiment, the CosyVoice voice large model is selected, which supports zero-sample cloning and supports the use of text and audio prompts; the target voice set requires clear pronunciation, good sound quality, and a total duration of not less than one minute. For target voices with poor sound quality, the sound quality enhancement model is used for processing. In the embodiment, the target voice contains six different emotional voices of a female anchor. The duration of each emotional voice is about 1 minute, and the sound quality is good or bad. After being processed by the sound quality enhancement model, the target voice reaches a usable state; different target voices are used with different Instructs or prompts for voice cloning, for example: the target voice and the corresponding text are used as audio and text prompts respectively, or the emotion is specified as happy and the gender is specified as female in the Instruct, and a total of 60,000 cloned voices with multiple emotions are generated.

[0047] S3. Use the voiceprint recognition model and the emotion recognition model to perform the first filtering on the cloned voice generated by S2;

[0048] Optionally, the voiceprint recognition model includes, but is not limited to, models such as ECAPA and ResNet, and the emotion recognition model includes, but is not limited to, models such as emotion2vec. In this embodiment, the ECAPA voiceprint model and the large version emotion recognition model of emotion2vec are selected. First, the voiceprint and emotion embedding vectors of each target voice and cloned voice are obtained using the voiceprint and emotion recognition models, and then the cosine similarity between the voiceprint and emotion embedding vectors of the cloned voice and the target voice is calculated, and the cloned voices with a relatively large distance are removed.

[0049] S4. Obtain a speech recognition model, obtain the recognition text for the cloned voices filtered in S3, and perform a second filtering according to the true training texts in the training text set in S1 to obtain a training set;

[0050] In this embodiment, the Paraformer model is used when processing Chinese, and the recognition result is converted from Chinese characters to phonemes. For example, "你好" is converted to "ni2 hao3". When processing English, the Whisper model is used, and the word recognition result is converted to phonemes. For example, "hello" is converted to "HH AH0 L OW1". The cloned voices with inconsistent phonemes between the recognition result and the true text are filtered to ensure the quality of the training set.

[0051] S5. Use the training set in S4 to train the speech synthesis model;

[0052] Optionally, the speech synthesis model includes, but is not limited to, models such as VITS and FastSpeech series models. In this embodiment, the BertVITS2 model based on the VITS series is selected, and the training set S4 constructed by the large model is used to train the model.

[0053] S6. Use the trained speech synthesis model to generate the voices of the test text set in S1, and use the S4 speech recognition model to count the error rate and error phonemes of the voices;

[0054] For example, in this embodiment, the Paraformer and Whisper speech recognition models are used to recognize the inference results of the speech synthesis model, and the insertion errors, substitution errors, and deletion errors of the phonemes of the recognition text compared with the true test texts in the test text set are counted. Finally, the word error rate is obtained, and the error phonemes and the corresponding error frequencies are recorded, and the error phoneme set is output in descending order of frequency, such as {'yi1','shi4', 'AH0', 'OW1'...}.

[0055] S7. Use the large language model in S1 to generate an enhanced training text set containing the error phonemes in S6, and then repeat the steps of S2 - S4 with the enhanced training text set to obtain an enhanced training set;

[0056] In this embodiment, we specify prompt to make the large language model play different roles (such as a sad historian, a happy female writer, etc.) to generate a large number of texts with different emotional types, and select the texts containing incorrect phoneme pronunciations into the enhanced training text set. Then, by generating cloned voices and filtering with emotion, voiceprint, and speech recognition models, we finally obtain the enhanced training set for fine-tuning the model.

[0057] S8. Fine-tune the speech synthesis model trained in S5 using the enhanced training set of S7;

[0058] It can be understood that based on the pre-trained speech synthesis model of S5, fine-tuning is performed using the above-mentioned enhanced training set. During the fine-tuning training process, when the training loss curve basically converges and stabilizes, and the model inference audio effect reaches stability at this time, fine-tuning is stopped.

[0059] S9. Repeat steps S6-S8 until the error rate drops to a threshold or a stable state;

[0060] It is understandable that in order to further improve the stability of the model, steps S6, S7, and S8 are repeated, including counting the word error rate and erroneous phonemes of the trained speech synthesis model, and fine-tuning the model by constructing an enhanced training set multiple times until the error rate is reduced to a threshold or reaches a stable state; for example, this embodiment repeats the above steps twice and finds that the word error rate of speech recognition is around 3%, so the training is completed to obtain the final speech synthesis model.

[0061] The customizable speech synthesis method provided in this embodiment 1 greatly shortens the acquisition cycle of customized speech synthesis data and reduces the cost of customized speech synthesis.

[0062] Embodiment 2:

[0063] An electronic device includes: a processor and a memory, wherein the processor uses a computer program stored in the memory to implement a customizable speech synthesis method as described in Example 1.

[0064] Embodiment 2:

[0065] Refer to the attached Figure 2 , a customizable speech synthesis device according to an embodiment of the present invention comprises:

[0066] The text generation module uses a large language model to generate training text sets and test text sets;

[0067] In this embodiment, ChatGPT was used to generate texts of six emotions: happy, sad, angry, natural, scared, and surprised. The generated texts were de-duplicated, and the phonemes of the generated texts were counted. For the phonemes not covered, ChatGPT was used again to generate texts. Finally, 60,000 training sets and 6,000 test sets were generated.

[0068] The voice cloning module uses a large voice model to clone the timbre and emotion of the target voice to obtain the cloned voice corresponding to the training text set.

[0069] In this embodiment, the CosyVoice large voice model is selected. This model supports zero-shot cloning and supports using text and audio prompts. The target voice set is required to have clear pronunciation, good sound quality, and a total duration of at least one minute. For target voices with poor sound quality, processing such as using a sound quality enhancement model is performed. In the embodiment, the target voice includes the voices of a certain female anchor with 6 different emotions, and the duration of each emotional voice is about 1 minute, with good and bad sound quality. After processing with the sound quality enhancement model, the target voice reaches an available state. Different target voices are paired with different Instructs or prompts for voice cloning. For example, the target voice and the corresponding text are used as audio and text prompts respectively, or the emotion is specified as happy and the gender is specified as female in the Instruct, etc. A total of 60,000 cloned voices with multiple emotions are generated.

[0070] The data screening module performs the first filtering and voice recognition on the cloned voices according to the distance between the cloned voices and the target voices in terms of voiceprint and emotion, and performs the second filtering on the cloned voices with pronunciation errors through the voice recognition results to obtain the training set.

[0071] In this embodiment, the ECAPA voiceprint model and the large version emotion recognition model of emotion2vec are selected. First, the voiceprint and emotion embedding vectors of each target voice and cloned voice are obtained using the voiceprint and emotion recognition models, and then the cosine similarity between the voiceprint and emotion embedding vectors of the cloned voice and the target voice is calculated. The cloned voices with a large distance are excluded. In this embodiment, Paraformer is used when processing Chinese, and the recognition results are converted from Chinese characters to phonemes. For example, "你好" is converted to "ni2 hao3". When processing English, Whisper is used, and the word recognition results are converted to phonemes. For example, "hello" is converted to "HH AH0 L OW1". The cloned voices with inconsistent phonemes between the recognition results and the real text are filtered to ensure the quality of the training set.

[0072] The model training module uses the training set to train the voice synthesis model.

[0073] This embodiment selects the BertVITS2 model based on the VITS series and uses the above-constructed training set to train the model.

[0074] The enhancement module uses the speech recognition model to count the error rate and erroneous phonemes of the speech synthesis model inference results, and fine-tunes the speech synthesis model by constructing enhanced training sets multiple times to make the speech synthesis model stable.

[0075] This embodiment uses the Paraformer and Whisper speech recognition models to identify the inference results of the speech synthesis model, and counts the insertion errors, replacement errors, and deletion errors of the recognized text compared with the real text phonemes, and finally obtains the word error rate. At the same time, the erroneous phonemes and the corresponding error frequencies are recorded, and the erroneous phoneme sets are output in descending order of frequency; this embodiment uses the enhancement module to construct an enhanced data set for model fine-tuning, and the word error rate of speech recognition is reduced from 5% to about 3%. The second construction of the enhanced data set for fine-tuning found that the error rate was also about 3%, so the final speech synthesis model was obtained.

[0076] Through the customizable speech synthesis device provided in this embodiment 3, data for training the speech synthesis model is obtained through the text generation module, the speech cloning module and the data screening module, and then the enhancement module is used to further improve the model stability. In addition, this method reduces the data cost of the customized speech synthesis model, shortens the model iteration cycle, improves the accuracy of customized speech synthesis, and improves user experience.

[0077] Although the present invention is described herein with reference to the illustrative embodiments of the present invention, the above embodiments are only preferred embodiments of the present invention, and the embodiments of the present invention are not limited to the above embodiments. It should be understood that those skilled in the art can design many other modifications and embodiments, which will fall within the scope and spirit of the principles disclosed in this application.

Claims

1. A customizable speech synthesis method, characterized in that: The following steps are involved: S1. Generate training text set and test text set using large language model; S2. Generate cloned speech corresponding to the training text set using the large speech model and the target speech set; S3. Using the voiceprint recognition model and the emotion recognition model, the generated cloned voice is first filtered; S4. Using the speech recognition model, obtain the recognized text of the cloned speech after the first filtering, and perform a second filtering on the recognized text according to the real training text in the training text set to obtain a training set; S5. Use the training set of S4 to train the speech synthesis model; S6. Generate speech of the test text set using the trained speech synthesis model, and use the speech recognition model to count the error rate and incorrect phonemes of the speech; S7. Use the large language model to generate an enhanced training text set containing the incorrect phonemes in S6, and then repeat steps S2-S4 from the enhanced training text set to obtain an enhanced training set; S8. Fine-tune the speech synthesis model trained in S5 using the enhanced training set; S9. Repeat steps S6-S8 until the error rate drops to a threshold or a stable state.

2. A customizable speech synthesis method according to claim 1, characterized in that: The training text set and the test text set cover all pronunciations.

3. A customizable speech synthesis method according to claim 1, characterized in that: The total duration of the target speech set is not less than one minute.

4. A customizable speech synthesis method according to claim 1, characterized in that: The voiceprint recognition model adopts the ECAPA voiceprint recognition model; the emotion recognition model adopts the large version of the emotion recognition model of emotion2vec.

5. A customizable speech synthesis method according to claim 1, characterized in that: The generated cloned voice is filtered for the first time, and the specific method includes: The voiceprint recognition model and emotion recognition model are used to obtain the voiceprint and emotion embedding vector of each target voice and the corresponding cloned voice. The distance between the voiceprint and emotion embedding vector of the cloned voice and the target voice is calculated. The distance measurement includes Euclidean distance and cosine similarity, and the cloned voices with a longer distance are filtered out.

6. A customizable speech synthesis method according to claim 1, characterized in that: The speech recognition model converts the obtained recognition text into phonemes, so as to facilitate comparison and filtering with the real training text phonemes in the training text set.

7. A customizable speech synthesis method according to claim 1, characterized in that: The method of using the speech recognition model to count the error rate and erroneous phonemes of the speech includes: The speech recognition model is used to recognize the speech of the test text set generated by the speech synthesis model, and the insertion errors, replacement errors, and deletion errors of the recognized text compared with the real test text in the test text set are counted, and the word error rate is obtained; at the same time, the erroneous phonemes and the corresponding error frequencies are recorded, and the erroneous phoneme set is output after sorting by error frequency.

8. A customizable speech synthesis method according to claim 1, characterized in that: Fine-tune the trained speech synthesis model. The number of fine-tuning iterations is determined by the loss curve during training and the sampling test of the inference audio.

9. A customizable speech synthesis device, characterized in that: include: The text generation module uses a large language model to generate training text sets and test text sets; The voice cloning module uses a large voice model to clone the timbre and emotion of the target voice and obtain the cloned voice corresponding to the training text set; The data screening module performs the first filtering and speech recognition on the cloned voice according to the voiceprint and emotion distance between the cloned voice and the target voice, and performs the second filtering on the cloned voice with pronunciation errors through the speech recognition results to obtain the training set; Model training module, which uses the training set to train the speech synthesis model; The enhancement module uses the speech recognition model to count the error rate and erroneous phonemes of the speech synthesis model inference results, and fine-tunes the speech synthesis model by constructing enhanced training sets multiple times to make the speech synthesis model stable.

10. An electronic device, characterized in that: include: A processor and a memory, wherein the processor uses a computer program stored in the memory to implement a customizable speech synthesis method as described in any one of claims 1-8.

Images