TTS audio generation system and method based on sound cloning

Abstract

The invention discloses a TTS audio generation system and method based on sound cloning, the system comprises a text splitting unit, a semantic generation unit, a voice generation unit and a quality verification unit, the text splitting unit is used for splitting an input text into a plurality of text segments; the semantic generation unit is used for converting the text obtained by splitting into semantic contents of different styles; the voice generation unit is used for generating to-be-output audio which is similar to the reference audio and has the content conforming to the semantic content based on the voice features of the reference audio and the semantic content; and the quality checking unit is used for checking the accuracy of the audio to be output according to the reference. According to the method, dynamic verification processing is embedded in the TTS generation process, the matching degree of the generated audio and the original text is compared sentence by sentence, and correction logic can be triggered immediately if missing words are found, so that the accuracy is greatly improved, and the speed and the efficiency are not influenced.

Description

Technical Field

[0001] This application relates to the field of speech technology, and in particular to a TTS audio generation system and method based on voice cloning. Background Technology

[0002] Speech technology involves not only difficult but also very complex technologies. It is generally based on generating text from sound cloning, and then synthesizing various speech sounds from the generated text. It involves multiple steps, and each step may introduce errors.

[0003] One of the more difficult problems to solve is that voice cloning and speech synthesis systems often suffer from missing words, extra words, elision, or pronunciation abnormalities. The reason why these problems are difficult to solve is mainly due to the following aspects:

[0004] Front-end text processing includes errors such as word segmentation and part-of-speech tagging, handling of polyphonic characters / special words, prediction errors of rare and unregistered words, and punctuation and prosody.

[0005] The problem with acoustic models is that most current models are built on neural networks. However, neural network models are essentially "probabilistic generative models" rather than deterministic rule systems. Therefore, the audio predicted by neural networks based on the input text during training, analysis, and inference has a certain degree of uncertainty, which can lead to errors. Summary of the Invention

[0006] This application addresses the technical problems existing in the prior art by providing a TTS audio generation system based on sound cloning, comprising a text segmentation unit, a semantic generation unit, a speech generation unit, and a quality verification unit, wherein:

[0007] The text splitting unit is used to split the input text into several segments;

[0008] The semantic generation unit is used to transform the split text into semantic content of different styles;

[0009] The speech generation unit is used to generate an output audio that is similar to the reference audio and whose content conforms to the semantic content, based on the sound features of the reference audio and the semantic content.

[0010] The quality verification unit is used to verify the accuracy of the audio to be output based on a reference.

[0011] Furthermore, the accuracy of the audio to be output is verified as follows:

[0012] Input the audio to be output and its corresponding semantic content into the validation model to generate a word timeline of the audio to be output.

[0013] Analyze the timeline of each word to determine if there are any missing words. If a missing word is found, regenerate the audio for that segment of text and record the number of retries.

[0014] If no words are missing, record the result and continue to the next round of audio generation.

[0015] Furthermore, it also includes an anti-blocking unit, which counts the regenerated audio. If no missing words are obtained after generating the same text a preset number of times, the audio with the fewest missing words generated previously will be used as the output result.

[0016] Furthermore, the process of transforming the split text into semantic content of different styles includes:

[0017] Text preprocessing: After standardizing the text, each token is converted into a multi-dimensional word vector;

[0018] Contextual semantic encoding: The word vectors are input into the context encoder to generate a context-aware representation of each token, resulting in a semantic vector;

[0019] Style fusion: The prosodic and stylistic features of the reference audio are extracted using a style encoder to obtain a style vector, which is then fused with the semantic vector to obtain semantic content of different styles.

[0020] This invention also provides a TTS audio generation method based on sound cloning, applied to the TTS audio generation system based on sound cloning, comprising the following steps:

[0021] S1: The text splitting unit splits the input text into several segments;

[0022] S2: The semantic generation unit transforms the split text into semantic content of different styles;

[0023] S3: The speech generation unit generates an output audio that is similar to the reference audio and whose content conforms to the semantic content, based on the sound features of the reference audio and the semantic content.

[0024] S4: The quality verification unit performs an accuracy verification on the audio to be output based on a reference.

[0025] Furthermore, the accuracy of the audio to be output is verified as follows:

[0026] Input the audio to be output and its corresponding semantic content into the validation model to generate a word timeline of the audio to be output.

[0027] Analyze the timeline of each word to determine if there are any missing words. If a missing word is found, regenerate the audio for that segment of text and record the number of retries.

[0028] If no words are missing, record the result and continue to the next round of audio generation.

[0029] Furthermore, it also includes an anti-blocking unit that counts the regenerated audio. If no audio without missing characters is obtained after generating the same text a preset number of times, the audio with the fewest missing characters generated previously will be used as the output result.

[0030] Furthermore, the process of transforming the split text into semantic content of different styles includes:

[0031] Text preprocessing: After standardizing the text, each token is converted into a multi-dimensional word vector;

[0032] Contextual semantic encoding: The word vectors are input into the context encoder to generate a context-aware representation of each token, resulting in a semantic vector;

[0033] Style fusion: Use a style encoder to extract the prosody and style features of preset style audio or preset style tags to obtain style vectors, and fuse them with semantic vectors to obtain semantic content of different styles.

[0034] This application also provides a storage medium storing a computer program, which is executed by a processor to produce the TTS audio generation method based on sound cloning.

[0035] This application also provides an electronic device including a processor, a storage medium, and a bus. The storage medium stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the storage medium via the bus, and the processor executes the machine-readable instructions to perform the TTS audio generation method based on sound cloning as described above.

[0036] In practical applications, the modules described in the systems and methods disclosed in this application can be deployed on a single target server, or each module can be deployed independently on different target servers. In particular, as needed, to provide more powerful computing capabilities, the modules can also be deployed on a cluster of target servers.

[0037] Therefore, the technical effect achieved by the technical approach adopted in this application is as follows: the present invention embeds dynamic verification processing in the TTS generation process, compares the matching degree between the generated audio and the original text sentence by sentence, and if a missing character is found, the correction logic can be triggered immediately, thereby greatly improving the accuracy without affecting the speed and efficiency.

[0038] To provide a clearer and more comprehensive understanding of this application, the specific embodiments of this application will be described in detail below with reference to the accompanying drawings. Attached Figure Description

[0039] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0040] Figure 1 This is a schematic diagram of the audio generation system based on sound cloning according to an embodiment of this application.

[0041] Figure 2 This is a schematic diagram of the audio generation system based on sound cloning according to an embodiment of this application.

[0042] Figure 3 This is a schematic diagram of the verification process in an embodiment of this application. Detailed Implementation

[0043] Existing methods are typically optimized separately during the cloning or generation phases, lacking real-time calibration of the combined output of both. See [link to relevant documentation]. Figure 1 The technical solution of this application is a TTS audio generation system based on sound cloning, including a text segmentation unit, a semantic generation unit, a speech generation unit, and a quality verification unit, wherein:

[0044] The text splitting unit is used to split the input text into several segments;

[0045] The semantic generation unit is used to transform the split text into semantic content of different styles;

[0046] The speech generation unit is used to generate an output audio that is similar to the reference audio and whose content conforms to the semantic content, based on the sound features of the reference audio and the semantic content.

[0047] The quality verification unit is used to verify the accuracy of the audio to be output based on a reference.

[0048] The technical solution of this application is described below with reference to various preferred methods, procedures and implementation methods.

[0049] Please see Figure 2 , Figure 2 This is a schematic diagram of the audio generation system based on sound cloning according to an embodiment of this application.

[0050] S1: The text splitting unit splits the input text into several segments;

[0051] This application considers the various advantages and disadvantages of TTS models. When processing long texts, TTS models have longer inference time and higher GPU memory consumption, but better sentence coherence. When processing short texts, inference time is shorter and GPU memory consumption is lower, but sentence coherence is slightly inferior. To balance GPU memory usage, inference time, and sentence coherence, this application splits the input text into several segments, for example, 50 characters per segment, and inputs each segment into the TTS model.

[0052] To balance memory usage, inference time, and sentence coherence, this application's text segmentation unit will input the text T to be segmented. N The text is split into N smaller paragraphs, and the set of these paragraphs is denoted as T. N =[T_0,T_1,T_2,...,T_{N-1}].

[0053] Then initialize the outer loop counter, setting r = 0. The variable r is used to record the number of retries for the current text segment T_n.

[0054] S2: The semantic generation unit transforms the split text into semantic content of different styles.

[0055] The purpose of this step is to transform the text content into semantic information that the model can recognize, in order to support subsequent audio cloning and generation. This enables the generation of diverse semantic content based on different speaking styles, thereby enhancing the richness of the output audio.

[0056] In this step, a reference audio file (I) for the target timbre is provided. The sonic characteristics of this audio file will be cloned into the final output throughout the process.

[0057] As a preferred implementation, converting the split text into semantic content of different styles includes:

[0058] Text preprocessing: After standardizing the text, each token is converted into a multi-dimensional word vector;

[0059] Contextual semantic encoding: The word vectors are input into the context encoder to generate a context-aware representation of each token, resulting in a semantic vector;

[0060] Style fusion: Use a style encoder to extract the prosody and style features of preset style audio or preset style tags to obtain style vectors, and fuse them with semantic vectors to obtain semantic content of different styles.

[0061] S3: The speech generation unit generates an output audio that is similar to the reference audio and whose content conforms to the semantic content, based on the sound features of the reference audio and the semantic content.

[0062] The speech generation unit in this step receives two inputs: one is the semantic content generated in the previous step; the other is the voice characteristics of the reference audio I, and then synthesizes a piece of audio to be output.

[0063] S4: The quality verification unit verifies the accuracy of the audio to be output according to the reference.

[0064] Please refer to Figure 3 , after generating the audio (that is, synthesizing a piece of audio to be output, O in the flowchart N ), and its corresponding original text T N are sent into the verification model, and based on T N to verify O N to obtain the word timeline L. That is, the model will perform speech recognition and generate a word-level timeline, and judge whether there are missing words in the generated audio by comparison. Specifically:

[0065] Input the audio to be output and its corresponding semantic content into the verification model to generate the word timeline of the audio to be output;

[0066] Analyze the timeline of each word to judge whether there are missing words. If missing words are found, regenerate the audio of this text segment and record the number of retries;

[0067] If there are no missing words, it means the verification passes. Record the result and continue the next round of audio generation operation.

[0068] In addition, considering the issue of efficiency, during the verification process, to prevent continuous repeated verification, this application is equipped with an anti-blocking unit to count the regenerated audio. If the audio without missing words is not obtained after the same text segment is generated a preset number of times, the audio with the fewest missing words generated previously is used as the output result.

[0069] Take an example to illustrate: The model can generate the corresponding timeline L of the text in T N and O N in the audio of O N according to the input content T N , and then judge whether there are missing words according to the duration of each word in the timeline L. For example, the content of the timeline L is: [{word: I, start: 0, end: 0.5}, {word: love, start: 0.6, end: 1.1}, {word: in, start: 1.2, end: 1.7}, {word: China, start: 1.8, end: 1.8}]. Among them, the duration of the word "China" is 0, and it can be judged that it is missing a word during TTS. At this time, the semantics and audio of this sentence need to be regenerated and verified again.

[0070] This application also provides a storage medium storing a computer program that is executed by a processor as described in the TTS audio generation system based on sound cloning.

[0071] This application also provides an electronic device, including: a processor, a storage medium, and a bus. The storage medium stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the storage medium via the bus, and the processor executes the machine-readable instructions to perform the TTS-based audio generation system as described above.

[0072] It should be noted that those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, which may include, but is not limited to, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0073] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A TTS (Text-to-Speech) audio generation system based on sound cloning, characterized in that, It includes a text segmentation unit, a semantic generation unit, a speech generation unit, and a quality verification unit, among which: The text splitting unit is used to split the input text into several segments; The semantic generation unit is used to transform the split text into semantic content of different styles; The speech generation unit is used to generate an output audio that is similar to the reference audio and whose content conforms to the semantic content, based on the sound features of the reference audio and the semantic content. The quality verification unit is used to verify the accuracy of the audio to be output based on a reference.

2. The TTS audio generation system based on sound cloning as described in claim 1, characterized in that, The accuracy of the audio to be output is verified as follows: Input the audio to be output and its corresponding semantic content into the validation model to generate a word timeline of the audio to be output. Analyze the timeline of each word to determine if there are any missing words. If a missing word is found, regenerate the audio for that segment of text and record the number of retries. If no words are missing, record the result and continue to the next round of audio generation.

3. The TTS audio generation system based on sound cloning as described in claim 1, characterized in that, It also includes an anti-blocking unit, which counts the regenerated audio. If the same text is generated a preset number of times without obtaining an output audio without missing characters, the audio with the fewest missing characters generated previously will be used as the output result.

4. The TTS audio generation system based on sound cloning as described in claim 1, characterized in that, The process of converting the split text into semantic content of different styles includes: Text preprocessing: After standardizing the text, each token is converted into a multi-dimensional word vector; Contextual semantic encoding: The word vectors are input into the context encoder to generate a context-aware representation of each token, resulting in a semantic vector; Style fusion: The prosodic and stylistic features of the reference audio are extracted using a style encoder to obtain a style vector, which is then fused with the semantic vector to obtain semantic content of different styles.

5. A method for generating audio via TTS based on sound cloning, applied to the aforementioned TTS audio generation system based on sound cloning, characterized in that, Includes the following steps: S1: The text splitting unit splits the input text into several segments; S2: The semantic generation unit transforms the split text into semantic content of different styles; S3: The speech generation unit generates an output audio that is similar to the reference audio and whose content conforms to the semantic content, based on the sound features of the reference audio and the semantic content. S4: The quality verification unit performs an accuracy verification on the audio to be output based on a reference.

6. The TTS audio generation method based on sound cloning as described in claim 5, characterized in that, The accuracy of the audio to be output is verified as follows: Input the audio to be output and its corresponding semantic content into the validation model to generate a word timeline of the audio to be output. Analyze the timeline of each word to determine if there are any missing words. If a missing word is found, regenerate the audio for that segment of text and record the number of retries. If no words are missing, record the result and continue to the next round of audio generation.

7. The TTS audio generation method based on sound cloning as described in claim 5, characterized in that, It also includes an anti-blocking unit that counts the regenerated audio. If no audio without missing characters is obtained after generating the same text a preset number of times, the audio with the fewest missing characters generated previously will be used as the output result.

8. The method for generating audio systems based on sound cloning according to claim 5, characterized in that, The process of converting the split text into semantic content of different styles includes: Text preprocessing: After standardizing the text, each token is converted into a multi-dimensional word vector; Contextual semantic encoding: The word vectors are input into the context encoder to generate a context-aware representation of each token, resulting in a semantic vector; Style fusion: Use a style encoder to extract the prosody and style features of preset style audio or preset style tags to obtain style vectors, and fuse them with semantic vectors to obtain semantic content of different styles.

9. A storage medium storing a computer program, the computer program being executed by a processor to perform the TTS-based audio generation method according to any one of claims 5-8.

10. An electronic device, characterized in that, include: The device includes a processor, a storage medium, and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the storage medium via the bus, and the processor executes the machine-readable instructions to perform the TTS audio generation method based on sound cloning as described in any one of claims 5-8.

Images