A prompt word-based speech synthesis system, training method and inference method
Improvements to the global style acquisition module and the speech synthesis module have solved the problem that the TTS system cannot control speech style based on prompt words, achieving high-quality speech synthesis and cloning with precise style control and strong adaptability.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- 镁佳(北京)科技有限公司
- Filing Date
- 2024-07-16
- Publication Date
- 2026-04-28
AI Technical Summary
Existing TTS systems cannot completely break free from the specification of speaker identifiers and cannot control the style of synthesized speech based on input prompts, resulting in inaccurate speech synthesis.
A global style acquisition module and a speech synthesis module are adopted. The speaker style vector is extracted through the source style prompting module and the target style prompting module. Combined with the speech synthesis module, the target audio waveform is generated or the speech is cloned. The VITS architecture is improved by adding a global style fusion layer and a target style prompting module for feature extraction and optimization.
It achieves the generation of speech with prompting style based on text description, can perform speech cloning, and does not require specifying a unique speaker identifier. The synthesized speech has high quality and precise style control.
Smart Images
Figure CN118762685B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of speech synthesis technology, specifically to a speech synthesis system, training method, and inference method based on prompt words. Background Technology
[0002] Text-to-speech (TTS) is the task of synthesizing text input into speech waveforms. In recent years, end-to-end TTS has become very popular due to its naturalness and high quality. Among architectures such as Wave-Tacotron, FastSpeech2, and EfficientTTS, the VITS proposal has made significant improvements by employing conditional variational autoencoders, normalized streams, and high-fidelity adversarial training. PITS achieves tone control capabilities by adding a yinggram encoder and decoder to VITS, generating emotionally rich, high-quality speech without the need for annotated tone labels.
[0003] While existing TTS systems borrow from text representations for synthesis, they cannot completely break free from the specification of speaker identifiers and cannot control the style of synthesized speech based on input prompts, thus failing to accurately control the speech synthesis and cloning of the TTS system. Summary of the Invention
[0004] In view of this, the present invention provides a speech synthesis system, training method and inference method based on prompt words, to solve the technical problem that existing TTS systems cannot control the style of synthesized speech based on input prompt words, thereby accurately controlling the speech synthesis and cloning of the TTS system.
[0005] In a first aspect, the present invention provides a speech synthesis system based on prompt words, including a global style acquisition module and a speech synthesis module; the global style acquisition module includes a source style prompt module and a target style prompt module, the source style prompt module is used to extract style from the spectrogram of the input audio to obtain a source speaker style vector, and the target style prompt module is used to extract features based on the input prompt words to obtain a target speaker style vector; the speech synthesis module is used to generate a target audio waveform by performing speech synthesis based on the input phonemes and the target speaker style vector, or to generate a target audio waveform by performing speech cloning based on the input phonemes, the source speaker style vector, and the target speaker style vector.
[0006] The speech synthesis system based on prompt words in this embodiment of the invention extracts style from the spectrogram of the input audio using a source style prompt module to obtain a source speaker style vector. The target style prompt module extracts features based on the input prompt words to obtain a target speaker style vector. The speech synthesis module generates a target audio waveform by synthesizing speech based on the input phonemes and the target speaker style vector, or by cloning speech based on the input phonemes, the source speaker style vector, and the target speaker style vector. This allows style information to be integrated into the speech synthesis module, eliminating the need to specify a unique speaker identifier. Speech with a prompt style can be generated simply based on the text description, and speech cloning can be achieved.
[0007] Optionally, the speech synthesis module includes a text encoding layer, a guard layer, a style vector representation layer, a prosody prediction layer, a monotonic alignment search layer, a posterior probability layer, a flow layer, an inverse flow layer, and a decoding layer; the text encoding layer is used to extract the text latent vectors of the input phonemes; the guard layer is used to obtain Gaussian distribution parameters based on the text latent vectors; the style vector representation layer is used to receive the source speaker style vector and the target speaker style vector input from the global style acquisition module, and input the target speaker style vector to the prosody prediction layer, the inverse flow layer, and the decoding layer, and input the source speaker style vector to the flow layer and the posterior probability layer; the prosody prediction layer is used to generate a prosody persistence vector based on the text latent vector and the target speaker style vector; the monotonic alignment search layer is used to... The Gaussian distribution parameters and the prosody persistence vector are aligned with the text latent vector to obtain the speech synthesis latent layer vector; the posterior probability layer is used to generate a speech clone source to be decoded vector based on the spectrogram of the audio generated by the third-party audio synthesis model and the source speaker style vector; the flow layer is used to perform a flow transform on the speech clone source to be decoded vector in combination with the source speaker style vector to obtain the speech clone source latent layer vector; the inverse flow layer is used to perform an inverse flow transform on the speech synthesis latent layer vector in combination with the target speaker style vector to obtain the speech synthesis to be decoded vector, and is also used to perform a flow transform on the speech clone source latent layer vector in combination with the target speaker style vector to obtain the speech clone to be decoded vector; the decoding layer is used to decode the speech synthesis to be decoded vector or the speech clone to be decoded vector to obtain the target audio waveform.
[0008] The speech synthesis module in this method is an improvement on the VITS architecture, which has the advantage of high-quality synthesized speech.
[0009] Optionally, the source style cues module includes a reference audio encoding layer, an attention layer, and a global style fusion layer; the reference audio encoding layer is used to encode a reference audio vector based on the spectrogram of audio generated by a third-party audio synthesis model; the attention layer is used to perform feature enhancement on the reference audio vector; and the global style fusion layer is used to perform style fusion on the feature-enhanced reference audio vector to obtain the source speaker style vector.
[0010] By extracting source speaker style vectors through a reference audio coding layer, an attention layer, and a global style fusion layer, the system can extract and enhance the style features of the reference audio, providing rich style information for speech synthesis.
[0011] Optionally, the target style prompting module adopts a BERT structure, including a Transformer block, a pooling layer, and an adaptive layer; the Transformer block is used to learn the input prompt words to obtain prompt word embedding representations; the pooling layer is used to perform pooling operations on the prompt word embedding representations; the adaptive layer is used to adaptively optimize the pooled prompt word embedding representations to obtain the target speaker style vector.
[0012] Using Transformer blocks for deep learning of cue words improves the understanding of their semantics. Optimizing features through pooling and adaptive layers enhances the adaptability and accuracy of the target speaker's style vector.
[0013] Optionally, the global style acquisition module further includes a multi-task classification layer, which is used to match corresponding timbre tags for the source speaker style vector and the target speaker style vector.
[0014] By matching different timbre tags, the system can generate speech with different timbre features, thus improving the diversity of speech synthesis.
[0015] Secondly, the present invention provides a training method for a speech synthesis system, applied to the prompt-based speech synthesis system described in the first aspect, comprising: sequentially inputting the spectrogram of audio into a reference audio coding layer, an attention layer, and a global style fusion layer to obtain a source speaker style vector; sequentially inputting prompt words into a Transformer block, a pooling layer, and an adaptive layer to obtain a target speaker style vector; inputting both the source speaker style vector and the target speaker style vector into a multi-task classification layer, and combining the timbre labels in the multi-task classification layer, performing KL loss calculation based on the sum of the source speaker style vector and the target speaker style vector, and optimizing the model parameters of the source style prompt module and the target style prompt module based on the KL loss calculation result; and inputting the vector output by the attention layer as a style vector into the speech synthesis module to train the model parameters of the speech synthesis module.
[0016] Thirdly, the present invention provides a speech synthesis system inference method, applied to the prompt-based speech synthesis system described in the first aspect, including a speech synthesis inference mode and a speech cloning inference mode, wherein: the speech synthesis inference mode includes: sequentially inputting phonemes into a text encoding layer, a guard layer, and a monotonic alignment search layer to obtain a speech synthesis hidden layer vector; sequentially inputting prompt words into a Transformer block, a pooling layer, and an adaptive layer to obtain a target speaker style vector, and inputting the target speaker style vector into a decoding layer, a reverse flow layer, and a prosody prediction layer through a style vector representation layer; performing a reverse flow transformation on the speech synthesis hidden layer vector through the reverse flow layer in conjunction with the target speaker style vector to obtain a speech synthesis vector to be decoded; and inputting the speech synthesis vector to be decoded into the decoding layer to obtain a target audio waveform. The speech cloning inference mode includes: inputting phonemes into a third-party audio synthesis model to obtain a direct spectrogram of the audio; sequentially inputting the direct spectrogram into a reference audio coding layer, an attention layer, and a global style fusion layer to obtain a source speaker style vector; inputting the direct spectrogram and the source speaker style vector into a posterior probability layer to generate a speech clone source to be decoded vector; inputting the speech clone source to be decoded vector and the source speaker style vector into a streaming layer for streaming transformation to obtain a speech clone source hidden layer vector; sequentially inputting cue words into a Transformer block, a pooling layer, and an adaptive layer to obtain a target speaker style vector; inputting the speech clone source hidden layer vector and the target speaker style vector into an inverse streaming layer for inverse streaming transformation to obtain a speech clone to be decoded vector; and inputting the speech clone to be decoded vector into a decoding layer to decode to obtain a target audio waveform.
[0017] Fourthly, the present invention provides a computer device, comprising: a memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the computer instructions to perform the speech synthesis system training method described in the second aspect or the speech synthesis system inference method described in the third aspect.
[0018] Fifthly, the present invention provides a computer-readable storage medium storing computer instructions for causing a computer to execute the speech synthesis system training method described in the second aspect or the speech synthesis system inference method described in the third aspect.
[0019] In a sixth aspect, the present invention provides a computer program product, including computer instructions, which are used to cause a computer to execute the speech synthesis system training method described in the second aspect or the speech synthesis system inference method described in the third aspect. Attached Figure Description
[0020] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.
[0021] Figure 1 This is a structural block diagram of a speech synthesis system based on prompt words according to an embodiment of the present invention;
[0022] Figure 2 This is a schematic diagram of the training process of the prompt-based speech synthesis system according to an embodiment of the present invention;
[0023] Figure 3 This is a schematic diagram of the speech synthesis inference process of the speech synthesis system based on prompt words according to an embodiment of the present invention;
[0024] Figure 4 This is a schematic diagram of the speech cloning inference process of the speech synthesis system based on prompt words according to an embodiment of the present invention;
[0025] Figure 5 This is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. Detailed Implementation
[0026] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0027] Integrating style embeddings into TTS has been a prominent research area. Global Style Tokens (GSTs) employ unsupervised training without explicit labels and add soft, interpretable labels to the hidden layers of Tacotron. Finally, style transfer from a single reference audio clip is achieved. Researchers precisely utilize the variance adapter by incorporating GST into FastSpeech2 and adding sentiment-labeled layers or multi-label binary vectors to complete style embedding extraction. Cue-based learning can be summarized as pre-training, cueing, and prediction. Cue-controlled TTS aims to guide synthesis through textual descriptions, thereby further expanding the diversity of synthesized timbres.
[0028] This invention extends the architecture of VITS by injecting style embedding through TTS. Furthermore, the entire model we construct does not require a speaker ID; it can synthesize the speech of any speaker based on text using only prompts and obtain diverse, high-quality audio consistent with the prompts. At the same time, it can perform high-quality speech cloning based on prompt words from other TTS systems.
[0029] This embodiment provides a speech synthesis system based on prompt words, such as Figure 1 As shown, it includes a global style acquisition module and a speech synthesis module;
[0030] The global style acquisition module includes a source style prompting module and a target style prompting module. The source style prompting module is used to extract the style from the direct spectrum of the input audio to obtain the source speaker style vector. The target style prompting module is used to extract features based on the input prompt words to obtain the target speaker style vector.
[0031] The speech synthesis module is used to generate a target audio waveform by synthesizing speech based on the input phonemes and the target speaker style vector, or to generate a target audio waveform by cloning speech based on the input phonemes, the source speaker style vector, and the target speaker style vector.
[0032] Specifically, the prompt words are information describing the speech style, and can be prompt text or prompt speech.
[0033] The speech synthesis system based on prompt words in this invention extracts style from the spectrogram of the input audio by a source style prompt module to obtain a source speaker style vector, and extracts features from the input prompt words to obtain a target speaker style vector. The speech synthesis module generates a target audio waveform by synthesizing speech based on the input phonemes and the target speaker style vector, or by cloning speech based on the input phonemes, the source speaker style vector, and the target speaker style vector. Style information can be integrated into the speech synthesis module, eliminating the need to specify a unique speaker identifier. Speech with prompt style can be generated based solely on the text description, and speech cloning can be achieved.
[0034] The speech synthesis module of this invention is based on the improved VITS architecture, and the trained global style acquisition module replaces part of the speaker vector representation (speakerembedding) and freezes its parameters.
[0035] Specifically, such as Figure 2 , Figure 3 and Figure 4 As shown, the speech synthesis module includes a text encoder, a projection layer, a style embedding layer, a duration prediction layer, a monotonic alignment search layer, a posterior encoder, a flow layer, and a flow-inverse layer. -1 ) and decoding layer (Decoder);
[0036] The text encoding layer is used to extract the text latent vector h of the input phoneme. test ;
[0037] The protection layer is used for text-based implicit vector h test The Gaussian distribution parameters are obtained, including the Gaussian distribution mean statistic μ and the Gaussian distribution variance statistic δ.
[0038] The style vector representation layer is used to receive the source speaker style vector S and the target speaker style vector from the global style acquisition module. and target speaker style vector The source speaker style vector S is input to the phonology prediction layer, the inverse flow layer, and the decoding layer, and then input to the flow layer and the posterior probability layer.
[0039] The phonology prediction layer is used to predict the text based on the latent vector h. test Target speaker style vector Generate the musical duration vector d;
[0040] The monotonic alignment search layer is used to align the text latent vector h according to the Gaussian distribution parameters and the phonological duration vector d. test The hidden layer vector f(z) for speech synthesis is obtained;
[0041] The posterior probability layer is used to generate the speech clone source vector z(src) to be decoded based on the linear spectra of the audio generated by the third-party audio synthesis model and the source speaker style vector S.
[0042] The flow layer is used to combine the source speaker style vector S with the speech clone source to be decoded vector z(src) to obtain the speech clone source hidden layer vector f(z(src));
[0043] The inverse layer is used to combine the target speaker's style vector. The speech synthesis hidden layer vector f(z) is inversely transformed to obtain the speech synthesis decoding vector z, and a vector for combining the target speaker's style vector. The speech clone source hidden layer vector f(z(src)) is transformed by stream transformation to obtain the speech clone to be decoded vector z(tgt);
[0044] The decoding layer is used to decode the speech synthesis vector z or the speech clone vector z(tgt) to obtain the target audio waveform.
[0045] Furthermore, the source style cueing module includes a reference encoder, an attention layer, and a global style fusion layer (GFL);
[0046] The reference audio coding layer is used to encode the direct spectrum of audio generated by a third-party audio synthesis model to obtain a reference audio vector;
[0047] Attention layers are used to enhance the features of the reference audio vector;
[0048] The global style fusion layer is used to perform style fusion on the feature-enhanced reference audio vector to obtain the source speaker style vector.
[0049] Specifically, the source style cueing module is mainly used to extract features from the input audio. The global style fusion layer is a layer that integrates multiple aspects of information, including a fully connected layer, a randomly deactivated layer, a residual connection, and a normalization layer connected in sequence. Because its parameters can be trained and adjusted, it can achieve the following three objectives: 1. Since the source style cueing module is frozen, during fine-tuning, training the parameters of the global style fusion layer better adapts the GST from the source style cueing module to the entire model. 2. In subsequent training, the global style fusion layer can more easily align the source speaker style vector and the target speaker vector on the KL divergence, thereby achieving the goal of controlling the speaker style using the target speaker vector. 3. Arbitrary human voice synthesis can be performed subsequently using random cue words.
[0050] Furthermore, the target style hint module adopts a BERT structure, including a Transformer block (Transformer), a pooling layer (Pooler), and an adaptive layer (Adaptor);
[0051] The Transformer block is used to learn the input prompt words and obtain the prompt word embedding representation;
[0052] The pooling layer is used to perform pooling operations on the prompt word embedding representation;
[0053] The adaptive layer is used to adaptively optimize the pooled cue word embedding representation to obtain the target speaker style vector.
[0054] The target style prompting module is mainly used to extract features from prompt words. The target style prompting module is fine-tuned using an end-to-end approach. In addition, the global style acquisition module has added a multi-classification layer, which performs hard label optimization on speech styles such as gender, age, speech rate, volume, and pitch.
[0055] The multi-task classification layer is used to match corresponding timbre labels to the source speaker style vector and the target speaker style vector. Timbre labels include gender, age, speech rate, volume, pitch, etc.
[0056] In this embodiment of the invention, the global style fusion layer can enhance text control by aligning with the output of the target style cue module. Furthermore, a multi-task classification layer is added to specify volume, speed, gender, age, pitch, and style, making the synthesis more accurate.
[0057] Furthermore, such as Figure 2 As shown, the training process of the prompt-based speech synthesis system in the above embodiment is as follows:
[0058] Step S101: The direct spectrogram of the audio is sequentially input into the reference audio coding layer, the attention layer, and the global style fusion layer to obtain the source speaker style vector;
[0059] Step S102: Input the prompt words sequentially into the Transformer block, pooling layer and adaptive layer to obtain the target speaker style vector;
[0060] Step S103: Input both the source speaker style vector and the target speaker style vector into the multi-task classification layer, and combine the timbre labels in the multi-task classification layer to calculate the KL loss based on the source speaker style vector and the target speaker style vector. Optimize the model parameters of the source style prompting module and the target style prompting module based on the KL loss calculation results.
[0061] Step S104: Input the vector output by the attention layer as a style vector into the speech synthesis module to train the model parameters of the speech synthesis module.
[0062] By performing KL loss calculation, the model parameters of the source style cue module and the target style cue module are optimized based on the KL loss calculation results. The vector output by the attention layer is used as the style vector input to the speech synthesis module to train the model parameters of the speech synthesis module. Hard label optimization can be performed on gender, age, speech rate, volume, pitch, etc. to achieve alignment of the KL divergence between the source speaker style vector and the target speaker style vector. Further constraints are applied through the loss task to make the source speaker style vector and the target speaker style vector highly consistent at the latent space level, so that the target synthesized timbre is the same as or similar to the text description.
[0063] Furthermore, such as Figure 3 As shown and Figure 4 As shown, the reasoning process of the prompt-based speech synthesis system in the above embodiments includes a speech synthesis reasoning mode and a speech cloning reasoning mode, wherein:
[0064] Speech synthesis inference modes include:
[0065] Step S201: Input the phonemes sequentially into the text encoding layer, the guard layer, and the monotonic alignment search layer to obtain the speech synthesis hidden layer vector;
[0066] Step S202: The prompt words are sequentially input into the Transformer block, pooling layer and adaptive layer to obtain the target speaker style vector, and the target speaker style vector is input into the decoding layer, inverse flow layer and phonology prediction layer through the style vector representation layer.
[0067] Step S203: The speech synthesis hidden layer vector is transformed by inverse transformation of the inverse layer combined with the target speaker style vector to obtain the speech synthesis decoding vector;
[0068] Step S204: Input the speech synthesis vector to be decoded into the decoding layer to obtain the target audio waveform;
[0069] Voice cloning reasoning modes include:
[0070] Step S301: Input the phonemes into a third-party audio synthesis model to obtain the audio spectrogram;
[0071] Step S302: The direct spectrogram is sequentially input into the reference audio coding layer, the attention layer, and the global style fusion layer to obtain the source speaker style vector;
[0072] Step S303: Input the direct spectrogram and the source speaker style vector into the posterior probability layer to generate the speech clone source vector to be decoded;
[0073] Step S304: Input the speech clone source vector to be decoded and the source speaker style vector into the stream layer for stream transformation to obtain the speech clone source hidden layer vector.
[0074] Step S305: Input the prompt words sequentially into the Transformer block, pooling layer, and adaptive layer to obtain the target speaker style vector;
[0075] Step S306: Input the hidden layer vector of the speech clone source and the style vector of the target speaker into the inverse layer for inverse transformation to obtain the speech clone vector to be decoded;
[0076] Step S307: Input the speech clone to be decoded vector into the decoding layer to obtain the target audio waveform.
[0077] As can be seen from the above reasoning process, the prompt-based speech synthesis system of this invention uses the global style fusion layer in the source style prompt module as a bridge to enhance the guidance of the target style prompt module to the speech synthesis module. Through prompt word information, the pitch, speech rate, volume, and other features of the generated speech can be precisely controlled, and classification information can be accurately integrated into the speech synthesis through the target speaker style vector. Furthermore, it not only achieves style control of the generated speech but also enables high-quality speech cloning based on audio generated by a third-party audio synthesis model, according to prompt word information.
[0078] To verify the audio synthesis quality, this embodiment of the invention uses MOS to evaluate the synthesized audio of the model. We use the VITS model as a baseline and pre-train it using DB495RS and DB1 respectively. The trained models are labeled as Vits-DB495RS and Vits-DB1. Simultaneously, we use Vits-DB495RS as a pre-trained model and fine-tune it on DB-1, denoted as Vits-DB1-Restore-Vits-DB495RS. The model trained using the 2.2 Fun-Tune Method is labeled as PromptSyn. Audio synthesis is performed using both audio-prompt and text-prompt inference methods, labeled as PromptSyn-AP and PromptSyn-TP respectively. Finally, we use Vits-DB495RS as the synthesis model and employ the proposed speech cloning scheme for voice transfer, again using both Audio-Prompt-VC and Text-Prompt-VC methods, labeled as PromptSyn-AVC and PromptSyn-TVC respectively. The speech quality evaluation results obtained after using different models for speech synthesis are as follows:
[0079] Model Quality MOS (CI) RealPerson 4.43(±0.06) Vits-DB1 4.08(±0.08) Vits-DB495RS 4.14(±0.08) Vits-DB1-Restore-Vits-DB495RS 4.39(±0.06) PromptVITS-AP 4.17(±0.07) PromptVITS-TP 4.10(±0.08) PromptVITS-AVC 4.27(±0.08) PromptVITS-TVC 4.23(±0.07)
[0080] Comparing the experimental results, using speech as a prompt yields better synthesis results than using text as a prompt. Furthermore, the speech conversion scheme can effectively make the prompt synthesis more stable.
[0081] This invention also verifies whether the synthesized audio from the model is consistent with the prompts. We prepared two test sets in advance: the prompt text and the prompt audio. Verification was performed using both test sets.
[0082] Prompt Text Consistency: We utilize the prompt-based speech synthesis system, namely the PromptVITS model, from this embodiment of the invention to generate synthesized audio from prompt text. The content of the synthesized audio is randomly assigned, ensuring that the language logic is normal and the length is appropriate. Subsequently, MOS (Modular Orientation) is used for evaluation to verify whether the synthesized audio meets the requirements of the text prompt.
[0083] Audio cue consistency: Using the same PromptVITS model, synthesized audio is generated from the cue audio, and the content of the synthesized audio is consistent with the cue audio. Then, MOS (Mutually Exclusive Evaluation) is used for evaluation to verify whether the synthesized audio maintains the same sonic character as the cue audio.
[0084] The consistency verification results are shown in the table below.
[0085] Voice prompts Text prompt Consistent MOS (CI) 4.85(±0.06) 4.46(±0.06)
[0086] It can be seen that regardless of whether text or audio is used as the cue information, the synthesized audio and the cue information maintain a high degree of consistency. Audio cue information exhibits better consistency than text cue information.
[0087] This invention discloses a prompt-based speech synthesis system, a speech generation model based on the VITS framework, referred to as PromptVits. It uses only prompt information to control style and speaker, thereby accurately guiding the speech synthesis module in voice synthesis and cloning. PromptVits utilizes the source speaker style vector and target speaker style vector output by the global style acquisition module to completely replace the speaker identifier, guiding each module in cross-modal speech synthesis.
[0088] The target style suggestion module adopts a BERT architecture, vectorizing the text description using Transformer blocks, pooling layers, and adaptive layers. This vectorization is then aligned with the GST in the latent space, allowing for synthesis control based on the text description. To ensure the accuracy of GST-guided synthesis, a multi-task classification layer is constructed to capture information such as gender, age, and pitch, preventing the synthesized audio from being irrelevant to the description. Experiments show that PromptVits synthesizes high-fidelity audio with high consistency with the prompt content, regardless of whether it's text or audio cues. Furthermore, PromptVits can perform high-quality speech cloning based on prompt information from other speech synthesis models.
[0089] This invention also provides a schematic diagram of the structure of a computer device, such as... Figure 5 As shown, the computer device includes one or more processors 10, memory 20, and interfaces for connecting the components, including high-speed interfaces and low-speed interfaces. The components communicate with each other via different buses and can be mounted on a common motherboard or otherwise installed as needed. The processors can process instructions executed within the computer device, including instructions stored in or on memory to display graphical information of a GUI on external input / output devices (such as display devices coupled to the interfaces). In some alternative implementations, multiple processors and / or multiple buses can be used with multiple memories and multiple memory modules, if desired. Similarly, multiple computer devices can be connected, each providing some of the necessary operations (e.g., as a server array, a group of blade servers, or a multiprocessor system). Figure 5 Take a processor 10 as an example.
[0090] Processor 10 may be a central processing unit, a network processor, or a combination thereof. Processor 10 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The programmable logic device may be a complex programmable logic device (CAMP), a field-programmable gate array (FPGA), a general-purpose array logic (GDA), or any combination thereof.
[0091] The memory 20 stores instructions executable by at least one processor 10 to cause at least one processor 10 to perform the method shown in the above embodiments.
[0092] The memory 20 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the computer device. Furthermore, the memory 20 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some alternative embodiments, the memory 20 may optionally include memory remotely located relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.
[0093] The memory 20 may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as flash memory, hard disk or solid-state drive; the memory 20 may also include a combination of the above types of memory.
[0094] The computer device also includes an input device 30 and an output device 40. The processor 10, memory 20, input device 30, and output device 40 can be connected via a bus or other means. Figure 5 Taking the example of a connection between China and Israel via a bus.
[0095] Input device 30 can receive input numerical or character information, and generate key signal inputs related to user settings and function control of the computer device, such as a touchscreen, keypad, mouse, trackpad, touchpad, joystick, one or more mouse buttons, trackball, joystick, etc. Output device 40 may include display devices, auxiliary lighting devices (e.g., LEDs), and haptic feedback devices (e.g., vibration motors). The aforementioned display devices include, but are not limited to, liquid crystal displays, light-emitting diodes, displays, and plasma displays. In some alternative embodiments, the display device may be a touchscreen.
[0096] This invention also provides a computer-readable storage medium. The methods described above according to embodiments of the invention can be implemented in hardware or firmware, or implemented as computer code that can be recorded on a storage medium, or implemented as computer code downloaded via a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and then stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code, which, when accessed and executed by the computer, processor, or hardware, implements the methods shown in the above embodiments.
[0097] A portion of this invention can be applied as a computer program product, such as computer program instructions, which, when executed by a computer, can invoke or provide the methods and / or technical solutions according to the invention through the operation of the computer. Those skilled in the art will understand that the forms in which computer program instructions exist in a computer-readable medium include, but are not limited to, source files, executable files, installation package files, etc. Correspondingly, the ways in which computer program instructions are executed by a computer include, but are not limited to: the computer directly executing the instructions, or the computer compiling the instructions and then executing the corresponding compiled program, or the computer reading and executing the instructions, or the computer reading and installing the instructions and then executing the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to a computer.
[0098] Although embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention, and all such modifications and variations fall within the scope defined by the appended claims.
Claims
1. A speech synthesis system based on prompt words, characterized in that, This includes a global style acquisition module and a speech synthesis module; The global style acquisition module includes a source style prompting module and a target style prompting module. The source style prompting module is used to extract the style from the direct spectrum of the input audio to obtain the source speaker style vector. The target style prompting module is used to extract features based on the input prompt words to obtain the target speaker style vector. The speech synthesis module is used to generate a target audio waveform by cloning speech based on the input phonemes, the source speaker style vector and the target speaker style vector; The speech synthesis module includes a text encoding layer, a guard layer, a style vector representation layer, a phonology prediction layer, a monotonic alignment search layer, a posterior probability layer, a flow layer, an inverse flow layer, and a decoding layer. The text encoding layer is used to extract the text latent vectors of the input phonemes; The protective layer is used to obtain Gaussian distribution parameters based on the text latent vector; The style vector representation layer is used to receive the source speaker style vector and the target speaker style vector input by the global style acquisition module, and input the target speaker style vector to the phonology prediction layer, the inverse flow layer and the decoding layer, and input the source speaker style vector to the flow layer and the posterior probability layer; The phonology prediction layer is used to generate a phonology persistence vector based on the text latent vector and the target speaker style vector. The monotonic alignment search layer is used to align the text latent vector with the Gaussian distribution parameters and the phonological persistence vector to obtain the speech synthesis latent vector. The posterior probability layer is used to generate a speech clone source to be decoded vector based on the direct spectrum of the audio generated by the third-party audio synthesis model and the source speaker style vector; The flow layer is used to perform a flow transformation on the speech clone source to be decoded vector by combining the source speaker style vector to obtain the speech clone source hidden layer vector. The inverse flow layer is used to perform an inverse flow transformation on the speech synthesis hidden layer vector in combination with the target speaker style vector to obtain the speech synthesis decoding vector, and is used to perform a flow transformation on the speech clone source hidden layer vector in combination with the target speaker style vector to obtain the speech clone decoding vector. The decoding layer is used to decode the speech synthesis vector to be decoded or the speech clone vector to be decoded to obtain the target audio waveform.
2. The speech synthesis system based on prompt words according to claim 1, characterized in that, The source style cues module includes a reference audio coding layer, an attention layer, and a global style fusion layer; The reference audio coding layer is used to encode the direct spectrum of the audio generated by the third-party audio synthesis model to obtain a reference audio vector; The attention layer is used to enhance the features of the reference audio vector; The global style fusion layer is used to perform style fusion on the feature-enhanced reference audio vector to obtain the source speaker style vector.
3. The speech synthesis system based on prompt words according to claim 1, characterized in that, The target style hint module adopts a BERT structure, including a Transformer block, a pooling layer, and an adaptive layer; The Transformer block is used to learn the input prompt words and obtain the prompt word embedding representation; The pooling layer is used to perform pooling operations on the prompt word embedding representation; The adaptive layer is used to adaptively optimize the pooled cue word embedding representation to obtain the target speaker style vector.
4. The speech synthesis system based on prompt words according to claim 1, characterized in that, The global style acquisition module also includes a multi-task classification layer, which is used to match corresponding timbre tags for the source speaker style vector and the target speaker style vector.
5. A training method for a speech synthesis system, characterized in that, The system is applied to the prompt-word-based speech synthesis system as described in any one of claims 1-4, comprising: The audio spectrogram is sequentially input into the reference audio coding layer, the attention layer, and the global style fusion layer to obtain the source speaker style vector; The prompt words are sequentially input into the Transformer block, pooling layer, and adaptive layer to obtain the target speaker style vector; Both the source speaker style vector and the target speaker style vector are input into the multi-task classification layer. Combined with the timbre labels in the multi-task classification layer, KL loss is calculated based on the source speaker style vector and the target speaker style vector. The model parameters of the source style prompting module and the target style prompting module are optimized based on the KL loss calculation results. The vector output by the attention layer is used as a style vector and input into the speech synthesis module to train the model parameters of the speech synthesis module.
6. A reasoning method for a speech synthesis system, characterized in that, Applied to the prompt-based speech synthesis system as described in any one of claims 1-4, comprising a speech synthesis inference mode and a speech cloning inference mode, wherein: The speech synthesis reasoning mode includes: The phonemes are sequentially input into the text encoding layer, the guard layer, and the monotonic alignment search layer to obtain the speech synthesis hidden layer vector; The prompt words are sequentially input into the Transformer block, pooling layer and adaptive layer to obtain the target speaker style vector, and the target speaker style vector is then input into the decoding layer, inverse flow layer and phonology prediction layer through the style vector representation layer. The speech synthesis hidden layer vector is transformed by the inverse transformation of the speech synthesis hidden layer vector in combination with the target speaker style vector to obtain the speech synthesis decoding vector. The speech synthesis vector to be decoded is input into the decoding layer to obtain the target audio waveform; The voice cloning inference mode includes: Phonemes are input into a third-party audio synthesis model to obtain the audio's direct spectrum; The direct spectrogram is sequentially input into the reference audio coding layer, the attention layer, and the global style fusion layer to obtain the source speaker style vector; The direct spectrogram and the source speaker style vector are input into the posterior probability layer to generate the speech clone source vector to be decoded. The speech clone source vector to be decoded and the source speaker style vector are input into the flow layer for flow transformation to obtain the speech clone source hidden layer vector. The prompt words are sequentially input into the Transformer block, pooling layer, and adaptive layer to obtain the target speaker style vector; The hidden layer vector of the speech clone source and the style vector of the target speaker are input into the inverse layer and inverse transformation is performed to obtain the speech clone to be decoded vector; The voice clone vector to be decoded is input into the decoding layer to obtain the target audio waveform.
7. A computer device, characterized in that, include: The system includes a memory and a processor, which are communicatively connected to each other. The memory stores computer instructions, and the processor executes the computer instructions to perform the speech synthesis system training method as described in claim 5 or the speech synthesis system inference method as described in claim 6.
8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing the computer to execute the speech synthesis system training method as described in claim 5 or the speech synthesis system inference method as described in claim 6.
9. A computer program product, characterized in that, Includes computer instructions for causing a computer to perform the speech synthesis system training method of claim 5 or the speech synthesis system inference method of claim 6.