Controllable text-to-speech method based on decoupled multi-modal prompt and chain guide

Abstract

The application provides a controllable text-to-speech method based on decoupled multi-modal cues and chain-based guidance, which comprises the following steps: constructing a unified multi-modal style encoder for mapping style cues from reference audio or descriptive text to a shared embedding space to generate style-conditioned embeddings, and training the multi-modal style encoder; constructing a text-to-speech model based on a latent diffusion transformer, which takes the latent representation of mel-spectrogram as the processing object and receives content text, speaker prosody and speaking style as input conditions; in the inference stage of the text-to-speech model, a chain-based classifier-free guidance mechanism is adopted to realize independent and continuous control of the content, prosody and style of the generated speech by independently adjusting the guidance strength of the content, prosody and style-conditioned embeddings. The independent and continuous control of the content, prosody and style of the generated speech is realized. The training stability and convergence speed of the model are significantly improved.

Description

Technical Field

[0001] This application relates to the field of text-to-speech technology, specifically a controllable text-to-speech method based on decoupled multimodal prompts and chained guidance. Background Technology

[0002] Text-to-speech (TTS) technology aims to convert text into natural and intelligible speech. With the development of deep learning technology, the speech quality generated by modern TTS systems has approached that of real people. Therefore, research focus has gradually shifted to controllable text-to-speech, which allows for independent and flexible manipulation of various attributes of the generated speech (such as emotion, speech rate, and timbre) to meet the needs of a wider range of application scenarios.

[0003] Existing controllable TTS methods primarily control speech style through reference audio, discrete labels, or descriptive text. However, these methods face significant challenges in practical applications. The most significant problem is the frequent entanglement between speaker timbre and speaking style. For example, when using a reference audio to specify speaking style, the timbre information in the reference audio often "leaks" into the generated speech, causing a change in the speaker's identity rather than just their speaking style. Furthermore, most mainstream text-to-speech systems only support single-modal style cues (e.g., only audio or only text), limiting their flexibility and adaptability. While existing research has attempted to address these issues, their solutions are often deeply tied to specific model backbone networks, resulting in poor scalability, or their style cues may contain identity-related clues (such as gender), complicating strict style-timbre separation. Therefore, developing a TTS framework that effectively decouples style and timbre, supports multimodal input, and allows for fine-grained independent control is a pressing technical challenge in the field. Summary of the Invention

[0004] This invention addresses the problems of entangled timbre and style, limited control dimensions, and insufficient flexibility in the current field of controllable text-to-speech.

[0005] To address the aforementioned issues, this invention provides a controllable text-to-speech method, electronic device, and storage medium based on decoupled multimodal prompts and chained guidance.

[0006] In a first aspect, the present invention provides a controllable text-to-speech method based on decoupled multimodal prompts and chained guidance, comprising:

[0007] A unified multimodal style encoder is constructed to map style cues from reference audio or descriptive text to a shared embedding space to generate style conditional embeddings, and the multimodal style encoder is trained.

[0008] A text-to-speech model based on a latent diffusion transformer is constructed. The text-to-speech model takes the latent representation of the Mel spectrum as the processing object and receives the content text, speaker timbre and speaking style as input conditions to train the text-to-speech model.

[0009] In the inference stage of the text-to-speech model, a chain-based classifier-free guidance mechanism is adopted. By independently adjusting the guidance intensity of content, timbre, and style conditional embeddings, independent and continuous control over the content, timbre, and style of the generated speech is achieved.

[0010] Optionally, the unified multimodal style encoder is built based on a contrastive language-audio pre-trained model and trained by combining contrastive learning with multi-task supervision.

[0011] Among them, the contrastive learning loss is used to align the style embeddings of audio and text modalities, and the multi-task supervision loss includes cross-entropy loss for discrete style attributes and mean squared error loss for continuous style attributes.

[0012] Optionally, the training objective function of the multimodal style encoder is:

[0013] ;

[0014] In the formula, and These are the weighting coefficients. To compare learning loss, For cross-entropy loss, This represents the mean square error loss.

[0015] Optionally, the chain-based classifier-free guidance mechanism calculates the final predicted speech velocity field using the following formula:

[0016] ;

[0017] in, This represents the velocity field predicted by the model. This represents the velocity field predicted by the model under specific conditions. It is the hyperparameter of content guidance intensity. It is the hyperparameter of timbre guidance intensity. It is the hyperparameter of style guidance intensity; Indicates content conditions, Indicates style conditions, Indicates timbre conditions.

[0018] Optionally, the chain-based classifier-free guidance mechanism employs a hierarchical conditional dropout strategy during the text-to-speech model training phase, including:

[0019] Discard style conditions with the first probability ;

[0020] If style conditions If discarded, the timbre condition is discarded with the second probability. ;

[0021] When style conditions and timbre conditions When all are discarded, the content is discarded with the third probability. .

[0022] Optionally, it also includes employing a representation alignment strategy to stabilize text-to-speech model training and accelerate convergence, the strategy including:

[0023] The audio encoder of the pre-trained large speech model is used as the teacher model, and its output acoustic-semantic representation is extracted as the teacher representation.

[0024] The student representation of the intermediate layer of the latent diffusion transformer is upsampled and linearly projected to align its dimensions with the teacher representation.

[0025] Representation alignment is achieved by minimizing the cosine similarity loss between the student and teacher representations.

[0026] Optionally, the training process of the text-to-speech model is divided into two independent stages:

[0027] The first stage involves independently training the unified multimodal style encoder.

[0028] In the second stage, the latent diffusion transformer is trained, and the parameters of the trained multimodal style encoder are frozen.

[0029] Optionally, the speech generation process includes:

[0030] Given the text to be synthesized, a reference audio or speaker ID with a specified timbre, and a reference audio or descriptive text with a specified style;

[0031] Each encoder converts the input information into corresponding conditional embeddings.

[0032] Noise is sampled from a standard Gaussian distribution and used as the initial input to the potential diffusion converter;

[0033] In the reasoning process of conditional flow matching, a chain-like classifier-free guidance mechanism is used to predict the velocity field and generate a latent representation of the target Mel spectrum;

[0034] The latent representation of the Mel spectrum is converted into the final speech waveform using a vocoder.

[0035] In a second aspect, embodiments of the present invention provide an electronic device, including a processor, a communication interface, a memory, and a bus, wherein the processor, the communication interface, and the memory communicate with each other through the bus, and the processor can call logical instructions in the memory to execute the steps of the method provided in the first aspect.

[0036] Thirdly, embodiments of the present invention provide a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the controllable text-to-speech method based on decoupled multimodal prompting and chained guidance as described in the first aspect.

[0037] The beneficial effects of the controllable text-to-speech method based on decoupled multimodal prompts and chained guidance of the present invention are as follows:

[0038] This invention enables flexible style specification via both audio and text through a unified multimodal style encoder. The introduced chain-based classifier-free guidance mechanism allows for independent and continuous adjustment of content, timbre, and style dimensions in the text-to-speech domain. Simultaneously, the application of representation alignment strategies significantly improves the model's training stability and convergence speed. Attached Figure Description

[0039] Figure 1 This is a flowchart of a controllable text-to-speech method based on decoupled multimodal prompts and chained guidance in an embodiment of the present invention;

[0040] Figure 2 This is an overall architecture diagram of the controllable text-to-speech method based on decoupled multimodal prompts and chained guidance in an embodiment of the present invention;

[0041] Figure 3 This is a detailed structural diagram of the unified multimodal style encoder in an embodiment of the present invention;

[0042] Figure 4 This is a structural block diagram of the electronic device in an embodiment of the present invention. Detailed Implementation

[0043] To better understand the purpose, technical solution, and advantages of this application, the application is described and explained below in conjunction with the accompanying drawings and embodiments.

[0044] Unless otherwise defined, the technical or scientific terms used in this application shall have the general meaning understood by one of ordinary skill in the art to which this application pertains. Words such as “a,” “an,” “an,” “the,” “the,” and “these” used in this application do not indicate quantitative limitation and may be singular or plural. The terms “comprising,” “including,” “having,” and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that comprises a series of steps or modules (units) is not limited to the listed steps or modules (units) but may include steps or modules (units) not listed, or may include other steps or modules (units) inherent to these processes, methods, products, or devices. Words such as “connected,” “linked,” and “coupled” used in this application are not limited to physical or mechanical connections but may include electrical connections, whether direct or indirect. “Multiple” used in this application refers to two or more. “And / or” describes the relationship between related objects, indicating that three relationships may exist; for example, “A and / or B” can represent: A alone, A and B simultaneously, and B alone. Normally, the character " / " indicates that the objects before and after it are in an "or" relationship. The terms "first," "second," "third," etc., used in this application are merely to distinguish similar objects and do not represent a specific order of objects.

[0045] like Figures 1 to 3 As shown, the controllable text-to-speech method based on decoupled multimodal prompts and chained guidance provided in this embodiment of the invention includes:

[0046] S1. Construct a unified multimodal style encoder to map style cues from reference audio or descriptive text to a shared embedding space to generate style conditional embeddings, and train the multimodal style encoder.

[0047] The unified multimodal style encoder is built on a contrastive language-audio pre-trained model and trained by combining contrastive learning with multi-task supervision.

[0048] Among them, the contrastive learning loss is used to align the style embeddings of audio and text modalities, and the multi-task supervision loss includes cross-entropy loss for discrete style attributes and mean squared error loss for continuous style attributes.

[0049] Specifically, a unified multimodal style encoder (Style-CLAP) is constructed. This encoder, based on the contrastive language-audio pre-trained model (CLAP), maps style cues from reference audio and descriptive text to a shared embedding space. To train this encoder, all input audio is uniformly resampled to 48kHz. The multimodal style encoder is trained using a combination of contrastive learning and multi-task supervision. Contrastive learning loss is used. The style embedding used to align audio and text modalities is calculated using the following formula:

[0050] ;

[0051] in, and These are style embeddings extracted from the audio encoder and text encoder, respectively. Represents cosine similarity. This represents the temperature coefficient. Multi-task supervision is used to enhance the discriminative power of audio style representations, applying cross-entropy loss to discrete style attributes (such as sentiment). Mean squared error loss is applied to continuous style attributes (such as speech rate and energy). The overall training objective function for the multimodal style encoder is:

[0052] ;

[0053] In the formula, and These are the weighting coefficients. To compare learning loss, For cross-entropy loss, This represents the mean square error loss.

[0054] S2. Construct a text-to-speech model based on a latent diffusion transformer. The text-to-speech model takes the latent representation of the Mel spectrum as the processing object and receives the content text, speaker timbre and speaking style as input conditions to train the text-to-speech model.

[0055] Specifically, constructing a text-to-speech model based on a Latent Diffusion Transformer (DiT) can be understood as building a text-to-speech framework based on a Latent Diffusion Transformer (DiT). This framework uses the latent representation of the Mel spectrum as the processing object. In the data preprocessing stage, the input audio is first read using the torchaudio library and uniformly upsampled to 44.1kHz. Subsequently, the audio waveform is converted to a Mel spectrum and downsampled by a factor of 1024 through a pre-trained Mel encoder, finally obtaining latent features with a feature dimension of 40. These latent features serve as the input to the Latent Diffusion Transformer (the core DiT module), responsible for mapping Gaussian noise to the latent representation of the target speech. The model input conditions consist of three parts: content text, speaker timbre, and speaking style, provided by the content encoder, speaker encoder, and style encoder, respectively. In addition, the framework also includes a duration predictor to control phoneme-level alignment based on content and style information.

[0056] S3. In the inference stage of the text-to-speech model, a chain-based classifier-free guidance mechanism is adopted. By independently adjusting the guidance intensity of content, timbre, and style conditional embeddings, independent and continuous control over the content, timbre, and style of the generated speech is achieved.

[0057] A chain-based classifier-free guided inference (cCFG) mechanism is introduced to achieve independent control over content, timbre, and style during the inference phase. This mechanism is implemented through a hierarchical conditional dropout strategy during the training phase, including:

[0058] With the first probability Discard style conditions ;

[0059] If style conditions If discarded, it will be treated with the second probability. Discard timbre conditions ;

[0060] Only when style conditions and timbre conditions Only when all are discarded is the third probability used. Discard content conditions .

[0061] In this embodiment, the first dropout probability is determined based on empirical values ​​during the training process, and the second and third probabilities are both 0.5.

[0062] During inference, the final predicted speech velocity field It is composed of multiple guide necklaces, and its calculation formula is as follows:

[0063] ;

[0064] in, This represents the velocity field predicted by the model. This represents the velocity field predicted by the model under specific conditions. It is the hyperparameter of content guidance intensity. It is the hyperparameter of timbre guidance intensity. It is the hyperparameter of style guidance intensity; the expressive intensity of each attribute can be controlled by independently adjusting these three parameters. Indicates content conditions, Indicates style conditions, Indicates timbre conditions.

[0065] Furthermore, this method also includes employing representation alignment strategies to stabilize text-to-speech model training and accelerate convergence. These strategies include:

[0066] The audio encoder of the pre-trained large speech model is used as the teacher model, and its output acoustic-semantic representation is extracted as the teacher representation.

[0067] Upsample and linearly project the student representation of the intermediate layer of the latent diffusion transformer to align its dimensions with the teacher representation;

[0068] Representation alignment is achieved by minimizing the cosine similarity loss between student and teacher representations.

[0069] Specifically, to stabilize model training and accelerate convergence, a representation alignment (REPA) strategy is employed. This strategy uses the audio encoder of a pre-trained large speech model (such as Whisper) as the teacher model, taking its output acoustic-semantic representation as the teacher model. This is injected into the DiT backbone network of the model of this invention. Specifically, the student representation of the DiT intermediate layer is... After upsampling and linear projection, its dimensions are aligned with the teacher's representation. Then, alignment is achieved by minimizing the cosine similarity loss between the two representations, calculated as follows:

[0070] ;

[0071] in, Indicates linear projection. Indicates an upsampling operation. This indicates taking the expected value over the time dimension.

[0072] It should be noted that the training process of the text-to-speech model of this invention is divided into two independent stages:

[0073] The first phase involves independently training a unified multimodal style encoder;

[0074] In the second stage, the latent diffusion transformer is trained, and the parameters of the trained multimodal style encoder are frozen.

[0075] Specifically, first, a unified multimodal style encoder is trained independently according to the method defined above and its weights are saved. Then, when training the latent diffusion transformer (TTS master model), the style encoder trained in the first stage is loaded and all its parameters are frozen so that it does not participate in gradient updates.

[0076] For example, the model is trained on a high-quality Chinese speech dataset of approximately 300 hours, built internally. The training of this invention is divided into two independent phases. In the first phase, a unified multimodal style encoder is independently trained according to a defined method, and its weights are saved. In the second phase, the TTS master model is trained. During the training of the master model, the style encoder trained in the first phase is loaded, and all its parameters are frozen so that it does not participate in gradient updates. The TTS master model is trained using 8 A800 GPUs, with a batch size of 38,400 frames per batch, for a total of 85,000 training steps. The learning rate is set to 7.5 × 1e-5, and a warm-up of 20,000 steps is used.

[0077] In this embodiment, the speech generation process includes:

[0078] Given the text to be synthesized, a reference audio or speaker ID with a specified timbre, and a reference audio or descriptive text with a specified style;

[0079] Each encoder converts the input information into corresponding conditional embeddings.

[0080] Noise is sampled from a standard Gaussian distribution and used as the initial input to the potential diffusion transformer;

[0081] In the reasoning process of conditional flow matching, a chain-like classifier-free guidance mechanism is used to predict the velocity field and generate a latent representation of the target Mel spectrum;

[0082] The latent representation of the Mel spectrum is converted into the final speech waveform using a vocoder.

[0083] As a concrete example, the first step is multimodal input and conditional embedding transformation. This step aims to transform different forms of input information into a unified, processable numerical feature representation. Given three input conditions: the text to be synthesized: the text content to be converted into speech, with a length of L; timbre reference audio: a speech sample specifying the speaker's timbre; and style reference: either reference audio specifying emotion or style, or descriptive text describing the desired style. The above input information is transformed using independent encoders: Text encoding: The text to be synthesized, with a length of L, is input to a text encoder. This encoder converts the text sequence into a series of high-dimensional text embedding vectors, resulting in a tensor of dimension (L, 512); Timbre encoding: The timbre reference audio is input to a speaker encoder. This encoder extracts identity-identifying timbre features from the audio and outputs a timbre embedding of dimension (1, 192); Style encoding: The style reference (audio or text) is input to a multimodal style encoder. This encoder can handle inputs from different modalities and outputs a fixed-dimensional style embedding of dimension (1, 512).

[0084] Step 2: Temporal Alignment and Conditional Fusion. Since the length L of the text embedding is usually inconsistent with the length T of the final generated audio, temporal alignment is required, and all conditional information needs to be fused. A frame-level duration predictor is used to predict the number of audio frames corresponding to each character or phoneme in the text. Based on the predicted duration information, the text embedding with dimension (L, 512) is upsampled in the temporal dimension to align its length with the target audio frame length T, resulting in an aligned text embedding with dimension (T, 512). Simultaneously, the timbre embedding with dimension (1, 192) and the style embedding with dimension (1, 512) are copied T times in the temporal dimension, respectively, making their dimensions (T, 192) and (T, 512) to ensure that the same timbre and style guidance are provided in each frame.

[0085] Step 3: Generation of Mel Spectrum Latent Variables Based on Conditional Flow Matching Model. This step is the core of the generation process, using a conditional flow matching model to generate latent variables for the target Mel spectrum from random noise. In this scheme, the prior distribution of the latent variables is assumed to be a standard Gaussian distribution. At the start of generation, random sampling is performed from this standard Gaussian distribution to generate a random noise tensor with the exact same shape as the target latent variable. Specifically, the dimension of this noise tensor is (T, 40), where T is the audio frame length and 40 is the feature dimension. This noise tensor forms the starting point of the generation process. The noise tensor generated in the previous step is concatenated with all aligned and copied conditional embeddings along the feature dimension (the second dimension). Specifically: noise input (T, 40), aligned text embedding (T, 512), copied timbre embedding (T, 192), and copied style embedding (T, 512). After concatenation along the feature dimensions, a comprehensive input tensor with a dimension of (T, 1256) is formed.

[0086] The conditional flow matching model used in this embodiment is fundamentally designed to learn a vector field defined by ordinary differential equations and dependent on the input conditions. This vector field describes a continuous, deterministic transformation path from a simple standard Gaussian distribution (noise) to a complex target data distribution. During the inference phase, a synthetic input tensor of dimension (T, 1256) is fed into the CFM model. The model integrates along this vector field with respect to the time variable t from 0 to 1 using a numerical solver (Euler method). This integration process can be viewed as gradually and smoothly transforming the initial noise tensor (T, 40) along the learned path into a latent representation of the target Mel spectrum that satisfies all given conditions. This process is deterministic, unlike the stochastic denoising of diffusion models.

[0087] To enhance the matching degree between the generated speech and the reference in terms of style and timbre, this scheme applies a classifier-free guidance mechanism in the aforementioned ODE solving process. In each integration step of the numerical solver, the model performs multiple forward propagation calculations: Conditional prediction: As described in the implementation, prediction is made using a complete or partial set of conditions (text, timbre, style). Unconditional prediction: All guidance conditions are discarded, and a single prediction is performed. The predicted vectors are linearly combined, and the generation path is more forcefully pushed towards the direction that meets the specified conditions through guidance strength. Because this guidance is applied iteratively and in a chain throughout the integration path, fine-grained control over different attributes of the generation process can be achieved.

[0088] Step 4: Speech Waveform Synthesis. Finally, the target Mel spectrum latent representation with dimension (T,40) generated in the previous step is input into the Mel decoder, and the resulting Mel spectrum is input into the pre-trained vocoder BigVGAN. The vocoder is responsible for converting this intermediate Mel spectrum representation into the final, audible one-dimensional speech waveform data.

[0089] Through the above steps, this embodiment can accurately generate high-quality, personalized speech waveforms deterministically based on input conditions using a conditional flow matching model, solving the problem of poor style and timbre control in traditional methods, and ensuring the stability and efficiency of the generation process.

[0090] As can be seen from the technical solution provided by the present invention, this invention proposes a novel controllable text-to-speech framework. This framework achieves flexible style specification through both audio and text via a unified multimodal style encoder. The introduced chain-based classifier-free guidance mechanism enables independent and continuous adjustment of content, timbre, and style dimensions in the text-to-speech domain. Simultaneously, the application of representation alignment strategies significantly improves the training stability and convergence speed of the model. Experiments demonstrate that this method performs excellently in terms of style controllability, speech naturalness, and speech intelligibility, providing an effective technical approach for generating expressive and highly controllable speech.

[0091] The method of this invention was compared with several open-source large-scale TTS baseline models. Evaluation metrics included attribute accuracy and word error rate (WER). Experimental results show that the method of this invention significantly outperforms all baseline models in style control accuracy, regardless of whether text or audio prompts are used. It also performs excellently in speech clarity (WER), approaching the optimal level. Subjectively, the naturalness of the synthesized speech is comparable to that of real recordings. Particularly in cross-speaker style transfer tasks, this method can accurately transfer style while maintaining the target speaker's timbre, demonstrating the effectiveness of timbre-style decoupling. Statistical results are shown in Table 1.

[0092] Table 1

[0093]

[0094] like Figure 4 As shown in the figure, an electronic device provided in this embodiment of the invention includes: a processor 610, a communication interface 620, a memory 630, and a communication bus 640, wherein the processor 610, the communication interface 620, and the memory 630 communicate with each other through the communication bus 640. The processor 610 can call logical instructions in the memory 630 to execute the method provided in the above embodiment.

[0095] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0096] This invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, is implemented to perform the methods provided in the above embodiments.

[0097] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of various embodiments or some parts of embodiments.

[0098] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A controllable text-to-speech method based on decoupled multimodal prompts and chained guidance, characterized in that, include: A unified multimodal style encoder is constructed to map style cues from reference audio or descriptive text to a shared embedding space to generate style conditional embeddings, and the multimodal style encoder is trained. A text-to-speech model based on a latent diffusion transformer is constructed. The text-to-speech model takes the latent representation of the Mel spectrum as the processing object and receives the content text, speaker timbre and speaking style as input conditions to train the text-to-speech model. In the inference stage of the text-to-speech model, a chain-based classifier-free guidance mechanism is adopted. By independently adjusting the guidance intensity of content, timbre, and style conditional embeddings, independent and continuous control over the content, timbre, and style of the generated speech is achieved. The training objective function of the multimodal style encoder is: ； In the formula, and These are the weighting coefficients. To compare learning loss, For cross-entropy loss, This is the mean square error loss; The process of generating speech includes: Given the text to be synthesized, a reference audio or speaker ID with a specified timbre, and a reference audio or descriptive text with a specified style; Each encoder converts the input information into corresponding conditional embeddings. Noise is sampled from a standard Gaussian distribution and used as the initial input to the potential diffusion converter; In the reasoning process of conditional flow matching, a chain-like classifier-free guidance mechanism is used to predict the velocity field and generate a latent representation of the target Mel spectrum; The latent representation of the Mel spectrum is converted into the final speech waveform using a vocoder.

2. The method according to claim 1, characterized in that, The unified multimodal style encoder is built on a contrastive language-audio pre-trained model and trained by combining contrastive learning with multi-task supervision. Among them, the contrastive learning loss is used to align the style embeddings of audio and text modalities, and the multi-task supervision loss includes cross-entropy loss for discrete style attributes and mean squared error loss for continuous style attributes.

3. The method according to claim 1, characterized in that, The chain-based classifier-free guidance mechanism calculates the final predicted speech velocity field using the following formula: ； in, This represents the velocity field predicted by the model. This represents the velocity field predicted by the model under given conditions. It is the hyperparameter of content guidance intensity. It is the hyperparameter of timbre guidance intensity. It is the hyperparameter of style guidance intensity; Indicates content conditions, Indicates style conditions, Indicates timbre conditions.

4. The method according to claim 3, characterized in that, The chain-based classifier-free guidance mechanism employs a hierarchical conditional dropout strategy during the text-to-speech model training phase, including: Discard style conditions with the first probability ; If style conditions If discarded, the timbre condition is discarded with the second probability. ; When style conditions and timbre conditions When all are discarded, the content is discarded with the third probability. .

5. The method according to claim 1, characterized in that, It also includes employing representation alignment strategies to stabilize text-to-speech model training and accelerate convergence, the strategies including: The audio encoder of the pre-trained large speech model is used as the teacher model, and its output acoustic-semantic representation is extracted as the teacher representation. The student representation of the intermediate layer of the latent diffusion transformer is upsampled and linearly projected to align its dimensions with the teacher representation. Representation alignment is achieved by minimizing the cosine similarity loss between the student and teacher representations.

6. The method according to claim 1, characterized in that, The training process of the text-to-speech model is divided into two independent stages: The first stage involves independently training the unified multimodal style encoder. In the second stage, the latent diffusion transformer is trained, and the parameters of the trained multimodal style encoder are frozen.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1 to 6.

8. A computer storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 6.

Images