Speech synthesis model training method and device, equipment and medium

Abstract

The invention relates to the technical field of artificial intelligence, is applied to the field of intelligent medical treatment and finance, and discloses a speech synthesis model training method, device and equipment and a medium, and the method comprises the steps: obtaining a training sample comprising a target text sequence, a target clean speech corresponding to the target text sequence, and a noisy prompt audio; according to the noisy prompt audio, acoustic prior features are extracted through a pre-trained voice model; according to the target text sequence, the acoustic prior feature and a randomly sampled diffusion time step, extracting a middle hidden feature output by a multi-layer diffusion converter in the speech synthesis model, and performing text and audio modal alignment according to the target text sequence and the acoustic prior feature; and generating a text alignment adaptive weight and an audio alignment adaptive weight according to the diffusion time step and hierarchical information of the diffusion converter, and performing weighted fusion on the text alignment sub-loss and the audio alignment sub-loss to obtain total alignment loss so as to optimize parameters of the speech synthesis model.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence technology and speech processing technology, and in particular to a speech synthesis model training method, apparatus, device and medium. Background Technology

[0002] In the medical and financial fields, business assistance can be provided through intelligent customer service or intelligent assistants. Intelligent customer service or intelligent assistants are one of the most widespread and important ways to implement voice processing technology in real-world scenarios. Voice interaction can provide voice prompts or play back responses to user actions through the client. This requires speech synthesis, which is an important area of ​​artificial intelligence. In recent years, flow matching (FM) has been used for zero-shot text-to-speech (ZS TTS) to improve the efficiency and naturalness of non-autoregressive synthesis. However, existing zero-shot text-to-speech methods generally lack robustness to noisy or paralinguistic audio (such as environmental reverberation, additive noise, other people's interference, laughter, and panting), making it difficult to effectively separate speaker identity from irrelevant disturbances. This leads to a decrease in speaker similarity, sound quality, and pronunciation stability of synthesized speech. Furthermore, zero-sample text-to-speech models based on stream matching rely on implicit guidance mechanisms when learning complex acoustic representations, resulting in weak constraints on the representation space. This can lead to problems such as unstable alignment, slow convergence, or alignment deviation, which in turn reduces the intelligibility of synthesized speech, causes disjointed pronunciation, and blurs details, making it difficult to balance quality and efficiency requirements. Summary of the Invention

[0003] This invention provides a method, apparatus, device, and medium for training a speech synthesis model, in order to solve the technical problems of existing speech synthesis methods lacking robustness, unstable acoustic representation alignment, and poor synthesized speech quality.

[0004] Firstly, a method for training a speech synthesis model is provided, including: Acquire training samples, which include target text sequences, target clean speech corresponding to the target text sequences, and noisy prompt audio; Based on the noisy audio prompt, acoustic prior features are extracted using a pre-trained speech model; Based on the target text sequence, the acoustic prior features, and a randomly sampled diffusion time step, the intermediate hidden features output by the multilayer diffusion transformer in the speech synthesis model are extracted. The intermediate hidden features are then aligned to the text modality based on the target text sequence, and the intermediate hidden features are then aligned to the audio modality based on the acoustic prior features. Based on the diffusion time step and the hierarchy information of the diffusion transformer, text alignment adaptive weights and audio alignment adaptive weights are generated. The text alignment sub-loss and audio alignment loss are then weighted and fused based on the text alignment adaptive weights and audio alignment adaptive weights to obtain the total alignment loss. Based on the total alignment loss, the parameters of the speech synthesis model are optimized and trained.

[0005] Secondly, a speech synthesis model training device is provided, comprising: The training sample acquisition module is used to acquire training samples, which include a target text sequence, target clean speech corresponding to the target text sequence, and noisy prompt audio. The acoustic prior feature extraction module is used to extract acoustic prior features based on the noisy cue audio using a pre-trained speech model. A dual-modal alignment module is used to extract intermediate hidden features from the output of a multilayer diffusion transformer in a speech synthesis model based on the target text sequence, the acoustic prior features, and a randomly sampled diffusion time step; perform text modal alignment on the intermediate hidden features based on the target text sequence; and perform audio modal alignment on the intermediate hidden features based on the acoustic prior features. The alignment loss calculation module is used to generate text alignment adaptive weights and audio alignment adaptive weights based on the diffusion time step and the hierarchy information of the diffusion transformer, and to perform weighted fusion of the text alignment sub-loss and audio alignment loss based on the text alignment adaptive weights and audio alignment adaptive weights to obtain the total alignment loss. The optimization training module is used to optimize the training of the parameters of the speech synthesis model based on the total alignment loss.

[0006] Thirdly, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described speech synthesis model training method.

[0007] Fourthly, a computer-readable storage medium is provided, which stores a computer program that, when executed by a processor, implements the steps of the above-described speech synthesis model training method.

[0008] In the above-mentioned speech synthesis model training method, apparatus, device, and medium, training samples including a target text sequence, target clean speech corresponding to the target text sequence, and noisy cue audio can be obtained through a client. Based on the noisy cue audio, acoustic prior features are extracted using a pre-trained speech model. Based on the target text sequence, acoustic prior features, and a randomly sampled diffusion time step, intermediate hidden features output from the multi-layer diffusion transformer in the speech synthesis model are extracted, and text and audio modal alignment is performed based on the target text sequence and acoustic prior features respectively. Based on the diffusion time step and the hierarchy information of the diffusion transformer, adaptive text alignment weights and adaptive audio alignment weights are generated, and the text alignment sub-loss and audio alignment sub-loss are weighted and fused to obtain the total alignment loss, thereby optimizing the parameters of the speech synthesis model. In this invention, by utilizing a pre-trained speech model to extract training samples from the noisy cue audio... Acoustic prior features are extracted to ensure that the model can stably obtain clear acoustic features even under noisy input, laying a robust foundation for subsequent representation alignment. By performing bimodal alignment of the intermediate hidden features output by the multi-layer diffusion transformer of the speech synthesis model in both text and audio modes, the speech synthesis model can achieve dual constraints of content consistency and acoustic consistency during training, improving alignment stability. Furthermore, adaptive text alignment weights and adaptive audio alignment weights are generated based on the diffusion time step and the hierarchy information of the diffusion transformer. This time-hierarchical adaptive alignment mechanism addresses the problem of inconsistent attention to content and acoustic features in flow matching at different diffusion time steps and different diffusion transformer hierarchies. This allows the speech synthesis model to obtain highly complementary and controllable representations from different layers of diffusion transformers, enhancing speech intelligibility and improving training stability, effectively improving the quality and effect of speech synthesis. Attached Figure Description

[0009] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0010] Figure 1 This is a schematic diagram of an application environment for a speech synthesis model training method according to an embodiment of the present invention; Figure 2 This is a flowchart illustrating a speech synthesis model training method according to an embodiment of the present invention; Figure 3 This is a schematic diagram of a speech synthesis model training device in one embodiment of the present invention; Figure 4This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention; Figure 5 This is another structural schematic diagram of a computer device according to one embodiment of the present invention. Detailed Implementation

[0011] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0012] The speech synthesis model training method provided in this embodiment of the invention can be applied to, for example... Figure 1In this application environment, the client communicates with the server via a network. The server can obtain training samples from the client, including the target text sequence, the target clean speech corresponding to the target text sequence, and the noisy cue audio. Based on the noisy cue audio, acoustic prior features are extracted using a pre-trained speech model. Based on the target text sequence, acoustic prior features, and a randomly sampled diffusion time step, intermediate hidden features output by the multi-layer diffusion transformer in the speech synthesis model are extracted, and text and audio modal alignment is performed based on the target text sequence and acoustic prior features respectively. Based on the diffusion time step and the hierarchy information of the diffusion transformer, adaptive weights for text alignment and adaptive weights for audio alignment are generated, and the text alignment sub-loss and audio alignment sub-loss are weighted and fused to obtain the total alignment loss, thereby optimizing the parameters of the speech synthesis model. In this invention, by utilizing a pre-trained speech model to extract acoustic prior features based on the noisy cue audio, the model can achieve better performance. Even with noisy input, clear acoustic features can be stably obtained, laying a robust foundation for subsequent representation alignment. By performing bimodal alignment of the intermediate hidden features output by the multi-layer diffusion transformer of the speech synthesis model in both text and audio modes, the speech synthesis model can achieve dual constraints of content consistency and acoustic consistency during training, improving alignment stability. Moreover, adaptive text alignment weights and adaptive audio alignment weights are generated based on the diffusion time step and the hierarchy information of the diffusion transformer. This time-hierarchical adaptive alignment mechanism addresses the problem of inconsistent attention to content and acoustic features in flow matching at different diffusion time steps and different diffusion transformer hierarchies. This allows the speech synthesis model to obtain highly complementary and controllable representations from different layers of diffusion transformers, enhancing speech intelligibility and improving training stability, effectively improving the quality and effect of speech synthesis. The client can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster consisting of multiple servers. The invention will be described in detail below through specific embodiments.

[0013] Please see Figure 2 As shown, Figure 2 A flowchart illustrating the speech synthesis model training method provided in this embodiment of the invention includes the following steps: S10: Obtain training samples, which include target text sequences, target clean speech corresponding to the target text sequences, and noisy prompt audio.

[0014] The speech synthesis model training method provided by this invention can be applied to intelligent question-answering engines such as intelligent customer service or intelligent assistants in various application scenarios to train the speech synthesis model in the intelligent question-answering engine. Intelligent question-answering engines are typically implemented through a server-side component that can receive user questions in real time. For example, in the medical field, users often ask questions via telephone or online chat, and intelligent question-answering engines are often needed to answer these medical inquiries to improve user experience.

[0015] For example, a user's question might be, "What is the normal range for blood pressure?" After receiving this question, the intelligent question-answering engine analyzes it and obtains the corresponding standard answer. Based on the text information of this standard answer, it then synthesizes speech. It's worth noting that the standard answer can refer to an answer written by an expert or an intelligent answer generated from previous intelligent question-answering sessions; there are no specific limitations here. Alternatively, in the financial application field, users often ask questions via phone or online chat. Intelligent customer service can be used to answer customers' inquiries about financial products, thereby improving the efficiency of financial product promotion and user experience.

[0016] S20: Based on the noisy audio prompt, extract acoustic prior features using a pre-trained speech model.

[0017] The noisy cue audio contains acoustic information of the target speaker. Specifically, step S20, namely, extracting acoustic prior features based on the noisy cue audio using a pre-trained speech model, includes: Based on the noisy cue audio, acoustic features and speech features are extracted using a pre-trained speech model. The acoustic features are the output features of the first layer of the pre-trained speech model, and the speech features are the output features of the final layer of the pre-trained speech model. Preferably, the acoustic features can be the output features of the lower layer of the pre-trained speech model, and the speech features can be the output features of the higher layer of the pre-trained speech model. The lower, middle, and higher layers of the speech model each account for one-third of the layers of the speech model, respectively. The acoustic features and the speech features are summed frame by frame to obtain the acoustic prior features.

[0018] Specifically, before step S20, that is, before extracting acoustic prior features from the pre-trained speech model based on the noisy cue audio, the speech synthesis model training method further includes: A teacher-student speech model architecture is constructed, and the speech models are trained using a multi-layer distillation mechanism to obtain pre-trained speech models. The teacher speech model uses clean audio as input, while the student speech model uses noisy frequencies as input, where the noisy frequencies are generated by adding noise to the clean audio. The structure of the student speech model is identical to that of the teacher speech model. Both the teacher and student speech models are self-supervised speech models (S3M). The S3M can employ a Waveform Language Model (WAVLM) to directly process the original audio waveform, using a Transformer architecture to learn acoustic and semantic information from the original waveform. The S3M uses only the original audio signal as training input and does not require explicit text or semantic labels. Different layers of the S3M learn different speech attributes. The lower layers of the S3M capture fine-grained acoustic details and speaker features, and their output is considered an "acoustic representation." The higher layers of the S3M encode highly abstract, context-dependent language content information, and their output is considered a "semantic representation." The parameters of the teacher's speech model are frozen, and the updates of the student's speech model are driven solely by distillation loss. This allows the student's speech model to continuously approximate the hierarchical output features of the teacher's speech model on clean audio during training, thereby improving noise robustness. By constructing a teacher-student speech model architecture and combining it with a multi-layer distillation mechanism for training, the pre-trained speech model can extract clean speaker audio features from noisy input. This enables the subsequent speech synthesis model to directly learn noise-independent generation capabilities without complex external preprocessing, achieving end-to-end robust training.

[0019] Preferably, the noisy frequency is audio generated by adding noise to the clean audio through data augmentation technology. The added noise includes additive noise (such as white noise) and / or convolutional noise (such as applying a room impulse response to simulate reverberation).

[0020] Specifically, the training of the speech model using the multi-layer distillation mechanism includes: Based on the clean audio, the first multi-level feature set is obtained through the teacher's speech model; Based on the noisy frequency and the student speech model, the second multi-level feature set is obtained; Calculate the feature differences between the first multi-level feature set and the second multi-level feature set at the corresponding levels, and perform a weighted summation of the feature differences at the corresponding levels according to the preset level weights to obtain the distillation loss; The parameters of the student speech model are updated based on the distillation loss.

[0021] Specifically, feature differences can be calculated for the output features of all layers of the speech model or for some preset layers. These preset layers correspond to at least two different network layers among the low, middle, and high layers of the speech model, and the number of preset layers is no less than two. By weighted summation of the feature differences of the corresponding layers according to preset layer weights to obtain the distillation loss, the distillation difficulty of different layers can be balanced. Higher weights are applied to the lower layers to enhance acoustic consistency, and higher weights are applied to the higher layers to improve content preservation. The feature differences are calculated using cosine similarity loss or mean squared error loss.

[0022] Specifically, the distillation loss is expressed by formula (1): (1) In the formula, Indicates distillation loss, Indicates the preset first Layer weights, Indicates the number of characteristic differences. This represents the feature difference between the output features of the first multi-level feature set and the second multi-level feature set at the 1st layer. The function representing the square operation of the L2 norm. Represents the second multi-level feature set. The output features of the layer Represents the first multi-level feature set. The output features of each layer are vectors.

[0023] S30: Based on the target text sequence, the acoustic prior features, and a randomly sampled diffusion time step, extract the intermediate hidden features output by the multilayer diffusion transformer in the speech synthesis model, perform text modal alignment on the intermediate hidden features according to the target text sequence, and perform audio modal alignment on the intermediate hidden features according to the acoustic prior features.

[0024] Specifically, based on the target text sequence, the acoustic prior features, and a randomly sampled diffusion time step, multiple corresponding intermediate hidden features are generated by extracting a multi-layered diffusion transformer (DiT) in the speech synthesis model. Text content consistency constraints are imposed on the intermediate hidden features by aligning them with the target text sequence in terms of text modality, and acoustic attribute regression constraints are imposed on the intermediate hidden features by aligning them with the acoustic prior features in terms of audio modality.

[0025] Specifically, step S30, which involves aligning the intermediate hidden features with the target text sequence according to their text modalities, includes: The intermediate hidden features output by each layer of diffusion transformers are linearly projected onto the intermediate semantic feature space to obtain the compressed features corresponding to each layer of diffusion transformers. The dimension of the intermediate semantic feature space is lower than that of the intermediate hidden features. By linearly projecting the intermediate hidden features of each layer onto the intermediate semantic feature space, acoustic redundancy information is removed, high-dimensional acoustic noise in Connectionist Temporal Classification (CTC) is removed, and core features related to the target text sequence are retained. For each time step of the compressed feature corresponding to each layer of diffusion transformer, the feature is mapped to a probability distribution on the text vocabulary, thus obtaining the frame-by-frame probability distribution sequence corresponding to each layer of diffusion transformer; wherein, the feature of each time step of the compressed feature is mapped to a probability distribution on the text vocabulary through the Softmax classification function; the frame-by-frame probability distribution sequence is the text prediction result. The loss between the frame-by-frame probability distribution sequence corresponding to each layer of the diffusion transformer and the target text sequence is calculated based on the loss function of the temporal connection classification algorithm, and is used as the text alignment sub-loss of each layer of the diffusion transformer.

[0026] The loss between the frame-by-frame probability distribution sequence of each layer and the target text sequence is the text alignment loss of each layer. The temporal connection classification algorithm allows the neural network to output a sequence containing whitespace and repeating characters, which is then converted into a text sequence through a specific decoding process. It can automatically learn the temporal mapping of text units, achieving accurate alignment between audio frames and characters. The training of the temporal connection classification algorithm prompts the latent space of the diffusion transform to learn the temporal-text mapping relationship from speech frames to text units, so that the latent space has the ability to align language structures.

[0027] Specifically, step S30, which involves aligning the intermediate hidden features with audio modalities based on the acoustic prior features, includes: The intermediate hidden features output by each diffusion transformer are mapped to a feature space that matches the acoustic prior features to obtain the mapped features corresponding to each diffusion transformer. The intermediate hidden features output by each diffusion transformer are mapped through the corresponding level of the Multilayer Perceptron (MLP) hierarchical projection module so that different dimensional transformations can be performed on each intermediate hidden feature. The parameters of the multilayer perceptron hierarchical projection modules at different levels are not shared to adapt to the speech modeling responsibilities at different levels. The feature difference between the mapping features corresponding to each layer of diffusion transformer and the acoustic prior features is calculated to obtain the audio alignment loss of each layer of diffusion transformer.

[0028] S40: Based on the diffusion time step and the hierarchical information of the diffusion transformer, generate text alignment adaptive weights and audio alignment adaptive weights, and perform weighted fusion of text alignment sub-loss and audio alignment loss based on the text alignment adaptive weights and the audio alignment adaptive weights to obtain the total alignment loss.

[0029] Specifically, step S40, which involves generating text alignment adaptive weights and audio alignment adaptive weights based on the diffusion time step and the hierarchy information of the diffusion transformer, and then weighting and fusing the text alignment sub-loss and audio alignment loss based on the text alignment adaptive weights and audio alignment adaptive weights to obtain the total alignment loss, includes: The diffusion time step is mapped to a first time embedding vector through a first embedding layer. Based on the first time embedding vector and the hierarchical information of the diffusion transformer, text alignment weights corresponding to each layer of the diffusion transformer are generated through a first time multilayer perceptron. The text alignment weights are then normalized to obtain adaptive text alignment weights. Wherein, the following methods can be used: As an embedding layer, the diffusion time steps are mapped to high-dimensional vectors, which can be encoded using sine and cosine positions. Normalization can be performed using the Softmax classification function. The diffusion time step is mapped to a second time embedding vector through the second embedding layer. Based on the second time embedding vector and the hierarchical information of the diffusion transformer, the audio alignment weights corresponding to each layer of the diffusion transformer are generated through the second time multilayer perceptron. The audio alignment weights are normalized to obtain the audio alignment adaptive weights. The text alignment loss is obtained by weighted summing of the text alignment sub-losses of each diffusion transformer layer according to the text adaptive weights. The audio alignment loss is obtained by weighted summing of the audio alignment sub-losses of each layer of the diffusion transformer according to the audio alignment adaptive weights. The total alignment loss is obtained by adding the text alignment loss to the audio alignment loss.

[0030] Specifically, the text alignment loss is calculated using formula (2): (2) In the formula, Indicates text alignment loss. This indicates the total number of levels in the diffusion converter. Indicates the sequence number of the diffusion converter level. Indicates the sequence number of the diffusion time step. Indicates the first Text alignment sub-loss of layer diffusion transform, Indicates the first Layer Text adaptive alignment weights for each time step.

[0031] Specifically, the audio alignment loss is calculated using formula (3): (3) In the formula, Indicates audio alignment loss. This indicates the total number of levels in the diffusion converter. Indicates the sequence number of the diffusion converter level. Indicates the sequence number of the diffusion time step. Indicates the first The audio alignment loss of the layer diffusion transformer, Represents acoustic prior features. Indicates the first Mapping characteristics of layer diffusion converters Indicates the first Layer Audio adaptive alignment weights for each time step.

[0032] S50: Optimize the parameters of the speech synthesis model based on the total alignment loss.

[0033] As can be seen, in the above scheme, by using a pre-trained speech model to extract acoustic prior features based on noisy audio prompts, the model can still stably obtain clear acoustic features under noisy input, laying a robust foundation for subsequent representation alignment. By performing bimodal alignment of the intermediate hidden features output by the multi-layer diffusion transformer of the speech synthesis model in both text and audio modes, the speech synthesis model can achieve dual constraints of content consistency and acoustic consistency during training, improving alignment stability. Moreover, by generating adaptive text alignment weights and adaptive audio alignment weights based on the diffusion time step and the hierarchy information of the diffusion transformer, the time-hierarchical adaptive alignment mechanism is used to solve the problem of inconsistent attention to content and acoustic features in flow matching at different diffusion time steps and different diffusion transformer levels. This allows the speech synthesis model to obtain highly complementary and controllable representations from different layers of diffusion transformers, enhancing speech intelligibility and improving training stability, effectively improving the quality and effect of speech synthesis.

[0034] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0035] In one embodiment, a speech synthesis model training device is provided, which corresponds one-to-one with the speech synthesis model training method described in the above embodiments. For example... Figure 3 As shown, the speech synthesis model training device includes a training sample acquisition module 101, an acoustic prior feature extraction module 102, a bimodal alignment module 103, an alignment loss calculation module 104, and an optimization training module 105. Detailed descriptions of each functional module are as follows: The training sample acquisition module 101 is used to acquire training samples, which include a target text sequence, target clean speech corresponding to the target text sequence, and noisy prompt audio. The acoustic prior feature extraction module 102 is used to extract acoustic prior features based on the noisy cue audio using a pre-trained speech model. The dual-modal alignment module 103 is used to extract intermediate hidden features output by the multilayer diffusion transformer in the speech synthesis model based on the target text sequence, the acoustic prior features and a randomly sampled diffusion time step, perform text modal alignment on the intermediate hidden features based on the target text sequence, and perform audio modal alignment on the intermediate hidden features based on the acoustic prior features. The alignment loss calculation module 104 is used to generate text alignment adaptive weights and audio alignment adaptive weights based on the diffusion time step and the hierarchy information of the diffusion transformer, and to perform weighted fusion of the text alignment sub-loss and audio alignment sub-loss based on the text alignment adaptive weights and audio alignment adaptive weights to obtain the total alignment loss. The optimization training module 105 is used to optimize the training of the parameters of the speech synthesis model based on the total alignment loss.

[0036] In one embodiment, the acoustic prior feature extraction module 102 is specifically used for: Based on the noisy cue audio, acoustic features and speech features are extracted using a pre-trained speech model, wherein the acoustic features are the output features of the first layer of the pre-trained speech model, and the speech features are the output features of the final layer of the pre-trained speech model. The acoustic features and the speech features are summed frame by frame to obtain the acoustic prior features.

[0037] In one embodiment, the acoustic prior feature extraction module 102 is further configured to: A teacher speech model-student speech model architecture is constructed, and the speech model is trained by a multi-layer distillation mechanism to obtain a pre-trained speech model. The teacher speech model uses clean audio as input audio, and the student speech model uses noisy frequencies as input audio. The noisy frequencies are audio generated by adding noise to the clean audio.

[0038] In one embodiment, the acoustic prior feature extraction module 102 is further configured to: Based on the clean audio, the first multi-level feature set is obtained through the teacher's speech model; Based on the noisy frequency and the student speech model, the second multi-level feature set is obtained; Calculate the feature differences between the first multi-level feature set and the second multi-level feature set at the corresponding levels, and perform a weighted summation of the feature differences at the corresponding levels according to the preset level weights to obtain the distillation loss; The parameters of the student speech model are updated based on the distillation loss.

[0039] In one embodiment, the dual-modal alignment module 103 is specifically used for: The intermediate hidden features output by each layer of the diffusion transformer are linearly projected onto the intermediate semantic feature space to obtain the compressed features corresponding to each layer of the diffusion transformer; wherein, the dimension of the intermediate semantic feature space is lower than the dimension of the intermediate hidden features. The feature mapping of each time step of the compressed feature corresponding to each layer of diffusion transformer is transformed into a probability distribution on the text vocabulary, thus obtaining the frame-by-frame probability distribution sequence corresponding to each layer of diffusion transformer. The loss between the frame-by-frame probability distribution sequence corresponding to each layer of the diffusion transformer and the target text sequence is calculated based on the loss function of the temporal connection classification algorithm, and is used as the text alignment sub-loss of each layer of the diffusion transformer.

[0040] In one embodiment, the dual-modal alignment module 103 is further configured to: The intermediate hidden features output by each layer of diffusion transformers are mapped to a feature space that matches the acoustic prior features to obtain the mapping features corresponding to each layer of diffusion transformers. The feature difference between the mapping features corresponding to each layer of diffusion transformer and the acoustic prior features is calculated to obtain the audio alignment loss of each layer of diffusion transformer.

[0041] In one embodiment, the alignment loss calculation module 104 is specifically used for: The diffusion time step is mapped to a first time embedding vector through the first embedding layer. Based on the first time embedding vector and the hierarchical information of the diffusion transformer, the text alignment weights corresponding to each layer of the diffusion transformer are generated through the first time multilayer perceptron. The text alignment weights are normalized to obtain the text alignment adaptive weights. The diffusion time step is mapped to a second time embedding vector through the second embedding layer. Based on the second time embedding vector and the hierarchical information of the diffusion transformer, the audio alignment weights corresponding to each layer of the diffusion transformer are generated through the second time multilayer perceptron. The audio alignment weights are normalized to obtain the audio alignment adaptive weights. The text alignment loss is obtained by weighted summing of the text alignment sub-losses of each diffusion transformer layer according to the text adaptive weights. The audio alignment loss is obtained by weighted summing of the audio alignment sub-losses of each layer of the diffusion transformer according to the audio alignment adaptive weights. The total alignment loss is obtained by adding the text alignment loss to the audio alignment loss.

[0042] This invention provides a speech synthesis model training device. By utilizing a pre-trained speech model to extract acoustic prior features based on noisy audio cues, the model can stably obtain clear acoustic features even under noisy input, laying a robust foundation for subsequent representation alignment. By performing bimodal alignment of the intermediate hidden features output by the multi-layer diffusion transformer of the speech synthesis model in both text and audio modes, the speech synthesis model can achieve dual constraints of content consistency and acoustic consistency during training, improving alignment stability. Furthermore, by generating adaptive text alignment weights and adaptive audio alignment weights based on the diffusion time step and the hierarchy information of the diffusion transformer, a time-hierarchical adaptive alignment mechanism is used to address the problem of inconsistent attention to content and acoustic features in flow matching across different diffusion time steps and different diffusion transformer hierarchies. This allows the speech synthesis model to obtain highly complementary and controllable representations from different layers of diffusion transformers, enhancing speech intelligibility and improving training stability, effectively improving the quality and effect of speech synthesis.

[0043] Specific limitations regarding the speech synthesis model training device can be found in the limitations of the speech synthesis model training method described above, and will not be repeated here. Each module in the aforementioned speech synthesis model training device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0044] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 4As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external clients via a network connection. When the computer program is executed by the processor, it implements the functions or steps of a speech synthesis model training method on the server side.

[0045] In one embodiment, a computer device is provided, which may be a client, and its internal structure diagram may be as follows: Figure 5 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it implements the functions or steps of a speech synthesis model training method on the client side.

[0046] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps: Acquire training samples, which include target text sequences, target clean speech corresponding to the target text sequences, and noisy prompt audio; Based on the noisy audio prompt, acoustic prior features are extracted using a pre-trained speech model; Based on the target text sequence, the acoustic prior features, and a randomly sampled diffusion time step, the intermediate hidden features output by the multilayer diffusion transformer in the speech synthesis model are extracted. The intermediate hidden features are then aligned to the text modality based on the target text sequence, and the intermediate hidden features are then aligned to the audio modality based on the acoustic prior features. Based on the diffusion time step and the hierarchy information of the diffusion transformer, text alignment adaptive weights and audio alignment adaptive weights are generated. The text alignment sub-loss and audio alignment loss are then weighted and fused based on the text alignment adaptive weights and audio alignment adaptive weights to obtain the total alignment loss. Based on the total alignment loss, the parameters of the speech synthesis model are optimized and trained.

[0047] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor: Acquire training samples, which include target text sequences, target clean speech corresponding to the target text sequences, and noisy prompt audio; Based on the noisy audio prompt, acoustic prior features are extracted using a pre-trained speech model; Based on the target text sequence, the acoustic prior features, and a randomly sampled diffusion time step, the intermediate hidden features output by the multilayer diffusion transformer in the speech synthesis model are extracted. The intermediate hidden features are then aligned to the text modality based on the target text sequence, and the intermediate hidden features are then aligned to the audio modality based on the acoustic prior features. Based on the diffusion time step and the hierarchy information of the diffusion transformer, text alignment adaptive weights and audio alignment adaptive weights are generated. The text alignment sub-loss and audio alignment loss are then weighted and fused based on the text alignment adaptive weights and audio alignment adaptive weights to obtain the total alignment loss. Based on the total alignment loss, the parameters of the speech synthesis model are optimized and trained.

[0048] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and client side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0049] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0050] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0051] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A method for training a speech synthesis model, characterized in that, include: Acquire training samples, which include target text sequences, target clean speech corresponding to the target text sequences, and noisy prompt audio; Based on the noisy audio prompt, acoustic prior features are extracted using a pre-trained speech model; Based on the target text sequence, the acoustic prior features, and a randomly sampled diffusion time step, the intermediate hidden features output by the multilayer diffusion transformer in the speech synthesis model are extracted. The intermediate hidden features are then aligned to the text modality based on the target text sequence, and the intermediate hidden features are then aligned to the audio modality based on the acoustic prior features. Based on the diffusion time step and the hierarchy information of the diffusion transformer, text alignment adaptive weights and audio alignment adaptive weights are generated. The text alignment sub-loss and audio alignment loss are then weighted and fused based on the text alignment adaptive weights and audio alignment adaptive weights to obtain the total alignment loss. Based on the total alignment loss, the parameters of the speech synthesis model are optimized and trained.

2. The speech synthesis model training method as described in claim 1, characterized in that, The step of extracting acoustic prior features from the noisy cue audio using a pre-trained speech model includes: Based on the noisy cue audio, acoustic features and speech features are extracted using a pre-trained speech model, wherein the acoustic features are the output features of the first layer of the pre-trained speech model, and the speech features are the output features of the final layer of the pre-trained speech model. The acoustic features and the speech features are summed frame by frame to obtain the acoustic prior features.

3. The speech synthesis model training method as described in claim 1, characterized in that, Before extracting acoustic prior features from the noisy cue audio using a pre-trained speech model, the speech synthesis model training method further includes: A teacher speech model-student speech model architecture is constructed, and the speech model is trained by a multi-layer distillation mechanism to obtain a pre-trained speech model. The teacher speech model uses clean audio as input audio, and the student speech model uses noisy frequencies as input audio. The noisy frequencies are audio generated by adding noise to the clean audio.

4. The speech synthesis model training method as described in claim 3, characterized in that, The training of the speech model using a multi-layer distillation mechanism includes: Based on the clean audio, the first multi-level feature set is obtained through the teacher's speech model; Based on the noisy frequency and the student speech model, the second multi-level feature set is obtained; Calculate the feature differences between the first multi-level feature set and the second multi-level feature set at the corresponding levels, and perform a weighted summation of the feature differences at the corresponding levels according to the preset level weights to obtain the distillation loss; The parameters of the student speech model are updated based on the distillation loss.

5. The speech synthesis model training method as described in claim 1, characterized in that, The text modality alignment of the intermediate hidden features according to the target text sequence includes: The intermediate hidden features output by each layer of the diffusion transformer are linearly projected onto the intermediate semantic feature space to obtain the compressed features corresponding to each layer of the diffusion transformer; wherein, the dimension of the intermediate semantic feature space is lower than the dimension of the intermediate hidden features. The feature mapping of each time step of the compressed feature corresponding to each layer of diffusion transformer is transformed into a probability distribution on the text vocabulary, thus obtaining the frame-by-frame probability distribution sequence corresponding to each layer of diffusion transformer. The loss between the frame-by-frame probability distribution sequence corresponding to each layer of the diffusion transformer and the target text sequence is calculated based on the loss function of the temporal connection classification algorithm, and is used as the text alignment sub-loss of each layer of the diffusion transformer.

6. The speech synthesis model training method as described in claim 1, characterized in that, The audio modal alignment of the intermediate hidden features based on the acoustic prior features includes: The intermediate hidden features output by each layer of diffusion transformers are mapped to a feature space that matches the acoustic prior features to obtain the mapping features corresponding to each layer of diffusion transformers. The feature difference between the mapping features corresponding to each layer of diffusion transformer and the acoustic prior features is calculated to obtain the audio alignment loss of each layer of diffusion transformer.

7. The speech synthesis model training method as described in claim 1, characterized in that, The process involves generating text alignment adaptive weights and audio alignment adaptive weights based on the diffusion time step and the hierarchy information of the diffusion transformer. Then, the text alignment sub-loss and audio alignment sub-loss are weighted and fused based on these weights to obtain the total alignment loss, including: The diffusion time step is mapped to a first time embedding vector through the first embedding layer. Based on the first time embedding vector and the hierarchical information of the diffusion transformer, the text alignment weights corresponding to each layer of the diffusion transformer are generated through the first time multilayer perceptron. The text alignment weights are normalized to obtain the text alignment adaptive weights. The diffusion time step is mapped to a second time embedding vector through the second embedding layer. Based on the second time embedding vector and the hierarchical information of the diffusion transformer, the audio alignment weights corresponding to each layer of the diffusion transformer are generated through the second time multilayer perceptron. The audio alignment weights are normalized to obtain the audio alignment adaptive weights. The text alignment loss is obtained by weighted summing of the text alignment sub-losses of each diffusion transformer layer according to the text adaptive weights. The audio alignment loss is obtained by weighted summing of the audio alignment sub-losses of each layer of the diffusion transformer according to the audio alignment adaptive weights. The total alignment loss is obtained by adding the text alignment loss to the audio alignment loss.

8. A speech synthesis model training device, characterized in that, include: The training sample acquisition module is used to acquire training samples, which include a target text sequence, target clean speech corresponding to the target text sequence, and noisy prompt audio. The acoustic prior feature extraction module is used to extract acoustic prior features based on the noisy cue audio using a pre-trained speech model. A dual-modal alignment module is used to extract intermediate hidden features from the output of a multilayer diffusion transformer in a speech synthesis model based on the target text sequence, the acoustic prior features, and a randomly sampled diffusion time step; perform text modal alignment on the intermediate hidden features based on the target text sequence; and perform audio modal alignment on the intermediate hidden features based on the acoustic prior features. The alignment loss calculation module is used to generate text alignment adaptive weights and audio alignment adaptive weights based on the diffusion time step and the hierarchy information of the diffusion transformer, and to perform weighted fusion of the text alignment sub-loss and audio alignment loss based on the text alignment adaptive weights and audio alignment adaptive weights to obtain the total alignment loss. The optimization training module is used to optimize the training of the parameters of the speech synthesis model based on the total alignment loss.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the speech synthesis model training method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the speech synthesis model training method as described in any one of claims 1 to 7.

Images