Speech synthesis method and system for controllable latent variable modeling based on semantic distillation

Abstract

The invention relates to the technical field of speech synthesis, and particularly discloses a speech synthesis method and system for controllable latent variable modeling based on semantic distillation, and the method comprises the steps: converting a Mel spectrum into continuous latent variable distribution through a speech coding module, generating continuous latent variables through re-parameterization sampling, introducing a self-supervised model for semantic distillation, and carrying out the semantic distillation. According to the method, alignment of latent variables and semantic features is constrained through marginal cosine similarity and distance matrix structure loss, a text encoder maps a phoneme sequence into latent variable distribution, time sequence alignment of a text and the latent variables is achieved in combination with monotonic alignment search, and a decoder reconstructs the latent variables into a Mel spectrum. According to the method, waveform synthesis through a vocoder and total loss function joint optimization reconstruction, KL divergence, distillation, text alignment and confrontation loss are carried out, discrete information loss is avoided through continuous latent variable modeling, semantic consistency and text alignment efficiency are enhanced, the naturalness, coherence and real-time performance of synthesized voice are improved, and the method is suitable for scenes such as voice assistants and virtual anchors.

Description

Technical Field

[0001] The present invention relates to the field of speech synthesis technology, and in particular to a speech synthesis method and system based on controllable latent variable modeling of semantic distillation. Background Art

[0002] With the development of technology, end-to-end text-to-speech (TTS) models based on encoder-decoder architectures have become mainstream in speech synthesis. These methods typically map text to intermediate audio representations such as mel-spectrograms, and then synthesize waveforms using a neural vocoder. To improve representation capabilities, researchers have introduced semantic information and self-supervised representation learning methods. For example, pre-trained models such as HuBERT and Wav2Vec are used to extract semantic features to assist in modeling.

[0003] In terms of speech representation, discrete representations (such as the Vector Quantized Variational Autoencoder (VQ-VAE)) have gradually been introduced. These methods improve semantic expression and auditory quality by projecting HuBERT semantic information into a unified discrete space, quantizing it, and then reconstructing it. While these methods improve representation compressibility and modeling stability through discretization, they still suffer from the following core flaws:

[0004] 1. The discrete coding method forces the continuous speech features to be mapped into a finite codebook, resulting in the loss of fine-grained information such as phoneme details and intonation changes, which is particularly evident in emotional expressions and complex speech scenarios.

[0005] 2. Discretization modules (such as residual vector quantization (RVQ)) require fine-tuning of parameters to maintain convergence. Non-differentiable quantization operations can easily lead to unstable training processes and increase the difficulty of model optimization.

[0006] 3. The inference stage cannot directly map from text to discrete latent variable space and needs to rely on additional modules such as classifiers or lookup tables, which reduces the efficiency of end-to-end synthesis and makes it difficult to meet real-time requirements.

[0007] 4. Existing methods mostly use self-supervised semantic features such as HuBERT as auxiliary distillation, which does not fully integrate its continuous expression capabilities and has low structural utilization, resulting in semantic drift or contextual disconnection in the synthesized speech.

[0008] Therefore, there is an urgent need to design speech synthesis methods and systems based on controllable latent variable modeling based on semantic distillation to solve these problems. Summary of the Invention

[0009] The present invention aims to provide a speech synthesis method based on controllable latent variable modeling of semantic distillation, comprising the following steps:

[0010] Get input text and speech training samples;

[0011] Extracting Mel spectrum features according to the speech training sample, and inputting the Mel spectrum features into a speech coding module to generate continuous latent variable distribution parameters;

[0012] Performing reparameterized sampling based on the continuous latent variable distribution parameter to obtain a continuous latent variable;

[0013] Inputting the Mel spectrum features into the self-supervised semantic model in a frozen state to extract the teacher semantic vector;

[0014] Calculating the distillation loss between the continuous latent variable and the teacher semantic vector through a semantic distillation module;

[0015] Convert the input text into a phoneme sequence and input it into a text encoder to generate a text hidden layer representation, and map the text hidden layer representation into a predicted latent variable distribution parameter through a text alignment module;

[0016] Constructing an alignment loss based on the predicted latent variable distribution parameters and the continuous latent variable distribution parameters output by the speech encoding module;

[0017] Inputting the continuous latent variable into a decoder module to generate a reconstructed Mel spectrum, and inputting the reconstructed Mel spectrum into a vocoder module to synthesize a speech waveform;

[0018] The distillation loss, alignment loss, and Mel spectrum reconstruction loss are jointly optimized to train an end-to-end speech synthesis model.

[0019] Furthermore, the step of extracting Mel-spectrogram features according to the speech training sample and inputting the Mel-spectrogram features into a speech coding module to generate continuous latent variable distribution parameters includes:

[0020] Perform mean-variance normalization on the input Mel spectrum;

[0021] The normalized Mel spectrum is input into a five-layer one-dimensional convolutional network. After each convolution layer, layer normalization and activation operations are performed, and residual connections are added.

[0022] Output the mean vector and log variance vector at the time frame level;

[0023] Constructing a Gaussian distribution based on the mean vector and the logarithmic variance vector, and obtaining a continuous latent variable by reparameterized sampling;

[0024] The KL divergence between the Gaussian distribution and the standard normal distribution is calculated as the regularization loss.

[0025] Furthermore, the step of calculating the distillation loss between the continuous latent variable and the teacher semantic vector by the semantic distillation module includes:

[0026] Aligning the continuous latent variable to the feature dimension of the teacher semantic vector through a linear projection layer;

[0027] Calculate the directional similarity between the projected student vector and the teacher semantic vector at each time frame;

[0028] Construct the internal structure relationship similarity matrix between the student vector sequence and the teacher semantic vector sequence;

[0029] The distillation loss is obtained by weighting the direction similarity deviation and the structure similarity deviation.

[0030] Furthermore, the steps of converting the input text into a phoneme sequence, inputting the sequence into a text encoder to generate a text latent representation, and mapping the text latent representation into a predicted latent variable distribution parameter through a text alignment module include:

[0031] Processing phoneme sequences through a four-layer Transformer encoder to generate context-aware text representations;

[0032] Mapping the text representation into a predicted mean vector and a predicted log-variance vector at the phoneme level;

[0033] Calculate the KL divergence between the predicted distribution and the true latent variable distribution output by the speech encoding module;

[0034] A monotone alignment search algorithm is used to construct the optimal alignment path between phonemes and time frames;

[0035] Project the prediction result distribution to the time frame level distribution according to the alignment path.

[0036] Furthermore, the step of constructing an alignment loss based on the predicted latent variable distribution parameter and the continuous latent variable distribution parameter output by the speech encoding module includes:

[0037] Mapping the continuous latent variable to an intermediate feature dimension through a linear layer;

[0038] The features are processed sequentially through five layers of residual convolution blocks, each layer including convolution, layer normalization, activation and residual connection;

[0039] Feed the convolutional output into a bidirectional gated recurrent unit to capture long-term temporal dependencies;

[0040] The gated recurrent unit output is converted to the target Mel-spectrogram dimension through a linear transformation layer;

[0041] The L1 distance between the reconstructed Mel spectrum and the original Mel spectrum and the vocoder discriminator feature distance are calculated as the perceptual loss.

[0042] Furthermore, the step of jointly optimizing the distillation loss, alignment loss, and Mel spectrum reconstruction loss to train the end-to-end speech synthesis model includes:

[0043] Generative adversarial network architecture is used to process and reconstruct the Mel spectrum;

[0044] Stepwise upsampling to audio waveform through fractional convolutional layers;

[0045] Use a multi-scale discriminator to evaluate waveform naturalness;

[0046] Jointly optimize the adversarial loss and the discriminator feature matching loss.

[0047] The present invention also discloses a speech synthesis system based on controllable latent variable modeling of semantic distillation, comprising:

[0048] Data acquisition module, used to obtain input text and speech training samples;

[0049] Speech coding module, used to generate continuous latent variable distribution parameters and sample them based on Mel spectrum features;

[0050] Semantic distillation module to align continuous latent variables with the teacher semantic vector of the self-supervised semantic model;

[0051] A text alignment module, which implements end-to-end mapping of text representations to latent variable distribution parameters;

[0052] Decoder module, used to reconstruct the continuous latent variable into a Mel spectrum;

[0053] Vocoder module, used to synthesize Mel spectrum into speech waveform;

[0054] Joint training module for optimizing distillation loss, alignment loss, and spectral reconstruction loss.

[0055] Furthermore, the speech coding module includes:

[0056] Normalization unit, used to perform mean-variance normalization of the Mel spectrum;

[0057] Convolutional coding unit, consisting of five layers of one-dimensional convolutional networks with residual connections;

[0058] A distribution parameter generation unit, used to output a mean vector and a logarithmic variance vector;

[0059] Reparameterized units for sampling continuous latent variables based on Gaussian distribution;

[0060] Regularization unit, used to calculate the KL divergence constrained latent variable space.

[0061] The present application also provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0062] The present application also provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of the above method when executed by a processor.

[0063] The beneficial effects of this application are:

[0064] First, the present invention adopts a variational autoencoder framework to generate continuous latent variables, retains the continuously changing information of speech features through Gaussian distribution parameters, avoids the hard truncation loss of discrete quantization, and solves the problem that traditional discrete coding maps continuous speech features to a finite codebook through quantization, resulting in the loss of fine-grained information such as phoneme details and intonation changes, and the reduction of the naturalness of the synthesized speech.

[0065] Second, the continuous latent variables in this invention achieve end-to-end gradient backpropagation through reparameterized sampling, replacing the non-differentiable operations of discrete modules and making the training process smoother. The KL divergence loss constrains the latent variable distribution to be close to the standard normal distribution, preventing degradation to a constant or localized aggregation, thereby improving convergence speed. This solves the problems of existing discrete modules, such as non-differentiability, the need for independent parameter adjustment, the complex training process, the susceptibility to local optimality, and poor stability.

[0066] Third, the text alignment module of the present invention directly maps the phoneme sequence into latent variable distribution parameters through a four-layer Transformer encoder, and uses a monotonic alignment search algorithm to dynamically establish temporal alignment between phonemes and frames. The inference stage does not require any audio participation or auxiliary modules, and can directly generate latent variables. This solves the problem that the existing methods need to rely on additional modules such as classifiers or lookup tables to realize the mapping of text to discrete latent variables during inference, resulting in high latency and complex deployment.

[0067] Fourthly, the semantic distillation module in the present invention forces the latent variables to be aligned with the HuBERT semantic vector in terms of frame-by-frame feature direction and sequence structure relationship through marginal cosine similarity loss and marginal distance matrix structure loss, which solves the problems of insufficient utilization of self-supervised features such as HuBERT in existing methods, weak semantic representation of latent variables, and proneness to semantic drift or syntactic errors.

[0068] Fifth, the decoder in this invention uses five layers of residual convolution combined with a bidirectional GRU to capture everything from phoneme-level local features to sentence-level long-term temporal dependencies. Combined with L1 loss and perceptual loss, it more accurately restores the high-level perceptual features of the Mel spectrum, solving the problem of traditional vocoders relying on a single loss function, resulting in insufficient waveform naturalness, especially poor performance in high-frequency details and long-term temporal dependencies. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 This is a schematic diagram of a method flow chart according to an embodiment of the present application.

[0070] Figure 2 A schematic diagram of the training process of a method according to an embodiment of the present application.

[0071] Figure 3 Schematic diagram of the reasoning flow of a method according to an embodiment of the present application.

[0072] The realization of the objectives, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0073] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0074] like Figure 1 As shown in FIG, the speech synthesis method based on controllable latent variable modeling of semantic distillation includes the following steps:

[0075] S1, obtain input text and speech training samples;

[0076] S2, extracting Mel-spectrogram features based on the speech training sample, and inputting the Mel-spectrogram features into a speech coding module to generate continuous latent variable distribution parameters;

[0077] S3, performing reparameterized sampling based on the continuous latent variable distribution parameter to obtain a continuous latent variable;

[0078] S4, inputting the Mel spectrum feature into the self-supervised semantic model in a frozen state to extract the teacher semantic vector;

[0079] S5, calculating the distillation loss between the continuous latent variable and the teacher semantic vector through the semantic distillation module;

[0080] S6, converting the input text into a phoneme sequence, inputting the sequence into a text encoder to generate a text latent representation, and mapping the text latent representation into a predicted latent variable distribution parameter through a text alignment module;

[0081] S7, constructing an alignment loss based on the predicted latent variable distribution parameter and the continuous latent variable distribution parameter output by the speech encoding module;

[0082] S8, inputting the continuous latent variable into a decoder module to generate a reconstructed Mel spectrum, and inputting the reconstructed Mel spectrum into a vocoder module to synthesize a speech waveform;

[0083] S9, jointly optimize the distillation loss, alignment loss and Mel spectrum reconstruction loss to train an end-to-end speech synthesis model.

[0084] According to the above steps S1-S9, the present invention builds an end-to-end speech synthesis framework, integrates continuous latent variable modeling, semantic distillation and text alignment technology, and realizes high-quality generation from input text to speech waveform, solves the problems of information loss, training instability and low text alignment efficiency existing in the existing discrete latent variable model, and improves the naturalness, semantic consistency and reasoning speed of speech synthesis.

[0085] The present invention adopts continuous latent variables (VAE framework) instead of discrete coding, enhances the semantic representation ability of latent variables through semantic distillation, and designs a text alignment module to realize direct mapping of text to latent variables, forming an end-to-end path of "text-latent variable-Mel spectrum-speech waveform", avoiding a series of problems caused by discretization.

[0086] In one embodiment, the step of extracting mel-spectrogram features from the speech training sample and inputting the mel-spectrogram features into a speech coding module to generate continuous latent variable distribution parameters includes:

[0087] S21, performing mean-variance normalization on the input Mel spectrum;

[0088] S22, input the normalized Mel spectrum into a five-layer one-dimensional convolutional network, perform layer normalization and activation operations after each convolution layer, and add residual connections;

[0089] S23, outputs the mean vector and logarithmic variance vector at the time frame level;

[0090] S24, constructing a Gaussian distribution based on the mean vector and the logarithmic variance vector, and obtaining a continuous latent variable through reparameterized sampling calculation;

[0091] S25, calculating the KL divergence between the Gaussian distribution and the standard normal distribution as a regularization loss.

[0092] According to the above steps S21-S25, this embodiment realizes efficient mapping from Mel-spectrogram features to continuous latent variable distribution through structural design and parameter optimization of the speech coding module, constructs a sampleable Gaussian distribution to generate continuous latent variables, and ensures the structuring and continuity of the latent variable space through KL divergence constraints, solves the information loss and training instability problems caused by quantization of existing discrete latent variable models, and provides high-quality intermediate representation for subsequent semantic distillation and text alignment.

[0093] In this embodiment, this module is the core component for implementing continuous latent variable modeling. Its task is to map the input audio feature sequence into sampleable latent distribution parameters, which serve as the intermediate semantic representation in the entire generation path. This module uses a variational autoencoder (VAE) structure to learn acoustic-semantic representations in a stable and controllable manner.

[0094] The input to this module is the sequence of mel spectra corresponding to each speech sample. The mel spectra are extracted using linear frequency mapping, with a dimension of 80, a frame shift of 10ms, and a frame length of 25ms. This results in a two-dimensional tensor of shape T × 80, where T is the number of time frames. The input is first subjected to uniform mean-variance normalization to suppress global differences in distribution due to factors such as speaker and recording conditions.

[0095] To encode the temporal information and high-dimensional features of audio, this module uses a deep convolutional neural network as the backbone encoder structure. The entire encoding network consists of five layers of one-dimensional convolution (1D convolution), each with a kernel size of 5 and 256 channels. The activation function uniformly uses GELU activation, and the convolution is followed by a LayerNorm normalization module to accelerate training and improve model stability. To enhance expressiveness and training depth, a residual connection mechanism is used between each convolution layer to ensure that the speech features do not suffer from vanishing gradients or excessive compression during transmission.

[0096] The encoder finally outputs a shape of T×2d z A tensor of d z is the latent variable dimension (set to 64 in this paper). The output tensor is divided into two parts, representing the mean vector of each time frame and the logarithmic variance vector During the training phase, a reparameterized sampling strategy is used to generate the latent variable z from the Gaussian distribution. The calculation formula is:

[0097]

[0098] This sampling mechanism can realize gradient backpropagation, making the distribution learning process end-to-end optimized.

[0099] In order to ensure that the learned latent variable space has good controllability and structural consistency, this model introduces the KL divergence loss term to constrain the encoder output distribution to the standard normal distribution. Keep it close. The KL loss is calculated on a frame-by-frame basis, and the overall loss is adjusted by multiplying the coefficient β in the total loss function to control excessive regularization pressure in the early stages of training and gradually improve the normalization of the latent variables.

[0100] In one embodiment, the step of calculating the distillation loss between the continuous latent variable and the teacher semantic vector by the semantic distillation module includes:

[0101] S51, aligning the continuous latent variable to the feature dimension of the teacher semantic vector through a linear projection layer;

[0102] S52, calculating the directional similarity between the projected student vector and the teacher semantic vector at each time frame;

[0103] S53, constructing the internal structure relationship similarity matrix between the student vector sequence and the teacher semantic vector sequence;

[0104] S54, according to the direction similarity deviation and the structure similarity deviation, the distillation loss is obtained by weighting.

[0105] According to the above steps S51-S54, this embodiment, through the design of a semantic distillation module, performs multi-level alignment on the continuous latent variables generated by the speech encoding module and the teacher semantic vectors extracted by the self-supervised semantic model (such as HuBERT). Through directional similarity constraints and structural relationship constraints, the latent variables are forced to learn semantic discrimination ability and context consistency, thereby solving the problems of insufficient semantic representation and weak context association of latent variables in the existing technology, and improving the semantic fidelity and syntactic rationality of the synthesized speech.

[0106] Semantic consistency in speech synthesis means that the synthesized speech must accurately express the semantic content of the text (such as the intonation of interrogative sentences and the emotional tendency of word meanings). However, existing methods often use self-supervised semantic features in a shallow auxiliary manner (such as only as additional input features), without deeply aligning the structure of the latent variable space and the semantic space. For example, when synthesizing "I like sunny days", traditional TTS models may fail to fully capture the emotional semantics of "like" in the latent variables, resulting in bland emotional expression in the speech. In addition, the quantitative nature of discrete latent variables makes it difficult to model continuous changes in semantics (such as the difference in degree from "hot" to "scorching"), which requires deep distillation of continuous latent variables and semantic features.

[0107] The semantic distillation module in this paper is designed to guide latent variables to learn semantically discriminative structural representations during the training phase, enabling them to reconstruct speech while retaining distributional features consistent with the self-supervised semantic model. To this end, the present invention introduces the output of a HuBERT-based teacher model as a soft label signal to impose feature alignment constraints on the continuous latent variables output by the speech encoding module.

[0108] The input of the module is a Mel spectrum audio sequence consistent with the encoder, which is processed by the frozen HuBERT model and the semantic vector of a Transformer intermediate layer is extracted as the target representation. Assume that the representation is F∈R T×df , where T is the number of frames, df is the HuBERT feature dimension (e.g. 768). Meanwhile, the latent variable output from the VAE encoder is Z∈R T ×dz , where d z <d f .

[0109] To perform alignment, we first pass a set of trainable linear projection parameters W∈R d z×d f , projecting the latent variable space to the same feature dimension as the teacher representation, recorded as:

[0110] Z′=ZW;

[0111] in is the aligned student vector.

[0112] Semantic distillation is jointly optimized using two types of structural loss functions, namely marginal cosine similarity loss and marginal distance matrix structure loss, which are defined as follows:

[0113] 1. Marginal cosine similarity loss

[0114] This loss term measures the degree of alignment between the student and the teacher’s feature directions at each frame. For each time frame t∈[1,T], the student representation is defined as z′ t , the teacher is represented by f t , then the loss is defined as:

[0115]

[0116] Where m1 is the minimum margin threshold for cosine alignment, with a typical value of 0.8;

[0117] ReLU ensures that only frames that fail to meet the similarity criteria are penalized, allowing the model to focus on low-similarity regions.

[0118] 2. Marginal distance matrix structure loss

[0119] This loss term measures the similarity between the internal structural relationships of two feature sequences. By constructing two T×T cosine similarity matrices, one from Z′ and the other from F, for any frame pair (i, j), the relative structural difference is defined as the feature cosine similarity difference between the two frames. The loss form is as follows:

[0120]

[0121] Among them, m2 is the tolerance boundary of structural difference, which encourages the model to retain the overall structural information while maintaining semantic discrimination ability. Only the part of the structural difference exceeding this margin will be penalized, thereby improving convergence efficiency.

[0122] 3. Overall loss and training method

[0123] The final distillation loss of this module is composed of the weighted sum of the above two components:

[0124]

[0125] Among them, λ1 and λ2 control the weights of directional alignment and structural alignment respectively. In the early stage of training, structural alignment can be given priority, and the proportion of directional alignment can be increased in the later stage to enhance local fine expression.

[0126] This semantic distillation module participates in the training phase as an additional loss, without changing the main network's structure and process. During inference, the module is completely removed, introducing no additional computational overhead. This is achieved through explicit semantic guidance of the latent variable space structure.

[0127] In one embodiment, the steps of converting the input text into a phoneme sequence, inputting the sequence into a text encoder to generate a text latent representation, and mapping the text latent representation into a predicted latent variable distribution parameter through a text alignment module include:

[0128] S61 processes the phoneme sequence through a four-layer Transformer encoder to generate context-aware text representation;

[0129] S62, mapping the text representation into a predicted mean vector and a predicted logarithmic variance vector at the phoneme level;

[0130] S63, calculating the KL divergence between the predicted distribution and the true latent variable distribution output by the speech coding module;

[0131] S64, uses a monotone alignment search algorithm to construct the optimal alignment path between phonemes and time frames;

[0132] S65 , projecting the prediction result distribution to the time frame level distribution according to the alignment path.

[0133] According to the above steps S61-S65, the present invention establishes a direct mapping relationship between the input text and the continuous latent variable space through the design of the text alignment module, realizes end-to-end prediction from the phoneme sequence to the latent variable distribution, solves the inefficiency problem of the existing technology in relying on additional modules (such as classifiers or lookup tables) for text-latent variable mapping during the inference stage, and improves the real-time performance of speech synthesis and the ease of system deployment.

[0134] The text alignment module in this invention aims to achieve end-to-end text-driven control of the speech generation system. By establishing a direct mapping relationship between text and latent variable distribution, the text alignment module can generate continuous latent variables relying solely on input text during the inference phase, thereby bypassing the complex components of traditional methods that rely on lookup tables, classifiers, or phoneme prior encoding, significantly improving generation efficiency and system deployability.

[0135] The input of the text alignment module in the present invention is the text representation of the corresponding speech, which is first processed into a phoneme sequence by a standard phoneme transcription tool (such as ESPNET or Montreal Forced Aligner). Each phoneme is mapped to a length d p The trainable embedding vectors of p The embedded sequence is fed into a text encoder consisting of a multi-layer Transformer model with four layers of self-attention modules, each with an 8-head attention mechanism and a feedforward sublayer, and a hidden dimension of 256. The encoder as a whole is capable of modeling both local pronunciation patterns and global language context.

[0136] The encoder outputs a context-aware representation corresponding to each phoneme position, which is mapped into two output branches that match the latent variable distribution parameters through a linear transformation layer, namely and Used to construct the distribution of the predicted latent variable. Since the phoneme length L is not consistent with the audio frame length T, an explicit frame-phoneme alignment strategy must be introduced to ensure the rationality of the alignment.

[0137] To this end, the present invention adopts the Monotonic Alignment Search (MAS) algorithm based on the confidence of the attention matrix. During the training phase, the optimal alignment path is constructed based on the KL distance between the latent variable distribution output by the speech encoder and the predicted distribution of the text encoder. The specific process is as follows:

[0138] Calculate the KL divergence between the predicted distribution of each text position and the true latent variable distribution of the speech;

[0139] Construct a two-dimensional cost matrix (phoneme × frame);

[0140] Apply a dynamic programming algorithm (such as Viterbi or DTW variants) to find the minimum cost path that satisfies monotonicity and coverage constraints;

[0141] The alignment path is used as a mask to project the text prediction distribution to each frame position and generate a frame-level latent variable distribution estimate.

[0142] The predicted distribution is then aligned with the true latent variable distribution output by the speech encoder for supervised training, using KL divergence loss and MSE dual loss for joint optimization. The loss term is defined as:

[0143]

[0144] where μ vae , The output of the speech encoder from the audio path, α is a control term.

[0145] In the inference phase, only the text encoder path needs to be retained. After the input text is phoneme-processed, it is directly passed through this module to generate the latent variable score (μ, σ 2 ), and use the mean as the decoding input:

[0146] z inference =μ text ;

[0147] Full-path text-driven synthesis can be achieved without any audio participation or auxiliary model intervention.

[0148] In one embodiment, the step of constructing an alignment loss based on the predicted latent variable distribution parameter and the continuous latent variable distribution parameter output by the speech encoding module includes:

[0149] S71, mapping the continuous latent variable to an intermediate feature dimension through a linear layer;

[0150] S72, process the features sequentially through five layers of residual convolution blocks, each layer including convolution, layer normalization, activation and residual connection;

[0151] S73, the convolution output is fed into a bidirectional gated recurrent unit to capture long-term temporal dependencies;

[0152] S74, converting the gated recurrent unit output into the target Mel-spectrogram dimension through a linear transformation layer;

[0153] S75 , calculating the L1 distance between the reconstructed Mel spectrum and the original Mel spectrum and the vocoder discriminator feature distance as the perceptual loss.

[0154] According to the above steps S71-S75, this embodiment achieves high-precision reconstruction from continuous latent variables to Mel-spectrum through the structural design of the decoder module and the optimization of the loss function, solving the problems of semantic information loss and rough restoration of acoustic features during latent variable decoding in the prior art, ensuring that the acoustic details (such as pitch and resonance peaks) of the synthesized speech are consistent with the original speech, and at the same time improving the naturalness of the timbre through perceptual loss.

[0155] The main task of the decoder module in this invention is to convert the continuous latent variables generated by the encoder module into It is converted into an intermediate representation with complete speech features, namely the Mel spectrum, for time domain waveform recovery of the subsequent vocoder. As the decoder part of the variational autoencoder architecture, the decoder module must be designed to balance the accuracy of speech content restoration and the decodability of the latent variable structure.

[0156] In this paper, the decoder module adopts a deep residual convolutional structure designed symmetrically with the encoder to ensure consistency in the mapping between the temporal dimension and the semantic information. The input latent variable sequence is first upsampled to the intermediate hidden layer channel dimension (set to 256 dimensions) through a linear mapping layer, and then passes through 5 layers of residual convolution blocks in sequence. Each residual block contains the following substructure:

[0157] One-dimensional convolution (kernel size = 5, stride = 1, padding = 2), the number of output channels is 256;

[0158] LayerNorm normalization;

[0159] GELU nonlinear activation;

[0160] Dropout layer (dropoutrate=0.1);

[0161] Identity mapping residual connection.

[0162] After each convolutional block, the backbone feature channels remain unchanged, ensuring temporal alignment and preserving feature information. Furthermore, to enhance perception of audio temporal structure, a bidirectional gated recurrent unit (Bi-GRU) is introduced after the fifth convolutional layer to capture long-range dependencies and temporal dynamics. The hidden dimension of this GRU layer is 128, and the output sequence length remains T.

[0163] The GRU output is then fed into a linear transformation layer, which converts the dimension to the target Mel spectrum dimension (e.g., 80). The Tanh activation function normalizes the numerical range to a suitable range for acoustic representation. The final output is a continuous Mel spectrum sequence with a shape of T × 80.

[0164] In order to ensure that the generated Mel spectrum has high-quality restoration capabilities, the decoder module uses reconstruction loss for supervision during the training phase. The loss function consists of two parts:

[0165] L1 reconstruction loss: used to measure the frame-by-frame difference between the generated spectrum and the original Mel spectrum to ensure the restoration of basic audio content;

[0166] Perceptual loss: Select the intermediate feature layer of the discriminator of a pre-trained vocoder (such as HiFi-GAN) to measure the high-level perceptual gap between the generated spectrum and the true spectrum in the feature space to enhance the timbre and clarity.

[0167] The joint loss is defined as follows:

[0168]

[0169] Where M is the true Mel spectrum, is the decoder output, D l () is the intermediate feature of the llth layer of the vocoder discriminator, and β is the perceptual loss weight.

[0170] The decoder module forms a closed VAE loop with the encoder module in the system architecture and is the key path for latent variable learning during the training phase. During the inference phase, its input comes from the continuous latent variable z generated by the text alignment module, which can directly generate an intermediate spectrum representation for the subsequent vocoder module to restore the high-fidelity speech waveform.

[0171] like Figure 2 As shown in the figure, a schematic diagram of the overall training process of the system with the input voice "What's the weather like today?" is shown to facilitate understanding of the overall operation logic of the system. Figure 3 It shows a schematic diagram of the overall reasoning process of a system with the input voice "What's the weather like today?" to facilitate understanding of the overall reasoning logic of the system.

[0172] In one embodiment, the step of jointly optimizing the distillation loss, alignment loss, and mel-spectrogram reconstruction loss to train an end-to-end speech synthesis model includes:

[0173] S91, uses a generative adversarial network architecture to process and reconstruct the Mel spectrum;

[0174] S92, gradually upsamples to an audio waveform through fractional convolutional layers;

[0175] S93, using a multi-scale discriminator to evaluate waveform naturalness;

[0176] S94, jointly optimize the adversarial loss and the discriminator feature matching loss.

[0177] According to the above steps S91-S94, this embodiment converts the Mel spectrum output by the decoder into a high-fidelity time-domain speech waveform through the architectural design and multi-loss joint optimization of the vocoder module, thereby solving the shortcomings of existing neural vocoders in waveform naturalness, real-time performance, and multi-scale detail restoration, ensuring that the synthesized speech is close to real-person pronunciation in dimensions such as pitch, timbre, and rhythm, while meeting the low-latency requirements of practical applications.

[0178] The vocoder module is responsible for converting the intermediate Mel spectra generated by the decoder into the final time-domain speech waveform and is the final output module of the speech synthesis system. To ensure high quality, naturalness, and real-time performance, this paper uses a neural vocoder based on the Generative Adversarial Network (GAN) architecture, using the HiFi-GAN structure as a reference design, and combines it with a multi-scale discriminator for end-to-end optimization.

[0179] Specifically, the vocoder module input is the Mel-spectral feature sequence output by the decoder, with a shape of T × 80, where T is the number of time frames. The input first passes through several layers of one-dimensional convolution and residual blocks for feature extraction and upsampling, gradually increasing the temporal resolution to the audio sampling rate. The residual block structure used in the module includes convolutional layers, activation layers, and normalization layers to effectively capture local temporal dependencies.

[0180] During the upsampling process, fractional convolution or transposed convolution is used to refine the Mel spectrum features layer by layer into a high-resolution time domain feature representation. The final convolution layer outputs a single-channel waveform signal as the generated audio waveform.

[0181] To improve the quality of synthesized sound, the vocoder incorporates a multi-scale discriminator system during training, including a period discriminator and a multi-scale spectral discriminator. Each discriminator assesses and provides feedback on the naturalness and detail of the generated waveform based on different time scales and frequency characteristics. Through adversarial training strategies, the vocoder continuously improves the realism and consistency of the generated speech.

[0182] The loss function combines the following parts:

[0183] Adversarial loss (GAN loss): Pushes the generated waveform closer to the real distribution;

[0184] Feature Matching Loss: The multi-layer feature space distance within the discriminator promotes detail restoration and stable training.

[0185] By jointly optimizing the above losses, the vocoder module achieves high-fidelity, low-latency speech waveform synthesis, meeting the dual requirements of naturalness and real-time performance in practical applications.

[0186] The present invention also discloses a speech synthesis system based on controllable latent variable modeling of semantic distillation, comprising:

[0187] Data acquisition module, used to obtain input text and speech training samples;

[0188] Speech coding module, used to generate continuous latent variable distribution parameters and sample them based on Mel spectrum features;

[0189] Semantic distillation module to align continuous latent variables with the teacher semantic vector of the self-supervised semantic model;

[0190] A text alignment module, which implements end-to-end mapping of text representations to latent variable distribution parameters;

[0191] Decoder module, used to reconstruct the continuous latent variable into a Mel spectrum;

[0192] Vocoder module, used to synthesize Mel spectrum into speech waveform;

[0193] Joint training module for optimizing distillation loss, alignment loss, and spectral reconstruction loss.

[0194] The core of the speech synthesis system based on controllable latent variable modeling with semantic distillation in this paper lies in an end-to-end speech synthesis system, which comprises five core components: an encoder, a semantic distillation module, a text alignment module, a decoder, and a vocoder. To achieve high-quality and stable joint training, this system designs a composite total loss function consisting of multiple sub-losses to coordinate the training objectives of each module, promoting the semantic richness of the latent variable, speech restoration capability, and text alignment accuracy.

[0195] Among them, the total loss function is defined as:

[0196] L total =αL recon +βL KL +γL distill +δL text-align +ηL GAN

[0197] The functions of each sub-loss are described as follows:

[0198] Reconstruction loss L recon : Mainly constrains the similarity between the decoder output Mel spectrum and the true Mel spectrum, usually using L1 norm loss and perceptual loss to ensure accurate restoration of acoustic features;

[0199] KL divergence loss L KL : Constrain the speech encoder latent variable distribution to be consistent with the standard normal distribution, ensuring the structure and continuity of the latent variable space and preventing degradation;

[0200] Semantic distillation loss L distill : It consists of marginal cosine similarity loss and marginal distance matrix structure loss, which promotes the deep alignment of latent variables with the semantic features output by the self-supervised model HuBERT, and enhances the semantic discrimination ability;

[0201] Text alignment loss L text-align : Through KL divergence and mean squared error, the distribution of latent variables predicted by the text encoder is promoted to match the distribution of latent variables output by the speech encoder, achieving monotonic alignment of the text and latent variable spaces;

[0202] Adversarial loss L GAN : For the vocoder part, the generative adversarial loss and feature matching loss of the multi-scale discriminator are used to improve the naturalness of the waveform and the details of the sound quality.

[0203] The hyperparameters α, β, γ, δ, and η are used to balance the contribution of each sub-loss. Typical values ​​are obtained by tuning the validation set to ensure stable training and optimal generation results.

[0204] This total loss function achieves joint optimization of multiple objectives, ensuring the structural rationality of the latent variable space, rich semantic expression, and high-fidelity reconstruction of the audio signal, effectively improving the performance and generalization ability of the end-to-end text-driven speech synthesis system.

[0205] In one embodiment, the speech encoding module includes:

[0206] Normalization unit, used to perform mean-variance normalization of the Mel spectrum;

[0207] Convolutional coding unit, consisting of five layers of one-dimensional convolutional networks with residual connections;

[0208] A distribution parameter generation unit, used to output a mean vector and a logarithmic variance vector;

[0209] Reparameterized units for sampling continuous latent variables based on Gaussian distribution;

[0210] Regularization unit, used to calculate the KL divergence constrained latent variable space.

[0211] The present application also provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0212] The present application also provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of the above method when executed by a processor.

[0213] Those skilled in the art will understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, value library or other media provided in this application and used in the embodiments may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct RAMbus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM).

[0214] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, apparatus, article, or method comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, apparatus, article, or method. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, apparatus, article, or method comprising the element.

[0215] The above description is only a preferred embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent results or equivalent process transformations made using the contents of the present invention description and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A speech synthesis method based on controllable latent variable modeling with semantic distillation, characterized in that: The following steps are involved: Get input text and speech training samples; Extracting Mel spectrum features according to the speech training sample, and inputting the Mel spectrum features into a speech coding module to generate continuous latent variable distribution parameters; Performing reparameterized sampling based on the continuous latent variable distribution parameter to obtain a continuous latent variable; Inputting the Mel spectrum features into the self-supervised semantic model in a frozen state to extract the teacher semantic vector; Calculating the distillation loss between the continuous latent variable and the teacher semantic vector through a semantic distillation module; Convert the input text into a phoneme sequence and input it into a text encoder to generate a text hidden layer representation, and map the text hidden layer representation into a predicted latent variable distribution parameter through a text alignment module; Constructing an alignment loss based on the predicted latent variable distribution parameters and the continuous latent variable distribution parameters output by the speech encoding module; Inputting the continuous latent variable into a decoder module to generate a reconstructed Mel spectrum, and inputting the reconstructed Mel spectrum into a vocoder module to synthesize a speech waveform; The distillation loss, alignment loss, and Mel spectrum reconstruction loss are jointly optimized to train an end-to-end speech synthesis model.

2. The speech synthesis method based on controllable latent variable modeling based on semantic distillation according to claim 1, characterized in that: The step of extracting Mel spectrum features according to the speech training sample and inputting the Mel spectrum features into the speech coding module to generate continuous latent variable distribution parameters includes: Perform mean-variance normalization on the input Mel spectrum; The normalized Mel spectrum is input into a five-layer one-dimensional convolutional network. After each convolution layer, layer normalization and activation operations are performed, and residual connections are added. Output the mean vector and log variance vector at the time frame level; Constructing a Gaussian distribution based on the mean vector and the logarithmic variance vector, and obtaining a continuous latent variable by reparameterized sampling; The KL divergence between the Gaussian distribution and the standard normal distribution is calculated as the regularization loss.

3. The speech synthesis method based on controllable latent variable modeling based on semantic distillation according to claim 1, characterized in that: The step of calculating the distillation loss between the continuous latent variable and the teacher semantic vector by the semantic distillation module includes: Aligning the continuous latent variable to the feature dimension of the teacher semantic vector through a linear projection layer; Calculate the directional similarity between the projected student vector and the teacher semantic vector at each time frame; Construct the internal structure relationship similarity matrix between the student vector sequence and the teacher semantic vector sequence; The distillation loss is obtained by weighting the direction similarity deviation and the structure similarity deviation.

4. The speech synthesis method based on controllable latent variable modeling based on semantic distillation according to claim 3, characterized in that: The steps of converting the input text into a phoneme sequence, inputting the sequence into a text encoder to generate a text hidden layer representation, and mapping the text hidden layer representation into a prediction latent variable distribution parameter through a text alignment module include: Processing phoneme sequences through a four-layer Transformer encoder to generate context-aware text representations; Mapping the text representation into a predicted mean vector and a predicted log-variance vector at the phoneme level; Calculate the KL divergence between the predicted distribution and the true latent variable distribution output by the speech encoding module; A monotone alignment search algorithm is used to construct the optimal alignment path between phonemes and time frames; Project the prediction result distribution to the time frame level distribution according to the alignment path.

5. The speech synthesis method based on controllable latent variable modeling based on semantic distillation according to claim 1, characterized in that: The step of constructing an alignment loss based on the predicted latent variable distribution parameter and the continuous latent variable distribution parameter output by the speech encoding module includes: Mapping the continuous latent variable to an intermediate feature dimension through a linear layer; The features are processed sequentially through five layers of residual convolution blocks, each layer including convolution, layer normalization, activation and residual connection; Feed the convolutional output into a bidirectional gated recurrent unit to capture long-term temporal dependencies; The gated recurrent unit output is converted to the target Mel-spectrogram dimension through a linear transformation layer; The L1 distance between the reconstructed Mel spectrum and the original Mel spectrum and the vocoder discriminator feature distance are calculated as the perceptual loss.

6. The speech synthesis method based on controllable latent variable modeling based on semantic distillation according to claim 1, characterized in that: The step of jointly optimizing the distillation loss, alignment loss, and Mel spectrum reconstruction loss to train the end-to-end speech synthesis model includes: Generative adversarial network architecture is used to process and reconstruct the Mel spectrum; Stepwise upsampling to audio waveform through fractional convolutional layers; Use a multi-scale discriminator to evaluate waveform naturalness; Jointly optimize the adversarial loss and the discriminator feature matching loss.

7. A speech synthesis system based on controllable latent variable modeling with semantic distillation, characterized by: include: Data acquisition module, used to obtain input text and speech training samples; Speech coding module, used to generate continuous latent variable distribution parameters and sample them based on Mel spectrum features; Semantic distillation module to align continuous latent variables with the teacher semantic vector of the self-supervised semantic model; A text alignment module, which implements end-to-end mapping of text representations to latent variable distribution parameters; Decoder module, used to reconstruct the continuous latent variable into a Mel spectrum; Vocoder module, used to synthesize Mel spectrum into speech waveform; Joint training module for optimizing distillation loss, alignment loss, and spectral reconstruction loss.

8. The speech synthesis system based on controllable latent variable modeling based on semantic distillation according to claim 7, characterized in that: The speech coding module comprises: Normalization unit, used to perform mean-variance normalization of the Mel spectrum; Convolutional coding unit, consisting of five layers of one-dimensional convolutional networks with residual connections; A distribution parameter generation unit, used to output a mean vector and a logarithmic variance vector; Reparameterized units for sampling continuous latent variables based on Gaussian distribution; Regularization unit, used to calculate the KL divergence constrained latent variable space.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images