Multi-speaker emotion speech synthesis method and system based on potential diffusion model

Abstract

The invention relates to the technical field of artificial intelligence and emotional speech synthesis, in particular to a multi-speaker emotional speech synthesis method and system based on a potential diffusion model. In the emotional speech synthesis method, a variational auto-encoder comprising an encoder Enc and a decoder Dec is firstly constructed, a speech linear spectrum is mapped to a potential space through the Enc, a potential acoustic representation z0 is obtained through re-parametric sampling, and then the potential acoustic representation z0 is mapped into a waveform through the Dec. Receiving a target text, and converting the target text into a frame-level text representation mu; then receiving a reference voice and a speaker category label s, and purifying to obtain an emotion embedded surface; and then a diffusion model is constructed in the potential space based on mu, s and face, score functions with and without emotional conditions are calculated and weighted fusion is carried out, and target potential acoustic representation is output. And finally, mapping the representation input Dec into a waveform to obtain a target emotional speech, thereby realizing multi-speaker emotional speech synthesis with controllable emotions and high naturalness.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and emotional speech synthesis technology, specifically a multi-speaker emotional speech synthesis method and system based on a latent diffusion model. Background Technology

[0002] Speech synthesis technology is an important component of human-computer interaction, capable of converting text information into natural and fluent speech output. With the continuous development of applications such as intelligent voice assistants, virtual digital humans, and audiobooks, simply generating clear and understandable speech is no longer sufficient to meet practical needs. How to accurately express emotions and adapt to different speaker timbres during speech synthesis has become one of the current research hotspots in the field of speech synthesis.

[0003] In multi-speaker emotional speech synthesis tasks, models typically need to simultaneously model text content, speaker timbre features, and emotional features. However, due to the inherent complexity of speech signals, emotional information and speaker timbre information are often highly coupled at the acoustic level. Existing techniques tend to encode speaker-related information along with emotional features during extraction, leading to speaker leakage during cross-speaker emotional transfer or synthesis. This means that synthesized speech exhibits timbre shifts alongside emotional changes, affecting speaker consistency and naturalness. To alleviate these issues, some existing methods introduce additional constraint assumptions or explicit feature separation mechanisms to distinguish between emotional and timbre information. However, because multiple factors are intertwined in speech signals, these methods usually rely on strong prior assumptions or complex network structures, making it difficult to achieve stable and universal cross-speaker emotional synthesis while ensuring accuracy in emotional expression.

[0004] On the other hand, controlling the intensity of emotion is also one of the key technical challenges in multi-speaker emotional speech synthesis. Emotional speech involves not only the differentiation of emotion categories but also the continuous variation in the strength of emotional expression. However, in real-world speech datasets, accurate annotation of emotion intensity is often costly and lacks a unified standard, making it difficult for most existing methods to directly model emotion intensity. To achieve emotion intensity control, some technical solutions manually adjust specific acoustic parameters or introduce additional control networks to realize emotion changes. However, such methods usually rely on empirical parameter settings, the adjustment process is complex, and they are prone to causing speech quality degradation or unstable emotional expression when the emotion intensity changes significantly, limiting their application in practical systems.

[0005] In summary, existing multi-speaker emotional speech synthesis technologies still have shortcomings in effectively decoupling emotion from speaker timbre features and in flexibly and stably controlling the intensity of emotion. Summary of the Invention

[0006] To address the technical problems of unstable cross-speaker emotion transfer in multi-speaker emotional speech synthesis caused by speaker interference and weak emotion discriminability in emotion embedding, this invention provides a method for multi-speaker emotional speech synthesis. Building upon the refinement process, and to solve the technical problems of emotion coupling with text and speaker, uncontrollable emotion intensity, and low naturalness of synthesized speech in multi-speaker emotional speech synthesis, this invention provides a method for multi-speaker emotional speech synthesis based on a latent diffusion model. Based on the above two methods, this invention also provides a multi-speaker emotional speech synthesis system based on a latent diffusion model that applies both methods.

[0007] To achieve the above objectives, the present invention provides the following technical solution: An emotion embedding purification method for multi-speaker emotion-based speech synthesis includes the following purification steps: A1. Receive reference speech that matches and does not match the target text, and convert it into Mel spectrogram; A2. Two sentiment encoders with shared parameters are used to extract features from each Mel spectrum, outputting the sentiment matching embedding z. i Emotional mismatch embedding z j ; A3. Calculate z j The mean μ(z) j ) and variance σ(z) j ), z i variance σ(z) i ) and mean μ(z i ),pass The emotional embedding e is calculated pure ; A4, e pure The input is fed into a pre-built classifier, and the target emotion label of the reference speech is used as a supervision signal. The parameters of the emotion encoder and the emotion classifier are updated based on error backpropagation. A5, in z i A gradient inversion layer and a speaker classifier are inserted into the transmission path. The gradient inversion layer scales the classification error in reverse and then sends it back to the emotion encoder, causing the emotion encoder parameter update direction to deviate from the speaker classification target. A6. Iterate through steps A3-A5 until the stopping condition is met. Then, during the inference phase, use only the emotion encoder to extract the emotion embedding e from the Mel spectrum of the reference speech. pure .

[0008] As a further improvement to the above scheme: the reference speech includes speech segments that match the target text content of the reference speech itself, and speech segments that do not match the target text content of the reference speech itself.

[0009] As a further improvement to the above scheme, step A4 is as follows: Construct an emotion classifier consisting of three cascaded linear layers, embedding the emotion into e. pure The input is a sentiment classifier, with the target sentiment label c corresponding to the reference speech as the supervision signal, and the cross-entropy loss function is used. Calculate the sentiment classification error, and based on the sentiment classification error, backpropagate to synchronously update the parameters of each sentiment encoder and sentiment classifier, thereby strengthening e. pure The ability to distinguish emotional categories; among them, L c For sentiment classification error, log is the natural logarithm; p(c|e pure For a given emotion embedding e pure When, the probability of the corresponding true sentiment label c.

[0010] As a further improvement to the above scheme, the specific content of step A5 is as follows: In z i A gradient inversion layer and a single-layer linear speaker classifier are inserted into the transmission path, and the z-order of the speech after passing through the gradient inversion layer is processed. i Input the speaker classifier, use the speaker category label s corresponding to the reference speech as the supervision signal, and employ the cross-entropy loss function. The speaker classification error is calculated; during backpropagation, the gradient inversion layer scales the speaker classification error backward and then feeds it back to the emotion encoder, causing the emotion encoder's parameter update direction to deviate from the speaker classification target, so that the emotion encoder can remove z. i The speaker's identity information included; among which, L s Speaker classification error; p(s|z) i ) is given z i When, the probability corresponding to the real speaker category label s.

[0011] As a further improvement to the above scheme, step A6 is as follows: Joint sentiment classification error L e Speaker classification error L s Construct the total loss function L total = L e +α·L s , where α is the corresponding loss L s The weight coefficient is 0.02; iterative optimization is performed based on this total loss function: in each iteration, steps A3-A5 are repeated, and all parameters of the emotion encoder, emotion classifier, and speaker classifier are updated synchronously through backpropagation; when the total loss L total The iteration stops when the convergence reaches the preset threshold.

[0012] A multi-speaker emotional speech synthesis method based on a latent diffusion model includes the following synthesis steps: B1. Construct a variational autoencoder including encoder Enc and decoder Dec. Take the linear spectrum of the input speech, map the linear spectrum to the latent space through Enc, obtain the latent acoustic representation z0 based on reparameterized sampling, and map z0 to the time-domain speech waveform through Dec. B2. Receive the target text, convert it into a phoneme-level text representation, and upsample it to a frame-level text representation μ. B3. Receive the reference speech and its corresponding speaker category label s, and refine the reference speech using the aforementioned emotion embedding refinement method to obtain the emotion embedding e. pure ; B4, based on μ, s, and e pure A latent diffusion model is constructed in the latent space; in the latent diffusion model, score functions with and without emotional conditions are calculated respectively, and the target latent acoustic representation is output based on the weighted fusion of the two score functions. B5. Input the latent acoustic representation of the target into Dec, which maps it to a time-domain speech waveform to obtain the target emotional speech.

[0013] As a further improvement to the above solution, the sub-step of step B1 is as follows: B11. Construct a variational autoencoder (VAE) consisting of an encoder Enc and a decoder Dec. The input feature of the VAE is the linear spectrum y obtained by the short-time Fourier transform of the speech signal. B12. Extract features from the linear spectrum y using encoder Enc, map it to a low-dimensional latent space, and output the mean vector μ of the latent variable z. z Sum of variance vectors Construct the posterior probability distribution of the latent variable z. ,in, Given an input y, the posterior probability distribution of the latent variable z; It follows a normal distribution; B13. The latent acoustic representation z0 is obtained by sampling from the posterior probability distribution based on the reparameterized sampling method; B14. The decoder Dec employs an end-to-end waveform generation structure, directly mapping the latent acoustic representation z0 to a time-domain speech waveform. ; B15. During the training phase, the variational autoencoder optimizes the model parameters by maximizing the variational lower bound and trains the loss function L. VAE Including Mel spectrum reconstruction loss, KL divergence loss, and adversarial loss: ; In the formula, L1 is the L1 loss function, x is the real speech waveform, and Mel(·) is the Mel-spectrum transformation function; D KLLet p(z) be the KL divergence loss function, and let p(z) be the standard normal distribution of the latent variable z. B16. After training is complete, freeze all model parameters of the variational autoencoder. Its output latent acoustic representation is used for the modeling and generation of subsequent latent diffusion models.

[0014] As a further improvement to the above scheme, the sub-steps of step B2 are as follows: B21. Receive the target text corresponding to the speech to be synthesized, and convert the target text into a standard phoneme sequence; B22. Construct a text encoder. Input the standard phoneme sequence into the text encoder for feature encoding and output a phoneme-level text representation h. text ; B23. Construct a duration predictor. The duration predictor uses the true duration at the phoneme level obtained after aligning the training speech data using the Montreal forced alignment tool as the supervision signal, and represents the h in phoneme-level text. text As input, the system learns to establish a mapping relationship between phoneme-level text representations and phoneme durations, which is used to predict the duration of each phoneme. B23. During the text upsampling phase, based on the phoneme duration information output by the duration predictor, the phoneme-level text representation h is processed. text Extending the processing along the time dimension involves copying and stretching the phoneme-level text representation h. text To frame-level resolution, the phoneme-level text representation h text Expanded into a frame-level text representation μ aligned with the temporal dimension of subsequent acoustic features; During the inference phase, the duration predictor does not rely on true alignment information; it directly uses the phoneme-level text representation h corresponding to the input target text. text Predict the duration of phonemes and complete the phoneme-level text representation h based on the predicted duration. text The upsampling process to frame-level text representation μ; The frame-level text representation μ is used as the text conditional input for the subsequent potential diffusion model.

[0015] As a further improvement to the above scheme, the sub-steps of step B4 are as follows: B41. The low-dimensional latent space constructed by the variational autoencoder (VAE) with the model parameters frozen after training is used as the modeling space. The encoder of the variational autoencoder (VAE) has mapped the high-dimensional acoustic features to the low-dimensional latent space and output the latent acoustic representation. B42. Construct a latent diffusion model in a low-dimensional latent space. The denoising network for the latent diffusion model adopts a neural network based on the DiT structure. B43, frame-level text representation μ, speaker category labels s, and refined sentiment embedding e pureAs a joint conditional input latent diffusion model, the latent diffusion model describes the forward and backward diffusion processes through continuous-time stochastic differential equations. The forward diffusion process involves progressively injecting Gaussian noise into the latent acoustic representation z0 output by the variational autoencoder (VAE), and embedding emotion e with a 20% probability. pure Set as a vector of all zeros This allows the model to learn to generate without emotional conditions; the backdiffusion process employs a deterministic sampling method based on probability flow ordinary differential equations. B44. During the backdiffusion process, the denoising network calculates two types of score functions: one is the score function s with sentiment conditions. θ (x t ,t,μ,s,e pure The scoring function takes the latent variable x at time t as its basis. t Time step t, frame-level text representation μ, speaker category label s, and refined sentiment embedding e pure The input is 1; the second is a scoring function s without emotional conditions. θ (x t ,t,μ,s, The scoring function takes the latent variable x at time t as its basis. t The inputs are time step t, frame-level text representation μ, and speaker category label s; a guidance coefficient γ is introduced to weight and fuse the two types of scoring functions to construct an emotion guidance scoring function. for: ; The intensity of emotional embedding's guidance on the sampling process is controlled by adjusting the guidance coefficient γ value; B45. Use the emotion-guided scoring function to replace the original scoring function in the latent variable update process of backdiffusion until the backdiffusion sampling is completed, and finally output the target latent acoustic representation that meets the requirements of text content, target speaker timbre and emotion intensity.

[0016] A multi-speaker emotional speech synthesis system based on a latent diffusion model includes a processor and a memory. The memory stores a computer program. When the computer program is executed by the processor, it implements the aforementioned emotion embedding purification method and / or the aforementioned multi-speaker emotional speech synthesis method based on a latent diffusion model.

[0017] Compared with the prior art, the beneficial effects of the present invention are: 1. In the purification method of this invention, firstly, reference speech that matches and does not match the target text is received and converted into Mel spectra to provide suitable input for subsequent extraction of pure sentiment features. Next, two sentiment encoders with shared parameters are used to extract features from the two types of Mel spectra respectively, outputting sentiment matching embeddings z. i Emotional mismatch embedding zj This ensures the consistency of the emotion embedding extraction logic, laying the foundation for subsequent emotion feature optimization. Secondly, z is calculated. j and z i The mean and variance of the mean are used to obtain the sentiment embedding e through the corresponding formula. pure This achieves preliminary standardization of emotional characteristics and strengthens emotional consistency; e pure By inputting a pre-built sentiment classifier and using the target sentiment label as a supervision signal, the parameters of the sentiment encoder and classifier are updated through error backpropagation, effectively enhancing the sentiment discrimination capability of the sentiment embedding. Furthermore, in z... i A gradient inversion layer and a speaker classifier are inserted into the propagation path. The error is calculated using the speaker category label as a supervision signal. During backpropagation, the gradient inversion layer scales the error inversely before propagating it back, forcing the emotion encoder parameters to deviate from the speaker classification target, thus eliminating speaker interference. Finally, the iteration continues until a stopping condition is met, outputting the refined emotion embedding e. pure This effectively solves the problems of speaker interference and weak emotional discriminativeness in emotional embedding.

[0018] 2. In the emotional speech synthesis method of this invention, firstly, a variational autoencoder containing an encoder Enc and a decoder Dec is constructed. The input speech linear spectrum is mapped to the latent space via Enc, and the latent acoustic representation z0 is obtained through reparameterized sampling. This latent acoustic representation z0 is then mapped to a time-domain speech waveform via Dec, compressing high-dimensional acoustic features and reducing modeling complexity. Next, the target text is received, converted into a phoneme-level text representation, and upsampled to a frame-level text representation μ, achieving temporal alignment between the text representation and acoustic features, reducing the coupling between emotion and text. Finally, reference speech and speaker category labels s are received, and the emotional embedding e is refined to obtain... pure This achieves decoupling of emotion from the speaker, providing stable emotional conditions. Furthermore, based on μ, s, and e... pure A diffusion model is constructed in the latent space, and score functions with and without emotional conditions are calculated separately. The target latent acoustic representation is output through weighted fusion, which flexibly controls the emotional intensity and solves the problem of emotional intensity coupling. Finally, the target latent acoustic representation is input into Dec and mapped to a time-domain speech waveform to obtain the target emotional speech. This solves the original technical problems and achieves multi-speaker emotional speech synthesis with controllable emotion and high naturalness. Attached Figure Description

[0019] Figure 1 This is a flowchart of the speech synthesis method in this invention.

[0020] Figure 2 This is a schematic diagram of the speech synthesis method in this invention. Detailed Implementation

[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] The core objective of this invention is to solve the technical problems in existing multi-speaker emotional speech synthesis, such as high coupling between emotional expression and text content, speaker timbre, uncontrollable emotional intensity, and insufficient naturalness of synthesized speech. The overall technical framework is as follows: Figure 1 (Method Flowchart) and Figure 2 (The model diagram shows solid arrows representing the training process and dashed arrows representing the inference process.) It mainly includes five core steps: Variational Autoencoder (VAE) construction and training, text conditional modeling, sentiment embedding purification, latent diffusion model construction and target latent acoustic representation generation, and speech waveform generation. These steps work together to achieve a complete workflow of "high-dimensional acoustic feature compression → text feature alignment → sentiment feature purification → joint conditional modeling → speech waveform decoding," ultimately outputting emotionally controllable and highly natural multi-speaker emotional speech.

[0023] I. Construction and Training of Variational Autoencoder (VAE)

[0024] The core objective of this section is to construct a low-dimensional latent space, compressing high-dimensional acoustic features into low-dimensional latent acoustic representations, thereby reducing the modeling complexity of subsequent diffusion models. Simultaneously, through end-to-end decoder design, it lays the foundation for improving the naturalness of synthesized speech. Specific implementation details are as follows: 1. Construct a variational autoencoder and obtain the linear frequency. The VAE consists of an encoder Enc and a decoder Dec. The encoder Enc uses a WaveNet network structure and outputs the mean vector μ of the latent variable z. z Sum of variance vectors The decoder Dec uses the Hifigan v1 generator, and the activation functions are leaky_relu and Tanh.

[0025] Collect training speech data (such as a multi-speaker emotional speech dataset containing 100 speakers, each speaker has 4 emotions: happy, sad, angry, and neutral, 50 speech samples for each emotion, sampling rate of 22050Hz, and duration of 2-5 seconds for each speech sample), perform short-time Fourier transform on each speech signal, and finally obtain the corresponding linear spectrum y.

[0026] 2. Construct the posterior probability distribution

[0027] The encoder Enc extracts features and outputs latent variables by inputting the linear spectrum y into the encoder Enc and outputting the mean vector μ of the latent variable z. z Sum of variance vectors And construct the posterior probability distribution of the latent variable z. , where N(·) represents the normal distribution, and q(z|y) is the posterior probability distribution of the latent variable z given the linear spectrum y of the input.

[0028] 3. Obtaining the underlying acoustic representation

[0029] According to the reparameterized sampling method, since the sampling is non-differentiable, when sampling from the posterior probability distribution q(z|y), the formula is used. Sampling is performed to obtain the latent acoustic representation z0; where ε is Gaussian noise that follows a standard normal distribution ( (where I is the identity matrix), ⊙ represents element-wise multiplication.

[0030] 4. Obtain time-domain speech waveform

[0031] The decoder Dec is used to reconstruct the speech signal based on the latent variable z. It employs an end-to-end waveform generation structure to directly map the latent variable z into a time-domain speech waveform. ,Right now: ; 5. Training the variational autoencoder During the training phase, the variational autoencoder optimizes the model parameters (all parameters of the encoder Enc and decoder Dec) by maximizing the variational lower bound, and trains the loss function L. VAE Including Mel spectrum reconstruction loss, KL divergence loss, and adversarial loss: ; In the formula, L1 is the L1 loss function, x is the real speech waveform, and Mel(·) is the Mel-spectrum transformation function; D KL Let p(z) be the KL divergence loss function, and let p(z) be the standard normal distribution of the latent variable z.

[0032] (1) Mel spectrum reconstruction loss: used to constrain the speech waveform output by the decoder to be as consistent as possible with the Mel spectrum of the real speech waveform, thereby improving the naturalness of the reconstructed speech.

[0033] (2) KL divergence loss: used to constrain the posterior probability distribution q(z|y) of the latent variable z to approximate the standard normal distribution p(z), avoid the collapse of the latent space, and ensure that the latent acoustic representation z0 has good generalization ability.

[0034] (3) Adversarial loss: It is achieved by constructing a Generative Adversarial Network (GAN). The generator is the decoder Dec of the VAE, and the discriminator adopts a combined structure of a Multi-Period Discriminator (MPD) and a Multi-Scale Discriminator (MSD).

[0035] Parameter settings for the training process: The Adam optimizer is used, with the learning rate set to 2e-4, the batch size to 54, and the number of training epochs to 600. When the L VAE loss does not decrease for multiple consecutive rounds on the validation set, the training stops.

[0036] 6. Model parameter freezing

[0037] After training is completed, all model parameters of the VAE are frozen (the parameters of both the encoder Enc and the decoder Dec are no longer updated). The constructed low-dimensional latent space serves as the modeling space for the subsequent latent diffusion model. The latent acoustic representation z0 output by the encoder Enc is used as the initial latent representation for the latent diffusion model (i.e., x0 in the subsequent steps), and the decoder Dec is used to receive the target latent acoustic representation output in the subsequent steps to complete the decoding and generation of the speech waveform.

[0038] II. Text conditional modeling

[0039] The core purpose of this part is to convert the target text into a frame-level text representation μ that is aligned with the time dimension of the acoustic features, providing a suitable text input for the joint conditional modeling of the subsequent latent diffusion model. The specific implementation details are as follows: 1. Obtain the standard phoneme sequence Receive the target text corresponding to the speech to be synthesized. First, perform standard preprocessing on the target text, and then convert the standardized target text into a standard phoneme sequence according to the phoneme conversion dictionary. For example, the target text "Today's weather is nice" is converted into the phoneme sequence "j in1 t ian1 t ian1 q i4 zh en1 h ao3".

[0040] 2. Obtain the phoneme-level text representation

[0041] A 6-layer Transformer encoder is adopted, with each layer containing 1 multi-head attention mechanism, 1 convolutional feed-forward neural network, and 2 layer normalizations. The standard phoneme sequence is input into the text encoder. First, the phoneme sequence is embedded and encoded, then the multi-head attention mechanism of the Transformer encoder strengthens the temporal correlation between phonemes, and the feed-forward neural network performs feature transformation, finally outputting the phoneme-level text representation h text .

[0042] 3. Construct a duration predictor

[0043] Duration prediction and text upsampling are used to construct a duration predictor: a 3-layer convolutional layer + 2-layer normalization layer is used, the activation function is ReLU, and the dropout is 0.5, which is used to predict the duration corresponding to each phoneme.

[0044] Training the duration predictor: Phoneme-acoustic alignment is performed on the training speech data using the MFA tool to obtain the true duration at the phoneme level (as a supervision signal), and the duration is represented as a phoneme-level text h. text Using the input, train a duration predictor to minimize the mean squared error (MSE) between the predicted duration and the actual duration. The optimizer uses Adam until the MSE on the validation set converges.

[0045] Text upsampling process: During the inference phase, the duration predictor does not rely on true alignment information, but directly uses the phoneme-level text representation h corresponding to the input target text. text It automatically predicts the duration of each phoneme; then, based on the predicted duration, it performs phoneme-level text representation h. text The extended processing is performed along the time dimension, specifically by processing the h corresponding to each phoneme. text The feature vector is copied and stretched according to the predicted duration, so that the extended text representation is fully aligned with the temporal dimension of the subsequent acoustic features (latent acoustic representation z0), and finally the frame-level text representation μ is obtained, which is consistent with the temporal dimension of the latent acoustic representation z0. This achieves temporal alignment between the text representation and the acoustic features, reduces the coupling between sentiment expression and text content, and provides adaptive input for subsequent joint conditional modeling.

[0046] III. Emotional Embedding and Refinement

[0047] This section constructs an emotion embedding purification method based on a domain adversarial adaptive instance normalization (AdaIN) module. This method purifies the reference speech to obtain an emotion embedding e that is effectively decoupled from the speaker's identity information. pure This provides a pure and stable emotional condition for emotional transfer in multi-speaker scenarios. The specific implementation details are as follows: 1. Obtain Mel spectrum Receive reference speech and corresponding speaker category labels s: The reference speech is a speech segment carrying the target emotion (such as happiness or sadness), which is consistent with the emotional needs of the target text; the speaker category labels s are the speaker identifiers corresponding to the reference speech, which are used for subsequent domain adversarial training.

[0048] The reference speech is converted into a Mel spectrum. At the same time, the original Mel spectrum is constructed by shuffling it in the batch dimension (e.g., rearranging the order 0,1,2,3,4 into 1,2,3,4,0), thereby forming a mismatch relationship with the text, resulting in two types of Mel spectra (text matching and text non-matching).

[0049] 2. Obtain the matching embedding

[0050] A two-parameter-shared sentiment encoder (with the same structure as the encoder Enc, consisting of 6 layers of 2DConv + 1 layer of GRU) is used to extract features from two types of Mel spectra, outputting sentiment matching embeddings z. i (Mel spectrum from text matching) and sentiment mismatch embedding z j (From a Mel spectrum with a text mismatch).

[0051] 3. Obtaining emotional embedding

[0052] Calculate z j The mean μ(z) j ) and variance σ(zj), z i The mean μ(z) i ) and variance σ(z) i ),pass The initial emotional embedding e was calculated. pure .

[0053] Construct an emotion classifier consisting of three cascaded linear layers, and classify e pure The input is a sentiment classifier, with the target sentiment label c corresponding to the reference speech as the supervision signal, and the cross-entropy loss function is used. The sentiment classification error is calculated, and the parameters of the sentiment encoder and sentiment classifier are updated synchronously based on error backpropagation. Where L... c For sentiment classification error, log is the natural logarithm; p(c|e pure For a given emotion embedding e pure When, the probability of the corresponding true sentiment label c.

[0054] 4. Domain confrontation training

[0055] In z i A gradient inversion layer and a single-layer linear speaker classifier are inserted into the transmission path, and the z-order of the speech after passing through the gradient inversion layer is processed. i Input the speaker classifier, use the speaker category label s as the supervision signal, and employ the cross-entropy loss function. Calculate the speaker classification error; during backpropagation, the gradient inversion layer scales the speaker classification error backward and then feeds it back to the emotion encoder, causing the emotion encoder's parameter update direction to deviate from the speaker classification target, thus eliminating z-classifiers. i It includes the speaker's identity information. Among them, Ls Speaker classification error; p(s|z) i ) is given z i When, the probability corresponding to the real speaker category label s.

[0056] 5. Obtain the refined emotional embedding

[0057] Joint sentiment classification error L e Speaker classification error L s Construct the total loss function L total = L e +α·L s Where α is the corresponding loss L s The weight coefficient is 0.02. The above steps are repeated in each iteration, synchronously updating the parameters of the emotion encoder, emotion classifier, and speaker classifier; when the total loss L... total When the convergence reaches the preset threshold, the iteration stops, achieving effective decoupling of emotion and speaker features, and providing stable emotion conditions for subsequent diffusion models.

[0058] IV. Generating the latent acoustic representation of the target

[0059] The core objective of this section is to construct a latent diffusion model based on the joint conditions of text, speaker, and emotion. Through weighted fusion using a scoring function, it achieves flexible control over emotion intensity and outputs a target latent acoustic representation that meets the requirements of text content, target speaker timbre, and emotion intensity. Specific implementation details are as follows: 1. Obtain the initial latent representation The modeling space is defined as the low-dimensional latent space constructed by the VAE after training and parameter freezing. The latent acoustic representation z0 of this space is used as the initial latent representation x0 of the latent diffusion model.

[0060] 2. Construct a potential diffusion model

[0061] A potential diffusion model is constructed, and the denoising network for the diffusion model adopts a neural network (DiT Blocks) based on the DiT structure (Denoising Transformer).

[0062] 3. Training the potential diffusion model μ

[0063] The forward and backward diffusion processes in the potential diffusion model follow a continuous-time stochastic differential equation design: (1) Forward diffusion process: Gaussian noise is gradually injected into the initial latent representation x0 of the VAE output, and its stochastic differential equation is: ; in: dx t Represents the latent variable xt A tiny change within an extremely short time dt.

[0064] x t Let z0 be the latent variable at time t. It refers to the intermediate state of the initial latent acoustic representation z0 of the VAE output after forward diffusion and noise injection at time t. It is a vector in the low-dimensional latent space, representing the acoustic features with noise.

[0065] t is the time step of the diffusion process, and its value ranges from [0,T] (usually T=1). t=0 corresponds to the initial state without noise, and t=T corresponds to the state that is completely noisy.

[0066] β t This is a noise scheduling function; in this example, the scheduling function is a linear scheduling function.

[0067] dt is a tiny increment in time, which is the time differential element in the differential equation, representing an extremely short time step.

[0068] Wt is a standard Wiener process, a continuous-time stochastic process, serving as a source of forward-spreading random noise; dW t It is the differential of the Wiener process, representing the increment of Gaussian noise injected at time t, which follows a normal distribution with a mean of 0 and a variance of dt.

[0069] (2) Back-diffusion process: A deterministic sampling form based on probability flow ordinary differential equations is adopted, and its expression is: ; in: For gradient operators; p t (x t |x0) represents the state from the initial state x0 to the state x at time t during forward diffusion. t The conditional probability density; This is the scoring function.

[0070] It is a reverse Wiener process. It is its differential.

[0071] Given an initial latent representation x0, the latent variable x at any time t t It follows a Gaussian distribution as follows: ; ; ; in: N(·) is normally distributed.

[0072] ρ(x0,t) represents the latent variable x at time t.t The mean function depends on the initial latent representation x0 and the time step t; (·) λ is the natural exponential function; λ(t) is the latent variable x at time t. t The variance of λ(t) depends only on the time step t; I is the identity matrix. When t→∞, λ(t)→I, that is, the latent variables are completely noise-induced and the variance approaches the identity matrix.

[0073] Training the latent diffusion model: using an initial latent representation x0 (latent acoustic representation z0), time step t, frame-level text representation μ, and refined sentiment embedding e pure The speaker category labels 's' are the training data. The training objective of the denoising network is to minimize the L2 loss of the score function prediction error. The loss function expression is: ; in: min is the minimization operator, and the goal is to find the implicit value in the score function s. θ The optimal model parameters θ are found to minimize the overall loss function. t Let λ(t) be represented.

[0074] For the initial latent representation x0 and the latent variable x at time t t The expectation represents the average over all samples on the training dataset.

[0075] The square of the L2 norm is used to calculate the squared error between the "predicted score" and the "true score," which measures the accuracy of the model's predictions.

[0076] 4. Construct an emotion-guided scoring function

[0077] The scoring function calculation and weighted fusion are performed by the denoising network calculating two types of scoring functions during the backdiffusion sampling process: (1) Score function s with emotional conditions θ (x t ,t,μ,s,e pure ): Input is the latent variable x at time t. t Time step t, frame-level text representation μ, speaker category label s, refined sentiment embedding e pure The output is a score containing emotional information, which is used to guide the sampling process to shift towards the target emotional direction.

[0078] (2) Scoring function s without emotional conditions θ (x t ,t,μ,s, ): The input is the latent variable x at time t. tTime step t, frame-level text representation μ, speaker category label s, all-zero vector The output is a score that does not include sentiment information.

[0079] By introducing a guiding coefficient γ to weight and fuse the two types of scoring functions, an emotion-guided scoring function is constructed: ; Where γ≥0, the specific value is adjusted by the user according to the emotional intensity requirements. The larger γ is, the higher the emotional intensity.

[0080] B45: Generate the latent acoustic representation of the target

[0081] The emotion-guided scoring function replaces the original scoring function and participates in the latent variable update process of backdiffusion sampling, progressively updating the latent variable x at time t. t Denoising is performed until sampling is completed at t=0, and the final output is the target latent acoustic representation that meets the requirements of text content, target speaker timbre and emotional intensity.

[0082] V. Obtaining Target Emotional Voice

[0083] The core objective of this section is to decode the latent acoustic representation of the target into a temporal speech waveform, thereby obtaining the final target emotional speech. The specific implementation details are as follows: The target latent acoustic representation is directly input into the trained and parameter-frozen VAE decoder Dec to generate the target emotional speech.

[0084] VI. Experimental Verification

[0085] To further verify the effectiveness of the multi-speaker emotional speech synthesis method in this invention, a comparative experiment was designed based on the DataOceanEmotional speech database (DOE) dataset, and the performance was benchmarked against existing advanced models using mainstream evaluation metrics.

[0086] 1. Comparison Model

[0087] We selected representative models in the field of multi-speaker emotional speech synthesis, such as iEmoTTS, CMETTS, and Mels-TTS, as benchmarks and compared them with the method of this invention (Ours).

[0088] 2. Evaluation Indicators

[0089] Performance is quantified using three core metrics, with all metrics having a confidence level of 95%. Mean Opinion Score (MOS): assesses the naturalness of synthesized speech; a higher score indicates a smoother and more natural speech. Difference-of-Sentence Meaning (DMOS-E): Measures the emotional similarity between synthesized speech and reference speech; a higher score indicates more accurate emotional transfer. Speaker Differential Mean Opinion Score (DMOS-S): Measures the similarity of synthesized speech to the timbre of the target speaker; a higher score indicates better speaker consistency.

[0090] 3. Experimental Results

[0091] The experimental results are shown in the table below:

[0092] 4. Results Analysis

[0093] In terms of naturalness (MOS), the method of this invention scored 4.11±0.048, which is higher than all the comparison models, indicating that the proposed method can generate more natural speech.

[0094] In terms of sentiment similarity (DMOS-E), the method of this invention leads with a score of 4.12±0.042, verifying the effectiveness of the sentiment embedding purification method.

[0095] In terms of speaker similarity (DMOS-S), the method of this invention scored 4.09±0.059, which is better than the comparison model. The results show that the method of this invention can better maintain the consistency of the target speaker's timbre and alleviate the problem of emotion and timbre coupling in existing methods to a certain extent.

[0096] In summary, the experimental results fully demonstrate that the method of the present invention has achieved performance breakthroughs in the naturalness of synthesized speech, the accuracy of emotional expression, and the consistency of speaker timbre, verifying the feasibility and superiority of the "emotional embedding purification + latent diffusion model joint modeling" technical solution.

[0097] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for emotion embedding purification in multi-speaker emotional speech synthesis, characterized in that, The purification process includes the following steps: A1. Receive reference speech that matches and does not match the target text, and convert it into Mel spectrogram; A2. Two sentiment encoders with shared parameters are used to extract features from each Mel spectrum, outputting the sentiment matching embedding z. i Emotional mismatch embedding z j ; A3. Calculate z j The mean μ(z) j ) and variance σ(z) j ), z i variance σ(z) i ) and mean μ(z i ),pass The emotional embedding e is calculated pure ; A4, e pure The input is fed into a pre-built classifier, and the target emotion label of the reference speech is used as a supervision signal. The parameters of the emotion encoder and the emotion classifier are updated based on error backpropagation. A5, in z i A gradient inversion layer and a speaker classifier are inserted into the transmission path. The gradient inversion layer scales the classification error in reverse and then sends it back to the emotion encoder, causing the emotion encoder parameter update direction to deviate from the speaker classification target. A6. Iterate through steps A3-A5 until the stopping condition is met. Then, during the inference phase, use only the emotion encoder to extract the emotion embedding e from the Mel spectrum of the reference speech. pure .

2. The emotion embedding purification method for multi-speaker emotional speech synthesis according to claim 1, characterized in that, The reference speech includes speech segments that match the target text content of the reference speech itself, as well as speech segments that do not match the target text content of the reference speech itself.

3. The emotion embedding purification method for multi-speaker emotional speech synthesis according to claim 1, characterized in that, Step A4 is as follows: Construct an emotion classifier consisting of three cascaded linear layers, embedding the emotion into e. pure The input is a sentiment classifier, with the target sentiment label c corresponding to the reference speech as the supervision signal, and the cross-entropy loss function is used. Calculate the sentiment classification error, and based on the sentiment classification error, backpropagate to synchronously update the parameters of each sentiment encoder and sentiment classifier, thereby strengthening e. pure The ability to distinguish emotional categories; among them, L c For sentiment classification error, log is the natural logarithm; p(c|e pure For a given emotion embedding e pure When, the probability of the corresponding true sentiment label c.

4. The emotion embedding purification method for multi-speaker emotional speech synthesis according to claim 1, characterized in that, The specific content of step A5 is as follows: In z i A gradient inversion layer and a single-layer linear speaker classifier are inserted into the transmission path, and the z-order of the speech after passing through the gradient inversion layer is processed. i Input the speaker classifier, use the speaker category label s corresponding to the reference speech as the supervision signal, and employ the cross-entropy loss function. The speaker classification error is calculated; during backpropagation, the gradient inversion layer scales the speaker classification error backward and then feeds it back to the emotion encoder, causing the emotion encoder's parameter update direction to deviate from the speaker classification target, so that the emotion encoder can remove z. i The speaker's identity information included; among which, L s Speaker classification error; p(s|z) i ) is given z i When, the probability corresponding to the real speaker category label s.

5. The emotion embedding purification method for multi-speaker emotional speech synthesis according to claim 1, characterized in that, The specific content of step A6 is as follows: Joint sentiment classification error L e Speaker classification error L s Construct the total loss function L total =L e +α·L s , where α is the corresponding loss L s The weight coefficient is 0.02; iterative optimization is performed based on this total loss function: in each iteration, steps A3-A5 are repeated, and all parameters of the emotion encoder, emotion classifier, and speaker classifier are updated synchronously through backpropagation; when the total loss L total The iteration stops when the convergence reaches the preset threshold.

6. A multi-speaker emotional speech synthesis method based on a latent diffusion model, characterized in that, The synthesis steps include the following: B1. Construct a variational autoencoder including encoder Enc and decoder Dec. Take the linear spectrum of the input speech, map the linear spectrum to the latent space through Enc, obtain the latent acoustic representation z0 based on reparameterized sampling, and map z0 to the time-domain speech waveform through Dec. B2. Receive the target text, convert it into a phoneme-level text representation, and upsample it to a frame-level text representation μ. B3. Receive the reference speech and the corresponding speaker category label s, and refine the reference speech using the emotion embedding refinement method described in any one of claims 1-5 to obtain the emotion embedding e. pure ; B4, based on μ, s, and e pure Construct a potential diffusion model in the potential space; In the latent diffusion model, score functions with and without emotional conditions are calculated separately, and the target latent acoustic representation is output based on the weighted fusion of the two score functions. B5. Input the latent acoustic representation of the target into Dec, which maps it to a time-domain speech waveform to obtain the target emotional speech.

7. A multi-speaker emotional speech synthesis method based on a latent diffusion model according to claim 6, characterized in that, The sub-steps of step B1 are as follows: B11. Construct a variational autoencoder (VAE) consisting of an encoder Enc and a decoder Dec. The input feature of the VAE is the linear spectrum y obtained by the short-time Fourier transform of the speech signal. B12. Extract features from the linear spectrum y using encoder Enc, map it to a low-dimensional latent space, and output the mean vector μ of the latent variable z. z Sum of variance vectors Construct the posterior probability distribution of the latent variable z. ,in, Given an input y, the posterior probability distribution of the latent variable z; It follows a normal distribution; B13. The latent acoustic representation z0 is obtained by sampling from the posterior probability distribution based on the reparameterized sampling method; B14. The decoder Dec employs an end-to-end waveform generation structure, directly mapping the latent acoustic representation z0 to a time-domain speech waveform. ; B15. During the training phase, the variational autoencoder optimizes the model parameters by maximizing the variational lower bound and trains the loss function L. VAE Including Mel spectrum reconstruction loss, KL divergence loss, and adversarial loss: ； In the formula, L1 is the L1 loss function, x is the real speech waveform, and Mel(·) is the Mel-spectrum transformation function; D KL Let p(z) be the KL divergence loss function, and let p(z) be the standard normal distribution of the latent variable z. B16. After training is complete, freeze all model parameters of the variational autoencoder. Its output latent acoustic representation is used for the modeling and generation of subsequent latent diffusion models.

8. A multi-speaker emotional speech synthesis method based on a latent diffusion model according to claim 6, characterized in that, The sub-steps of step B2 are as follows: B21. Receive the target text corresponding to the speech to be synthesized, and convert the target text into a standard phoneme sequence; B22. Construct a text encoder. Input the standard phoneme sequence into the text encoder for feature encoding and output a phoneme-level text representation h. text ; B23. Construct a duration predictor. The duration predictor uses the true duration at the phoneme level obtained after aligning the training speech data using the Montreal forced alignment tool as the supervision signal, and represents the h in phoneme-level text. text As input, the system learns to establish a mapping relationship between phoneme-level text representations and phoneme durations, which is used to predict the duration of each phoneme. B23. During the text upsampling phase, based on the phoneme duration information output by the duration predictor, the phoneme-level text representation h is processed. text Extending the processing along the time dimension involves copying and stretching the phoneme-level text representation h. text To frame-level resolution, the phoneme-level text representation h text Expanded into a frame-level text representation μ aligned with the temporal dimension of subsequent acoustic features; During the inference phase, the duration predictor does not rely on true alignment information; it directly uses the phoneme-level text representation h corresponding to the input target text. text Predict the duration of phonemes and complete the phoneme-level text representation h based on the predicted duration. text The upsampling process to frame-level text representation μ; The frame-level text representation μ is used as the text conditional input for the subsequent potential diffusion model.

9. A multi-speaker emotional speech synthesis method based on a latent diffusion model according to claim 6, characterized in that, The sub-steps of step B4 are as follows: B41. The low-dimensional latent space constructed by the variational autoencoder (VAE) with the model parameters frozen after training is used as the modeling space. The encoder of the variational autoencoder (VAE) has mapped the high-dimensional acoustic features to the low-dimensional latent space and output the latent acoustic representation. B42. Construct a latent diffusion model in a low-dimensional latent space. The denoising network for the latent diffusion model adopts a neural network based on the DiT structure. B43, frame-level text representation μ, speaker category labels s, and refined sentiment embedding e pure As a joint conditional input latent diffusion model, the latent diffusion model describes the forward and backward diffusion processes through continuous-time stochastic differential equations. The forward diffusion process involves progressively injecting Gaussian noise into the latent acoustic representation z0 output by the variational autoencoder (VAE), and embedding emotion e with a 20% probability. pure Set as a vector of all zeros This allows the model to learn to generate without emotional conditions; the backdiffusion process employs a deterministic sampling method based on probability flow ordinary differential equations. B44. During the backdiffusion process, the denoising network calculates two types of score functions: one is the score function s with sentiment conditions. θ (x t ,t,μ,s,e pure The scoring function takes the latent variable x at time t as its basis. t Time step t, frame-level text representation μ, speaker category label s, and refined sentiment embedding e pure The input is 1; the second is a scoring function s without emotional conditions. θ (x t ,t,μ,s, The scoring function takes the latent variable x at time t as its basis. t The inputs are time step t, frame-level text representation μ, and speaker category label s; a guidance coefficient γ is introduced to weight and fuse the two types of scoring functions to construct an emotion guidance scoring function. for: ； The intensity of emotional embedding's guidance on the sampling process is controlled by adjusting the guidance coefficient γ value; B45. Use the emotion-guided scoring function to replace the original scoring function in the latent variable update process of backdiffusion until the backdiffusion sampling is completed, and finally output the target latent acoustic representation that meets the requirements of text content, target speaker timbre and emotion intensity.

10. A multi-speaker emotional speech synthesis system based on a latent diffusion model, characterized in that, It includes a processor and a memory, the memory storing a computer program, which, when executed by the processor, implements the emotion embedding purification method as described in any one of claims 1-5, and / or implements the multi-speaker emotion speech synthesis method based on the latent diffusion model as described in any one of claims 6-9.

Images