Speech synthesis method, electronic device, and storage medium

Abstract

The application discloses a speech synthesis method, an electronic device and a storage medium, wherein the speech synthesis method comprises the following steps: training an acoustic model without emotion input, wherein a diffusion process is used in the training process of the acoustic model, and the training target of the acoustic model is to estimate the logarithmic gradient of a data distribution at any intermediate time in the diffusion process; training an emotion classifier, wherein the input of the emotion classifier at least comprises the logarithmic gradient corresponding to an intermediate time in the diffusion process; and using a soft label guidance technology to perform emotion-controllable speech synthesis sampling, wherein the diffusion process corresponds to a reverse denoising process of the speech synthesis sampling, a gradient term of the reverse denoising process is a target estimated by a speech synthesis model, and a soft label guidance term is mathematically equivalent to cross-entropy, one side of the cross-entropy is an output of the emotion classifier, and the other side of the cross-entropy is a distribution corresponding to a target emotion intensity.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of training of speech synthesis model, and particularly relates to a speech synthesis method, an electronic device and a storage medium. BACKGROUND

[0002] The related art mainly falls into two categories. One is to use a Relative Attributes Rank (RAR, a relative information modeling method) technology, to find an optimal ranking matrix by using a SVM (Support Vector Machine) method to solve an artificially constructed optimization problem, so as to obtain a relative emotional intensity value in advance for training. After the training is completed, the intensity value is used to control the synthesized emotion. The second category is to operate on an emotional embedding space, such as interpolation.

[0003] The inventor finds in the process of implementing the present application that the RAR-based method needs to calculate the relative emotional intensity value of all training data in advance. The solution of the optimization problem in this stage directly affects the subsequent training. The extraction of the emotional embedding expression is also crucial for the second category, and careful additional constraints often need to be imposed on the space to improve the effect. In addition, the synthesis quality of some methods is poor, which is most likely caused by the previous stage. SUMMARY

[0004] The embodiments of the present application provide a speech synthesis method, an electronic device and a storage medium, which are used to at least solve one of the above technical problems.

[0005] In a first aspect, the embodiments of the present application provide a speech synthesis method, comprising: training an acoustic model without emotional input, wherein a diffusion process is used in the training process of the acoustic model, and the training target of the acoustic model is to estimate the log gradient of the data distribution at any intermediate time in the diffusion process; training an emotion classifier, wherein the input of the emotion classifier at least includes the log gradient corresponding to a certain intermediate time in the diffusion process; and using a soft label guidance technology to perform emotion controllable speech synthesis sampling, wherein the diffusion process corresponds to a reverse denoising process of the speech synthesis sampling, the gradient term of the reverse denoising process is the target estimated by the speech synthesis model, and the soft label guidance term is mathematically equivalent to the cross-entropy, one side of which is the output of the emotion classifier, and the other side is the distribution corresponding to the target emotional intensity.

[0006] In a second aspect, an electronic device is provided, comprising at least one processor, and a memory connected with the at least one processor in communication, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the steps of the speech synthesis method of any of the embodiments of the present application.

[0007] In a third aspect, the embodiments of the present application further provide a computer program product, comprising a computer program stored on a non-volatile computer readable storage medium, the computer program comprising program instructions which, when executed by a computer, cause the computer to perform the steps of the speech synthesis method of any of the embodiments of the present application.

[0008] In the method of the embodiments of the present application, the emotional intensity can be controlled by the soft label guidance technology proposed by the inventor, which is extended based on the classifier guidance technology, so that the speech synthesis with controllable emotion and high quality can be obtained. BRIEF DESCRIPTION OF DRAWINGS

[0009] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced as follows. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0010] Figure 1 A flowchart of a speech synthesis method provided by an embodiment of the present application;

[0011] Figure 2 A training and sampling diagram of EmoDiff provided by an embodiment of the present application;

[0012] Figure 3 Evaluation of mean opinion score (MOS) and mel-celestial distortion (MCD) provided by an embodiment of the present application;

[0013] Figure 4 Classification probability diagram when the control intensity α∈{0.0, 0.2, 0.4, 0.6, 0.8, 1.0} is provided by an embodiment of the present application;

[0014] Figure 5 Diversity preference test of each emotion provided by an embodiment of the present application;

[0015] Figure 6 A structural schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0016] In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0017] Reference is made to Figure 1 which shows a flowchart of an embodiment of the speech synthesis method of the present application.

[0018] As shown in Figure 1 , in step 101, an emotionless input acoustic model is trained, wherein a diffusion process is used in the training process of the acoustic model, and the training target of the acoustic model is to estimate the log gradient of the data distribution for any intermediate time in the diffusion process;

[0019] In step 102, an emotion classifier is trained, wherein the input of the emotion classifier at least includes the log gradient corresponding to a certain intermediate time in the diffusion process;

[0020] In step 103, a soft label guidance technique is used for emotion controllable speech synthesis sampling, wherein the diffusion process corresponds to a reverse denoising process of the speech synthesis sampling, the gradient term of the reverse denoising process is the target estimated by the speech synthesis model, and the soft label guidance term is mathematically equivalent to the cross entropy, one side of which is the output of the emotion classifier and the other side is the distribution corresponding to the target emotion intensity.

[0021] In the method of the embodiments of the present application, the emotion intensity can be controlled by the soft label guidance technique proposed by the inventor, which is extended based on the classifier guidance technique, so that emotion controllable and high-quality speech synthesis can be obtained.

[0022] In some optional embodiments, the emotion classifier freezes the acoustic model parameters during training and only updates the weights of the emotion classifier.

[0023] In some optional embodiments, the emotion classifier is trained using a standard cross entropy loss LCE.

[0024] In some optional embodiments, the acoustic model learns how to generate realistic mel-spectra from a given text and a given sequence of durations through training, and the input of the emotion classifier further includes a representation related to the given text and a time related to the given text.

[0025] In some optional embodiments, the diffusion process adopts a denoising diffusion model using an exponential moving average on model weights. Thus, the performance of the denoising diffusion model can be improved.

[0026] It should be noted that the above method steps are not used to limit the execution order of each step, and in fact, some steps can be executed simultaneously or in the opposite order defined by the steps, which is not limited herein.

[0027] The following describes some problems encountered by the inventors in the process of implementing the present application and a specific embodiment of the finally determined scheme, so that those skilled in the art can better understand the scheme of the present application.

[0028] The inventors found that the above-mentioned defects are mainly caused by the following reasons: whether based on RAR or emotion embedding space method, a relatively complex optimization problem solving process is needed, which is likely to output a suboptimal result, resulting in subsequent model optimization difficulties.

[0029] When facing the above-mentioned defects, those skilled in the art usually use pre-training, increase data for optimization problem solving, use better solvers, etc. Because the scheme of the embodiment of the present application is based on a denoising diffusion model, which is a relatively novel generative model with deep mathematical principles, it is currently less used for speech synthesis, so it is not easy to think of.

[0030] The method of the embodiment of the present application is based on the classifier guidance technology of the denoising diffusion model, which extends it into a method called soft-label guidance to directly control the emotion intensity. Specifically, the inventors train an acoustic model that does not depend on emotion labels, and then train a classifier to classify emotions from the intermediate variables of the diffusion process. Subsequently, in the synthesis stage, the inventors directly use the gradient of the classifier to guide the synthesis, so that a specific emotion can be synthesized.

[0031] This method is very direct to train, only an additional emotion classifier needs to be trained, and this has been widely studied in the speech emotion recognition task, which is based on neural network optimization, and the training quality can be guaranteed. The soft-label guidance technology extended by the inventors can enable the inventors to directly control the emotion intensity using this classifier.

[0032] Please refer to Figure 2, which shows a structural block diagram of a specific example of a speech synthesis method and system provided by an embodiment of the present invention. Specifically, Figure 2 The training and sampling diagram of EmoDiff in the embodiment of the present application is shown. During training, xt is sampled directly from the known distribution p(xt|x0). When sampling, the score function Estimated by the score estimator. "SG" means stop gradient operation. The English and Chinese counterparts are as follows: mel-spec: Mel-spectrum, Score Estimator: Score Estimator, Forward SDE: Forward Diffusion SDE, Phoneme-dependent Gaussian noise: Phoneme-dependent Gaussian noise, Duration Predictor: Duration Predictor, Duration Adaptor: Duration Expander, PhonemeEncoder: Phoneme Encoder, Phoneme Sequence: Phoneme Sequence, Input Encoder: Input Encoder, InputDuration: Input Duration, Input Text: Input Text, Unconditional Acoustic Model Training: Unconditional Acoustic Model Training; Emotion Classifier: Emotion Classifier, Input Emotion: Target Emotion, Classifier Training: Emotion Classifier Training; Reverse-time SDE: Reverse-time SDE, SDE numericalsolver: SDE numerical solver, Unconditional Acoustic Model Sample: Unconditional Acoustic Model Sample, Soft-Label Guidance: Soft Label Guidance, Neutral: Neutral, Target emotion d with intensity a40% Neutral 60%: Target emotion with intensity a 40% neutral 60%, Intensity ControllableEmotional Sampling: Intensity controllable emotional sampling.

[0033] like Figure 2 As shown, Figure 2 (a) in the paper: The inventors train an acoustic model without emotion input. The model learns how to generate realistic mel spectrograms from a given text and a given duration sequence. The model uses a diffusion process, and the training goal is to estimate the logarithmic gradient of the data distribution at any intermediate time t of the diffusion process, that is,

[0034] Figure 2(b) in the above: the inventors train an emotion classifier. The input of this classifier is the intermediate variable x at some time t in the above diffusion process t and the text-dependent representation μ and the time t. The training criterion of this classifier is the usual cross-entropy classification criterion.

[0035] Figure 2 (c) in the above: the inventors use soft-label guided techniques for emotion controllable speech synthesis. The diffusion process described in (a) above now corresponds to a backward denoising process, where the gradient term is the target of the acoustic model estimation of the present application. The soft-label guiding term of the present application now corresponds to a negative cross-entropy, with one side being the output of the inventors' classifier and the other side being the distribution corresponding to the target emotion intensity. The emotion intensity value is now restricted to a value between 0 and 1. Suppose the inventors need an angry emotion with intensity 0.4, then this is equivalent to a categorical distribution over all emotion labels, with probability 0.4 for anger and probability 0.6 for neutral. The inventors need to simulate the denoising process by numerical methods, and at each iteration of the numerical simulation, the inventors guide the process by the soft label. Finally, the inventors are able to generate a realistic sample with the required emotion intensity.

[0036] In solving the above technical problems, the inventors have considered including the duration of each phoneme of the speech into the diffusion model and the denoising model, so that the duration can be directly controlled by the emotion. However, how to construct a better denoising diffusion model target is a more tricky problem, and the implementation is more difficult, so it is not further discussed.

[0037] The inventors also tried to improve the generation stage of the denoising diffusion model, such as using DPS (diffusion posterior sampling, a novel diffusion model sampling method), RK45 ODE solver, and the like. These methods are theoretically helpful to improve the quality of the generated diffusion model. None of them requires additional changes to the training process. Among them, DPS proposes to first recover x_0 from the noisy data x_t using the trained model, then add noise for one step, and directly use the noise-added result to calculate the gradient, which can approximate the result of the original denoising process. RK45 (Runge-Kutta-Fehlberg method, an ODE sampling method of 4th order with 5th order noise estimation) is a numerical solution method for ODE (ordinary differential equation). Since the denoising process is essentially a stochastic differential equation (SDE), it has an equivalent version of ODE, and therefore can be improved by using an ODE solution method such as RK45 to obtain higher-quality samples. However, it is found in practice that these methods reduce the diversity of the results, and therefore are not adopted.

[0038] In addition to being able to make emotion-controllable and high-quality speech synthesis, the model of the embodiments of the present application also has the following advantages: strong diversity, even within the same emotion. This is a direct advantage of using a denoising diffusion model. The soft label guidance method of the embodiments of the present application can be easily generalized into a method for controlling any combination of emotions, rather than just controlling the intensity of emotions. In a deeper sense, the method of the embodiments of the present application makes it possible to control any combination of multiple emotions.

[0039] The following verifies the beneficial effects of the embodiments of the present application relative to the prior art through specific experiments and experimental data.

[0040] Although current speech synthesis models can generate high-quality speech, emotion intensity controllable speech synthesis remains a challenging task. Most current works require an additional optimization process to compute emotion intensity values, which results in suboptimal results or degraded synthesis quality. In the embodiments of the present application, the inventors propose EmoDiff, a diffusion model based speech synthesis model. In EmoDiff, emotion intensity can be manipulated by our proposed soft label guiding technique, which is an extension of the classifier guiding technique. Specifically, EmoDiff uses a soft label to control emotion intensity instead of a one-hot vector. In this soft label, the inventors set the emotion to be controlled and the neutral emotion with a and 1-a, respectively, where a is the emotion intensity value, which can be chosen from 0 to 1. The experiments of the embodiments of the present application show that EmoDiff can accurately control emotion intensity while maintaining high-quality synthesis. In addition, EmoDiff can still achieve good diversity when given an emotion.

[0041] 1. INTRODUCTION

[0042] Although current TTS (text-to-speech) models, such as Grad-TTS, VITS, and VQ-TTS, can produce high-quality speech, intensity-controllable emotional TTS remains a challenging task. A similar task to emotional TTS is prosody modeling in TTS, which generally does not have data with specific labels. Unlike prosody modeling, emotional TTS usually utilizes a dataset with categorical emotion labels. Mainstream emotional TTS models can only synthesize emotional speech according to emotion labels without intensity controllability.

[0043] In strength-controllable TTS models, most existing work aims to correctly define and calculate emotion intensity values ​​for the training process. The most popular method for defining and obtaining emotion intensity is Relative Attributes Rank (RAR). RAR seeks a ranking matrix via a maximum margin optimization problem, which is solved by a support vector machine (SVM). The resulting solution is then fed into the model for training. Because this is a separate and artificially constructed stage, it can lead to suboptimal results and introduce bias into the training process. In addition to RAR, some work has explored manipulation of the emotion embedding space. Some related works design an algorithm to maximize the distance between emotion embeddings and interpolate the embedding space to control emotion intensity. Other related works quantize the distance between emotion embeddings to obtain emotion intensity. However, the structure of the embedding space also significantly affects the performance of these models, requiring the careful design of additional constraints. Intensity control in emotion-to-speech conversion has also been studied in related art, and similar methods exist. Some of these works also suffer from speech quality degradation. As an example, "MixedEmotion" is an autoregressive model that weights emotion embeddings using RAR intensity values. It uses pre-training to improve the synthesis quality, but there is still a noticeable quality drop.

[0044] To overcome these problems, the inventors needed a conditional sampling method that could directly control the weighted intensity of emotions. In this work, the inventors proposed a soft label guidance technique based on the classifier guidance technique used in speech emotion recognition tasks. Classifier guidance is an efficient sampling technique that uses the gradient of the classifier to guide the sampling trajectory of a given one-hot class label.

[0045] In this application, based on the extended soft label guidance, the inventors proposed EmoDiff, an emotional TTS model with sufficient intensity controllability. Specifically, the inventors first trained an acoustic model without emotion input. Then trained an emotion classifier on any xt in the diffusion process trajectory, where t is the timestamp in the diffusion process (any intermediate moment). In reasoning, the inventors used the classifier and soft emotion labels to guide the reverse denoising process, where the values ​​of the specified emotion and neutral emotion were set to α and 1-α respectively, rather than a unique heat distribution in which only the specified emotion is 1 and the others are 0. Experiments in the embodiments of the present application show that EmoDiff can accurately control the intensity of emotion while maintaining high voice quality. In addition, it can generate different voice samples even for the same emotion, which is also its advantage as a diffusion model.

[0046] In short, the main advantages of EmoDiff are:

[0047] 1. When using soft labels, the inventors define the emotion strength as the weight guided by the classifier. This achieves precise strength control in terms of classifier probabilities without the need for additional optimization. Thus, it enables the inventors to effectively generate speech with arbitrary specified emotion strength.

[0048] 2. It does not harm the synthesized speech. The generated samples have good quality and naturalness.

[0049] 3. It can generate different samples even under the same emotion.

[0050] 2. Diffusion model with classifier guidance

[0051] 2.1. Denoising diffusion model and TTS application

[0052] Denoising diffusion probability models have been proven successful in many generative tasks. In a score-based interpretation, the diffusion model constructs a forward stochastic differential equation (SDE) that transforms a data distribution p0(x0) into a known distribution p T (x T ), and generates realistic samples starting from noise using the corresponding backward-time SDE. This backward process is also called the "denoising" process. Then, a neural network is to estimate the score function with the goal of score matching. In applications, diffusion models bypass the training instability and mode collapse problems in GANs and outperform previous methods in sample quality and diversity.

[0053] Denoising diffusion models have also been used in TTS and vocoder tasks with significant results. In this application, the inventors build EmoDiff on top of GradTTS. Let x e R d be a frame of mel-spectrogram, which constructs a forward SDE:

[0054]

[0055] where B t is a standard Brownian motion, t e [0, 1] is the SDE time index. β t is called the noise schedule, where β t is increasing, and Then there is p1(x1) ~ N(x; μ, Σ). This SDE can also derive the conditional distribution x tx0 ~ N(p(x0,∑,μ,t),λ(∑,t)), where p(.), λ(.) are known closed forms. Therefore, the inventors can directly sample x t In practice, the inventors set ∑ to the identity matrix, so λ(∑,t) becomes λ t I, where λ t is a known closed form scalar. Meanwhile, the inventors condition the terminal distribution p1(x1) on the text, i.e. let μ=μ θ (y), where y is the aligned phoneme representation of the frame. The SDE of formula (1) has a corresponding backward-in-time SDE:

[0056]

[0057] where, is the score function to be estimated, and is a backward-in-time Brownian motion. It shares the trajectory of the distribution p t (x t ) with the forward SDE in formula (1). Therefore, starting from x1 ~ N(μ,∑), the inventors can obtain a realistic sample x0 ~ p(x0|y). A neural network s θ (x t ,y,t) is trained to estimate the score function, in the following score matching objective:

[0058]

[0059] 2.2. Classifier-guided conditional sampling

[0060] The denoising diffusion model provides a new way to model the conditional probability p(x|c), where c is a class label. Suppose the inventors now have an unconditional generative model p(x), and a classifier p(c|x). By Bayes' rule, we have:

[0061]

[0062] In the diffusion framework, to sample from the conditional distribution p(x|c), the inventors need to estimate the score function According to formula (4), the inventors only need to add the gradient of the classifier to the unconditional model. This method of conditional sampling is named classifier-guided, and has also been used in unsupervised TTS.

[0063] In practice, the classifier gradient is often scaled to control the strength of the guidance. The inventors now use Instead of the original Where γ ≥ 0 is called the bootstrap level. Larger γ will lead to highly class-correlated samples, while smaller γ will encourage sample variation.

[0064] Different from ordinary classifiers, the input of the classifier used here is all x along the trajectory of SDE in formula (1) t , instead of just the clean x0. The time index t can be anything in [0,1]. Therefore, the classifier can also be expressed as p(c|x t ,t).

[0065] Although formula (6) can effectively control the sampling of class labels c, it cannot be directly applied to soft labels, that is, labels weighted by intensity, because the guidance p(c|x) is not well defined at present. Therefore, the inventors extended this technique in the subsequent introduction for emotion intensity control.

[0066] 3.EmoDiff

[0067] 3.1. Unconditional Acoustic Model and Classifier Training

[0068] The training of EmoDiff mainly includes the training of unconditional acoustic model and emotion classifier. The inventors first train a diffusion-based acoustic model on the emotion data, but do not provide it with emotion conditions. This is called "unconditional acoustic model training". Figure 2 (a) in the figure. This model is based on GradTTS, but the inventors provide a clear duration sequence by forcing the aligner to facilitate duration modeling. At this stage, the training target is L dur +L diff , where L dur It is logarithmic in duration Loss, L diff is the diffusion loss, as shown in formula (3). In practice, the inventors also use the prior loss L prior = -logN(x0;μ,I) to encourage convergence. To simplify the notation, the inventors Figure 2 In (a), L diff to represent both diffusion and prior loss.

[0069] After training, the acoustic model can estimate the noisy Mel spectrogram x when the phoneme sequence y is input. t The score function of There is no conditional restriction on the emotion label at this time. According to Section 2.2, the inventor needs an emotion classifier to classify the noisy Mel spectrum x tAt the same time, since the inventor always has a text condition y, the classifier is expressed as p(e|x t ,y,t). Figure 2 As shown in (b), the input of the classifier consists of three parts: the timestamp t of the SDE, the noisy Mel spectrum x t and the Gaussian mean μ related to speech. This classifier is trained using the standard cross entropy loss L CE Please note that the inventors freeze the acoustic model parameters at this stage and only update the weights of the sentiment classifier. Otherwise, the trajectory of the diffusion model described above will change, resulting in mathematical errors.

[0070] Since we always need the text y as a condition in the paper, we omit it in the following sections and represent this classifier as p(e|x) to simplify the notation if there is no ambiguity.

[0071] 3.2. Intensity-Controllable Sampling with Soft-Label Guidance

[0072] In the embodiment of the present application, the inventors extend the classifier guidance to the soft label guidance, which can control the intensity weighting of emotions. Assume that the number of basic emotions is m, and each basic emotion e i There is a one-hot vector form e i ∈R m ,i∈{0,1,...,m-1}. For each e i For e, only the i-th dimension is 1. In particular, the inventors use e0 to represent neutrality. i The weighted emotion with the upper intensity of α is defined by the inventor as d = αe i +(1-α)e0. Then, the relationship between the logarithmic probability gradient of the classifier p(d|x) and x can be defined as:

[0073]

[0074]

[0075] The intuitive understanding of this definition is that the intensity α represents the emotion e i The contribution to the sampling trajectory of x. A larger α means that the inventor follows the trajectory of the emotion e i The trajectory with greater "power" samples x, otherwise it is e0. Therefore, the inventor can expand formula (4) to:

[0076]

[0077] When the intensity α is 1.0 (100% of the emotion ei ) or 0.0 (100% neutral), the above procedure simplifies to the standard classifier-guided form of equation (4). Thus, the inventors can use the soft-label guided equation (5) during sampling and produce a realistic sample with a specified intensity a of emotion d = ae i +(1 - a)e0.

[0078] Figure 2 : Training and sampling diagram of EmoDiff. In training, x t is sampled directly from the known distribution p(x t |x0). When sampling with a certain intensity of emotion, the score function is estimated by the score estimator. "SG" denotes the stop-gradient operation.

[0079] Figure 2 (c) in FIG. 4 illustrates the intensity-controllable sampling procedure. After feeding the acoustic model and obtaining the sequence of mu associated with the speech, the inventors sample x1~N(mu, I) and simulate the backward-in-time SDE from t = 1 to t = 0 by the numerical simulator. At each update step of the simulator, the inventors provide the current x t to the classifier and obtain the output probability p t (.|x t ). Then the guiding term is computed by equation (6). Similar to 2.2, the inventors also measure the guiding term by the guiding level g. Finally, the inventors obtain which not only understands the input text but also corresponds to the target emotion d with intensity a. This leads to an accurate intensity, which is well correlated with the probability of the classifier.

[0080] More generally, in addition to intensity control, the soft-label guidance of the embodiments of the present application can enable more complex control of mixed emotions. Let be the combination of all emotions, where w i ∈ [0, 1], Equation (5) can be generalized as:

[0081]

[0082] Then equation (6) can also be expressed in this generalized form. This extension can also be interpreted from the probabilistic perspective. Since the combination weights {w i} can be seen as the classification distribution p i (.) over the basic emotions {e e}, equation (7) is equivalent to

[0083]

[0084] where CE is the cross-entropy function. Equation (9) implies the fact that when following the gradient

[0085] logp(d|x) sampling, the inventors are actually reducing the target emotion distribution p e and the cross-entropy of the classifier output p(.|x). The gradient of the cross-entropy with respect to x can guide the sampling process. Therefore, this soft-label guiding technique can be generally used to control any arbitrary complex emotion as a weighted combination of several basic emotions.

[0086] In Figure 2 In (c) in Equation (8), the inventors use the cross-entropy as a concise notation for the soft-label guiding term. In the intensity control scheme of the embodiments of the present application, it simplifies to the aforementioned Equation (5).

[0087] 4. Experiments and Results

[0088] 4.1. Experimental Setup

[0089] The inventors use the English part of the Emotional Speech Dataset (ESD) for all experiments. It has 10 speakers, each with a comparison of MOS and MCD. The MOS is expressed with a 95% confidence interval. Note that "GradTTS w / emo label" in the experiments cannot control the emotion intensity.

[0090] There are 4 emotion categories: anger, happiness, sadness, surprise, and a neutral category. There are 350 parallel utterances for each speaker and emotion category, about 1.2 hours for each speaker. Melspectrogram and forced alignment are extracted by Kaldi

[32] in 12.5ms frame shift and 50ms frame length, then cepstral normalization is performed. The audio samples in these experiments are public.

[0091] In this application, the inventors only consider the emotion TTS problem of single-person speaking. In the following sections, the inventors train an unconditional GradTTS acoustic model for all 10 English speakers to obtain reasonable data coverage, and only perform classification and control for one female speaker (ID:0015). The unconditional GradTTS model is trained with an Adam optimizer with a learning rate of 10-4 for 11000000 steps. The inventors use an exponential moving average on the model weights, as it can improve the performance of the diffusion model. The structure of the classifier is a 4-layer 1D CNN, each layer with BatchNorm and Dropout. In the inference stage, the guiding level γ is fixed to 100.

[0092] The inventors chose HifiGAN trained on all English users here as vocoder for all experiments below.

[0093] 4.2. Emotional TTS quality

[0094] The inventors first measured speech quality, which includes audio quality and speech naturalness. The inventors compared the proposed EmoDiff with the following systems.

[0095] 1. GT and GT(voc.): Real recording and analyzed-synthesized results (re-synthesized with real mel-spectrogram through vocoder).

[0096] 2. MixedEmotion: It is an autoregressive model based on relative attribute ranking to pre-compute intensity values for training. It is very similar to Emovox's intensity-controllable emotion conversion.

[0097] 3. GradTTS w / emo label: A conditional GradTTS model with hard emotion labels as input. Therefore, it has no intensity controllability, but as a certified acoustic model, it should have good sample quality.

[0098] 4. Note that in this experiment, samples from EmoDiff and MixedEmotion are controlled at an intensity weight of a = 1.0, so they can be directly compared with other samples.

[0099] 5. Figure 3 The mean opinion score (MOS) and mel-cepstral distortion (MCD) are listed. The results show that the vocoder has little effect on sample quality, and the EmoDiff of the present application outperforms the MixedEmotion baseline by a large margin. At the same time, EmoDiff and hard-conditional GradTTS both have good, very close MOS results. Their MCD results only have a small difference. This means that EmoDiff does not impair the intensity controllability of sample quality as MixedEmotion does.

[0100] wherein, Figure 2 The English-Chinese translation comparison in the above is as follows: GT: real recording, GT(voc.): analyzed-synthesis, MixedEmotion does not need to be translated, GradTTS w / emo label: GradTTS with emotion input, EmoDiff is the proposed method, MOS: MOS subjective listening score, MCD: mel-cepstral distortion. The above translation content is universally applicable.

[0101] Figure 4: Classification probability for control intensity α∈{0.0, 0.2, 0.4, 0.6, 0.8, 1.0}. Error bars represent standard deviations. The English-Chinese translation is as follows: Emotion xxx intensity: intensity of xxx emotion, Classifierprob: classifier output probability. Surprise, Sad, Happy, and Angry represent the corresponding emotions (surprise, sadness, joy, anger, respectively. Since this application experiment was conducted in English, no translation is required here).

[0102] 4.3. Controllability of Emotional Intensity

[0103] To assess the controllability of emotion intensity, the inventors used a trained classifier to classify the synthesized samples into controlled intensities. The t input to the classifier was set to 0. The average classification probability for the target emotion category was used as the evaluation metric. Larger values ​​indicate greater discrimination confidence. For each emotion, EmoDiff and MixedEmotion, the inventors varied the intensity from α = 0.0 to 1.0. When the intensity is 0.0, it is equivalent to synthesizing 100% neutral samples. Greater intensities should result in higher probabilities.

[0104] Figure 4 The results are shown. To demonstrate the power of this classifier, the inventors plotted the classification probabilities for the ground truth data. To demonstrate the performance of the hard-conditioned GradTTS model, the inventors also plotted the probabilities for its synthetic samples. Because it lacks intensity control, the inventors only plotted the values ​​at an intensity of 1.0. Here, the standard deviation of each experiment is also presented as error bars.

[0105] It can be seen from the figure that, first of all, the trained classifier has reasonable performance on real data. It is worth mentioning that the classification accuracy on the validation set is 93.1%. Samples from GradTTS w / emo labels have some lower classification probabilities. Most importantly, the proposed EmoDiff always covers a larger range than the baseline, from intensity α = 0.0 to 1.0. The error range of EmoDiff is also always lower than the baseline, which means that the control of the embodiment of the present application is more stable. This proves the effectiveness of the soft label guidance technology proposed by the inventors. The inventors also noticed that sometimes EmoDiff achieves a higher classification probability at an intensity of 1.0 than the hard-conditioned GradTTS. This is also reasonable because conditioning on emotional labels during training does not guarantee better class correlation than classifier guidance if there is a strong classifier and a sufficient level of guidance.

[0106] 4.4. Diversity of Sentiment Samples

[0107] Despite generating high-quality and intensity-controllable emotional samples, EmoDiff also has good sample diversity, benefiting from the powerful generative capability of diffusion models. To evaluate the diversity of emotional samples, the inventors conducted a subjective preference test between EmoDiff and MixedEmotion for each emotion. Listeners were asked to choose the more diverse one or "cannot decide". Note that this test was conducted for each emotion at a weight of a = 1.0.

[0108] Figure 5 Diversity preference test of each emotion. The English-Chinese translation is as follows: Title "Diversity preference test of each emotion": Diversity preference test, Legend's Cannot Decide: Cannot decide, Percent: Percentage, Surprise, Sad, Happy, Angry respectively represent the corresponding emotions (Surprise, Sad, Happy, Angry, since the experiments in this application are done in English, they do not need to be translated).

[0109] Figure 5 The results of the preference are shown. It is clear that for each of the three emotion categories of anger, happiness, and surprise, EmoDiff has a large advantage in diversity. Only in the case of sadness, EmoDiff outperforms the baseline with a small margin. This is mainly because MixedEmotion is autoregressive, and the inventors found that its variation in duration accounted for a large proportion, especially for sad samples.

[0110] 5. Conclusion

[0111] In this application, the inventors studied the intensity control problem in emotional speech synthesis systems. The inventors defined emotions with intensity as the weighted sum of specific emotions and neutral emotions, with the weights being the intensity values. Under this modeling way, the inventors extended the classifier guidance technique to soft label guidance, which enabled the inventors to directly control any arbitrary emotion intensity, rather than a single class label. Through this technique, the proposed EmoDiff can achieve simple and effective control of emotional intensity through an unconditional acoustic model and emotion classifier. Subjective and objective evaluations show that EmoDiff is superior to the baseline in terms of TTS quality, intensity controllability, and sample diversity. In addition, the proposed soft label guidance can be generally applied to control more complex natural emotions, which the inventors will take as future work.

[0112] In some embodiments, the present application also provides a non-transitory computer readable storage medium storing computer executable instructions, which, when executed by a computer, cause the computer to perform any of the speech synthesis methods described above.

[0113] As an implementation, the non-transitory computer readable storage medium of the present application stores computer executable instructions, which are configured to:

[0114] training an emotionless input acoustic model, wherein a diffusion process is used in the training of the acoustic model, and the training objective of the acoustic model is to estimate the log gradient of the data distribution for any intermediate time instant in the diffusion process;

[0115] training an emotion classifier, wherein the input of the emotion classifier includes at least the log gradient corresponding to a certain intermediate time instant in the diffusion process;

[0116] performing emotion controllable speech synthesis sampling using soft label guiding techniques, wherein the diffusion process corresponds to a reverse denoising process of the speech synthesis sampling, the gradient term of the reverse denoising process is the objective estimated by the speech synthesis model, and the soft label guiding term is mathematically equivalent to cross entropy, one side of which is the output of the emotion classifier and the other side is the distribution corresponding to the target emotion intensity.

[0117] The non-transitory computer readable storage medium can include a program storage area and a data storage area, wherein the program storage area can store an operating system and at least one application required by a function; the data storage area can store data created according to the use of the speech synthesis method and system, etc. In addition, the non-transitory computer readable storage medium can include a high-speed random access memory, and can also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the non-transitory computer readable storage medium can optionally include a memory remotely arranged relative to the processor, and these remote memories can be connected to the speech synthesis method through a network. Examples of the above network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0118] The present application also provides a computer program product, which includes a computer program stored on a non-transitory computer readable storage medium, and the computer program includes program instructions, which, when executed by a computer, cause the computer to perform any of the speech synthesis methods described above.

[0119] Figure 6 is a structural schematic diagram of an electronic device provided by the present application, as Figure 6As shown, the device includes one or more processors 610 and a memory 620, Figure 6 In an example, the processor 610 is taken as an example. The device of the speech synthesis method and system can also include an input device 630 and an output device 640. The processor 610, the memory 620, the input device 630 and the output device 640 can be connected by a bus or other means, Figure 6 In an example, the connection by the bus is taken as an example. The memory 620 is the non-volatile computer readable storage medium described above. The processor 610 performs various functional applications and data processing of the server by running the non-volatile software programs, instructions and modules stored in the memory 620, that is, implements the speech synthesis method of the method embodiment described above. The input device 630 can receive input digital or character information, and generate key signal input related to user settings and function control of the training device of the speech synthesis model. The output device 640 can include a display device such as a display screen.

[0120] The above product can perform the method provided by the embodiment of the application, has the corresponding function modules and beneficial effects of performing the method. Technical details not described in detail in the embodiment can be referred to the method provided by the embodiment of the application.

[0121] As an implementation manner, the above electronic device is applied to a speech synthesis method, including: at least one processor; and a memory in communication connection with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to:

[0122] training an emotionless input acoustic model, wherein a diffusion process is used in the training process of the acoustic model, and the training target of the acoustic model is to estimate the log gradient of the data distribution for any intermediate time in the diffusion process;

[0123] training an emotion classifier, wherein the input of the emotion classifier at least includes the log gradient corresponding to a certain intermediate time in the diffusion process;

[0124] using a soft label guidance technique for emotion controllable speech synthesis sampling, wherein the diffusion process corresponds to a reverse denoising process of the speech synthesis sampling, a gradient term of the reverse denoising process is a target estimated by a speech synthesis model, and a soft label guidance term is mathematically equivalent to cross entropy, one side of which is an output of the emotion classifier, and the other side is a distribution corresponding to a target emotion intensity.

[0125] The electronic device of the embodiment of the application exists in various forms, including but not limited to:

[0126] (1) Mobile communication device: The feature of this kind of device is to have mobile communication function, and to provide voice and data communication as the main target. This kind of terminal includes: smart phone, multimedia phone, functional phone, and low-end phone, etc.

[0127] (2) Ultra mobile personal computer device: This kind of device belongs to the category of personal computer, has computing and processing function, and generally has mobile internet feature. This kind of terminal includes: PDA, MID and UMPC device, etc.

[0128] (3) Portable entertainment device: This kind of device can display and play multimedia content. This kind of device includes: audio and video player, palm game machine, electronic book, and smart toy and portable car navigation device.

[0129] (4) Server: The device providing computing service, the constitution of server includes processor, hard disk, memory, system bus, etc. The server is similar to general computer architecture, but due to the need to provide high reliable service, the requirements in processing capability, stability, reliability, security, scalability, manageability, etc. are higher.

[0130] (5) Other electronic devices with data interaction function.

[0131] The device embodiments described above are only schematic, wherein the units shown as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e. can be located in one place or distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the embodiments. Those skilled in the art can understand and implement without creative labor.

[0132] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be realized by means of software and necessary general hardware platform, and of course can also be realized by hardware. Based on such understanding, the above technical solutions essentially or say the part of contribution to the prior art can be embodied in the form of software product, which can be stored in a computer readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including a plurality of instructions to make a computer device (which can be a personal computer, server, or network device, etc.) execute the method of each embodiment or some part of the embodiment.

[0133] It should be pointed out finally that the above embodiments are only used to illustrate the technical solutions of the present application, but not to limit the same; and although the present application has been described in detail with reference to the foregoing embodiments, it should be appreciated by those skilled in the art that the technical solutions recorded in the foregoing embodiments can be modified, or some technical features thereof can be replaced equivalently; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A speech synthesis method, comprising: Train an acoustic model with no emotion input, wherein a diffusion process is used in the training process of the acoustic model, and the training goal of the acoustic model is to estimate the logarithmic gradient of the data distribution at any intermediate moment in the diffusion process, and the acoustic model with no emotion input can estimate the noisy mel-spectrogram x when the phoneme sequence y is input. t The logarithmic gradient of , where the diffusion process constructs a forward stochastic differential equation to transform the data distribution into a known distribution; Train a sentiment classifier, wherein the input of the sentiment classifier includes a noisy Mel spectrogram x t , the timestamp of the forward stochastic differential equation and the Gaussian mean related to the speech; Emotionally controllable speech synthesis sampling is performed using soft label guidance technology, a trained acoustic model without emotion input, and a trained emotion classifier, wherein the diffusion process corresponds to an inverse denoising process of the speech synthesis sampling, the gradient term of the inverse denoising process is the target estimated by the speech synthesis model, and the soft label guidance term is mathematically equivalent to the cross entropy, one side of which is the output of the emotion classifier and the other side is the distribution corresponding to the target emotion intensity.

2. The method according to claim 1, wherein The sentiment classifier freezes the acoustic model parameters during training and only updates the weight of the sentiment classifier.

3. The method according to claim 1, wherein The sentiment classifier is trained using the standard cross entropy loss (LCE).

4. The method according to claim 1, wherein The acoustic model learns how to generate realistic mel-spectrograms from a given text and a given duration sequence through training. The input of the sentiment classifier also includes a representation related to the given text and a duration related to the given text.

5. The method according to claim 1, wherein The diffusion process employs a denoised diffusion model that uses an exponential moving average on the model weights.

6. An electronic device comprising: At least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the steps of the method according to any one of claims 1 to 5.

7. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images