A fine-grained emotional speech synthesis method emphasizing controllability

Abstract

The present invention discloses a method for synthesizing emotional speech with controllable fine-grained emphasis, and belongs to the technical field of speech synthesis. The method comprises the following steps: unsupervised emphasis word annotation is performed on an existing English emotional speech data set to construct a data set; a speech synthesis model with multi-scale prosody control is constructed; the model is trained using the data set obtained in step 1; any English text with specified target emotion and emphasis is input into the trained model to obtain the target speech. The method proposed in the present invention does not require manual annotation of data, but relies on unsupervised generation of emphasis word annotation, thereby reducing the cost of data set production. In addition, through multi-scale prosody modeling, effective control of global and local prosody features is achieved. The present invention adopts a classifier-guided diffusion model for global emotion modeling, and simultaneously uses three features of duration, fundamental frequency and energy to control local prosody, which significantly improves the naturalness and accuracy of speech synthesis.

Description

Technical Field

[0001] The present invention belongs to the technical field of speech synthesis, and in particular relates to a fine-grained, controllable emotional speech synthesis method. Background Art

[0002] With the rapid development of speech synthesis technology, emotional expression and prosody control have become important research areas to enhance the human-computer interaction experience. Linguistic research shows that the prosodic features of speech are multi-layered, covering multiple levels from global emotional atmosphere to local phoneme stress. From a coarse-grained perspective, the overall style or emotional category of the entire discourse can be summarized. For example, it can be considered that the emotion conveyed by a sentence is "happiness". From a fine-grained perspective, each pronunciation unit in a sentence has its own characteristics, such as speed, pitch, and energy. Without fine-grained control, it will be difficult to generate delicate prosodic changes, such as extension, pauses, rising and falling tones, emphasis, etc. Therefore, a single-scale modeling method cannot meet the needs of complex prosodic control.

[0003] Emphasis, the emphasis placed on specific words or phrases in speech, is crucial to the expression of semantics. Changing the placement of emphasis in a sentence, such as "I never said he stole my bag," by moving the emphasis from "he" to "stole" can completely change its semantics. For speech synthesis models, these subtle features that affect semantics are essential in the model's processing if they are to accurately represent speech. However, traditional speech synthesis methods typically focus on single-scale features and ignore the multi-level nature of speech. For example, many TTS systems based on sequence-to-sequence models only model sentiment at a global level, failing to fully capture and reproduce fine-grained prosodic variations. Summary of the invention

[0004] In order to make up for the shortcomings of the prior art, the present invention provides a fine-grained emphasis-controllable emotional speech synthesis method. For any input English text with target emotion and emphasis annotations, the method adopts a multi-scale prosody-controlled speech synthesis model to generate speech that embodies the target emotion and has target emphasis.

[0005] The technical problem solved by the present invention can be achieved through the following specific technical solutions:

[0006] The method for synthesizing fine-grained emotional speech with controllable emphasis comprises the following steps:

[0007] Step 1: Perform unsupervised emphasis word annotation on the existing English emotional speech dataset to construct the dataset;

[0008] Step 2: Construct a speech synthesis model with multi-scale prosody control;

[0009] Step 3: Use the data set obtained in step 1 to train the speech synthesis model;

[0010] Step 4: Input any English text with specified target emotion and emphasis into the trained model to obtain the target speech.

[0011] Furthermore, the data set in step 1 is generated by performing unsupervised annotation on an existing English emotional speech data set through a pre-trained emphasis word recognition model, and each annotated speech data contains a matching text, emotion label, and emphasis label.

[0012] Furthermore, in step 2, the input of the speech synthesis model with multi-scale prosody control is a phoneme sequence extracted from the text, and the output is a target Mel-spectrogram, which can then generate corresponding speech through a vocoder.

[0013] Furthermore, the speech synthesis model with multi-scale prosody control is composed of a phoneme encoder, a local prosody predictor, an emphasis predictor, a Gaussian upsampling model, and a diffusion model in series; wherein the phoneme encoder extracts a potential representation of a sequence from an embedded phoneme sequence; there are three local prosody predictors, namely, a duration predictor, a fundamental frequency predictor, and an energy predictor, which are used to estimate the scalar values ​​of duration, fundamental frequency, and energy of each phoneme, respectively; the emphasis predictor further controls the local prosody features by calculating the emphasized word variance loss through the introduced word-level emphasis label to achieve word-level emphasis; the Gaussian upsampling module maps the phoneme-level features to the frame level by smoothly expanding each phoneme into multiple frames, which is used to solve the length mismatch problem between the phoneme sequence and the spectrogram sequence; the diffusion model introduces a sentiment classifier to guide its reverse denoising process, and uses the Mel-spectrogram features to restore the global prosody.

[0014] Furthermore, in the speech synthesis model with multi-scale prosody control, in the diffusion model, the forward SDE (Stochastic Differential Equation) describes the process of gradually disturbing the original data into noise, and its formula is:

[0015] ,

[0016] in, is the target distribution for regression during the generation process; It is with time The noise-related scheduling parameters, over time As the value of increases, the noise gradually increases;

[0017] Reverse SDE is used to gradually restore data from noise. Through the learning denoising process, a mel-spectrogram with a distribution similar to the original data is restored from Gaussian noise. The reverse denoising process also introduces emotional information through the emotion classifier to make the generated speech conform to the specified emotion category. The specific reverse SDE formula is:

[0018] ,

[0019] Among them, in addition to the denoising term in the conventional diffusion model , and an additional global rhythmic gradient term , used to guide the injection of emotional information by adjusting the weight parameters of the emotional gradient , the intensity of emotion expression in the generated speech can be controlled.

[0020] Furthermore, in the model training process of step 3, the sentiment classifier is pre-trained using cross entropy loss, and a loss function consisting of basic prediction error loss, emphasis word variance loss, diffusion loss, and prior loss is used for back propagation to learn network parameters, and the expression is:

[0021] ,

[0022] in, is the basic prediction error loss, which is the mean square error of the duration, fundamental frequency, and energy predicted in the local prosody predictor; To emphasize word variance loss, it is the mean square error that emphasizes word variance features; Diffusion loss is used to minimize the difference between the scores generated by the diffusion model and the scores of the real data, ensuring that the diffusion model can gradually restore high-quality Mel-spectrograms from noise; It is the prior loss term in the diffusion model, which is used to constrain the generated Mel-spectrogram to be closer to a given prior distribution; , , is the weight coefficient, which is used to balance the weights of different loss items in the total loss.

[0023] Furthermore, in the model training process, the variance features of emphasized words used include pitch variance and phoneme duration variance. Variance of phoneme duration Defined as:

[0024] ,

[0025] ,

[0026] in, and are the logarithmic pitch distributions in words and sentences, respectively, and are the average duration of each phoneme in words and sentences, respectively.

[0027] Furthermore, during the model training process, diffusion loss is the main loss function in the diffusion model, which is derived from the training objective of the diffusion model. This loss is used to estimate the process of restoring the Mel spectrum from noise:

[0028] ,

[0029] in, is the score function used to estimate the diffusion process; is a time step scheduling factor, which is used to control the weight of losses at different time steps;

[0030] The prior loss term in the diffusion model is defined as:

[0031] ,

[0032] in, Indicates that the generated Mel frequency spectrum has a mean value of , the covariance is the identity matrix The degree of match between the Gaussian distributions.

[0033] Compared with the prior art, the present invention has the following advantages:

[0034] (1) The dataset used by the fine-grained emphasis-controllable emotional speech synthesis method proposed in the present invention does not require manual annotation of the emphasis positions. Instead, it is generated by unsupervised annotation of the original emotional speech dataset through a pre-trained emphasis word recognition model, which significantly reduces the cost of dataset production.

[0035] (2) The present invention realizes effective control of prosodic features at the global scale (sentence level) and local scale (phoneme level, word level, frame level) through multi-scale prosodic modeling, thus making up for the deficiency of traditional speech synthesis models in single-scale modeling.

[0036] (3) The speech synthesis model with multi-scale prosodic modeling proposed in the present invention introduces a classifier-guided diffusion model for global prosodic modeling, and introduces three basic prosodic features (duration, fundamental frequency, and energy) and emphasized word variance loss for local emphasis control, thereby improving the expressiveness, naturalness, and accuracy of speech synthesis. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 It is a flow chart of the fine-grained and controllable emotional speech synthesis method of the present invention;

[0038] Figure 2 A schematic diagram of a speech synthesis model with multi-scale prosody control constructed for the present invention;

[0039] Figure 3 It is a schematic diagram of the structure of the local rhythm predictor involved in the present invention;

[0040] Figure 4 Schematic diagram of the Gaussian upsampling module structure involved in the present invention;

[0041] Figure 5 This is a schematic diagram of the principle of the classifier-guided diffusion model involved in the present invention. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical solution and advantages of the present invention clearer, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0043] like Figure 1 As shown, a fine-grained emphasis controllable emotional speech synthesis method includes the following steps:

[0044] Step 1: Perform unsupervised emphasis word annotation on the existing English emotional speech dataset to construct the dataset.

[0045] The data set is generated by unsupervised annotation of an existing English emotional speech data set through a pre-trained emphasis word recognition model. Each annotated speech data contains a matching text, an emotion label, and an emphasis label. In this embodiment, the method provided in the paper "Crowdsourced and Automatic Speech Prominence Estimation" is used as a pre-trained emphasis word recognition model to perform unsupervised emphasis annotation on the ESD (Emotional Speech Database) emotional speech data set. The ESD emotional speech data set is proposed in the paper "Emotional voice conversion: Theory, databases and ESD". It consists of 350 parallel voices recorded by 10 native English speakers and 10 native Chinese speakers, covering 5 emotion categories (neutral, happy, angry, sad and surprised). The present invention uses the English part.

[0046] Step 2: Construct a speech synthesis model with multi-scale prosody control.

[0047] like Figure 2The figure shows the speech synthesis model with multi-scale prosody control proposed by the present invention, whose input is a phoneme sequence extracted from text and output is a target Mel-spectrogram, which can then generate the corresponding speech through a vocoder. The speech synthesis model with multi-scale prosody control is composed of a phoneme encoder, a local prosody predictor, an emphasis predictor, a Gaussian upsampling model, and a diffusion model in series; the phoneme encoder extracts a potential representation of the sequence from the embedded phoneme sequence; there are three local prosody predictors, namely duration predictor, fundamental frequency predictor, and energy predictor, which are used to estimate the scalar values ​​of duration, fundamental frequency, and energy of each phoneme, respectively; the emphasis predictor further controls the local prosody features by calculating the variance loss of the emphasized words through the introduction of word-level emphasis labels to achieve word-level emphasis; the Gaussian upsampling module maps the phoneme-level features to the frame level by smoothly expanding each phoneme into multiple frames, which is used to solve the length mismatch problem between the phoneme sequence and the spectrogram sequence; the diffusion model guides its reverse denoising process by introducing a sentiment classifier, and uses the Mel-spectrogram features to restore the global prosody.

[0048] In this embodiment, the phoneme encoder includes a multi-head self-attention layer and two 1D convolutional layers, each with a residual connection and layer normalization; the local prosody predictor uses Figure 3 The structure shown in Figure 1 is composed of two 1D convolutional networks with ReLU activation and a linear layer; the Gaussian upsampling module adopts Figure 4 In the structure shown, three 1D convolutional layers project duration, pitch, and energy to match the dimensions of the phoneme encoding. The range predictor performs an addition operation on the projection and the phoneme encoding before predicting a series of ranges using a linear layer activated by SoftPlus. Finally, a Gaussian upsampling layer adds the pitch and energy projections to the phoneme encoding and upsamples the sequence using the duration of an integer frame as the mean and the range as the standard deviation; the principle of the classifier-guided diffusion model is as follows Figure 5 As shown in the figure, in the diffusion model, the forward SDE (Stochastic Differential Equation) describes the process of gradually disturbing the original data into noise, and its formula is:

[0049] ,

[0050] in, is the target distribution for regression during the generation process; is a time-dependent noise scheduling parameter, which changes with time As the value of increases, the noise gradually increases.

[0051] Inverse SDE is used to gradually restore data from noise. Through the learning denoising process, a mel-spectrogram similar to the original data distribution is restored from Gaussian noise; the inverse denoising process also introduces emotional information through the emotion classifier so that the generated speech conforms to the specified emotion category. The specific inverse SDE formula is:

[0052] ,

[0053] Among them, in addition to the denoising term in the conventional diffusion model , and an additional global rhythmic gradient term , which is used to guide the injection of emotional information. By adjusting the weight parameter of the emotional gradient , we can control the intensity of emotion expression in the generated speech. For example, a larger This will make the generated speech show stronger emotional characteristics.

[0054] Step 3: Use the data set obtained in step 1 to train the model.

[0055] In this embodiment, the sentiment classifier is pre-trained using cross entropy loss. The overall loss function consisting of basic prediction error loss, emphasis word variance loss, diffusion loss, and prior loss is used for back propagation to learn network parameters. The expression is:

[0056] ,

[0057] in, is the basic prediction error loss, which is the mean square error of the duration, fundamental frequency, and energy predicted in the local prosody predictor; To emphasize word variance loss, it is the mean square error that emphasizes word variance features; Diffusion loss is used to minimize the difference between the scores generated by the diffusion model and the scores of the real data, ensuring that the diffusion model can gradually restore high-quality Mel-spectrograms from noise; It is the prior loss term in the diffusion model, which is used to constrain the generated Mel-spectrogram to be closer to a given prior distribution; , , is the weight coefficient, which is used to balance the weights of different loss items in the total loss.

[0058] During the model training process, the variance features of emphasized words used include pitch variance and phoneme duration variance. Variance of phoneme duration Defined as:

[0059] ,

[0060] ,

[0061] in, and are the logarithmic pitch distributions in words and sentences, respectively, and are the average duration of each phoneme in words and sentences, respectively.

[0062] Diffusion loss is the main loss function in the diffusion model, which is derived from the training objective of the diffusion model. This loss is used to estimate the process of restoring the Mel spectrum map from noise:

[0063] ,

[0064] in, is the score function used to estimate the diffusion process; is a time step scheduling factor that controls the weight of losses at different time steps.

[0065] The prior loss term in the diffusion model is defined as:

[0066] ,

[0067] in, Indicates that the generated Mel frequency spectrum has a mean value of , the covariance is the identity matrix The degree of match between the Gaussian distributions.

[0068] Step 4: Input any English text with specified target emotion and emphasis into the trained model to obtain the target speech.

[0069] In the inference stage, you only need to input any English text with the specified target emotion and emphasis into the trained model to obtain the target speech.

[0070] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A fine-grained emotional speech synthesis method with controllable emphasis, characterized in that: The following steps are involved: Step 1: Perform unsupervised emphasis word annotation on the existing English emotional speech dataset to construct the dataset; Step 2: Construct a speech synthesis model with multi-scale prosody control; Step 3: Use the data set obtained in step 1 to train the speech synthesis model; Step 4: Input any English text with specified target emotion and emphasis into the trained model to obtain the target speech; In step 2, the input of the speech synthesis model with multi-scale prosody control is a phoneme sequence extracted from the text, and the output is a target Mel-spectrogram, which can then generate corresponding speech through a vocoder; the speech synthesis model with multi-scale prosody control is composed of a phoneme encoder, a local prosody predictor, an emphasis predictor, a Gaussian upsampling model, and a diffusion model in series; wherein the phoneme encoder extracts a sequence potential representation from an embedded phoneme sequence; there are three local prosody predictors, namely, a duration predictor, a fundamental frequency predictor, and an energy predictor, which are used to estimate the scalar values ​​of the duration, fundamental frequency, and energy of each phoneme, respectively; the emphasis predictor further controls the local prosody features by introducing a word-level emphasis label and calculating the emphasized word variance loss to achieve word-level emphasis; the Gaussian upsampling module maps the phoneme-level features to the frame level by smoothly expanding each phoneme into multiple frames, so as to solve the length mismatch problem between the phoneme sequence and the spectrogram sequence; the diffusion model guides its reverse denoising process by introducing an emotion classifier, and uses the Mel-spectrogram features to restore the global prosody.

2. According to claim 1, a fine-grained emphasis controllable emotional speech synthesis method is characterized in that: The data set in step 1 is generated by unsupervised labeling of an existing English emotional speech data set using a pre-trained emphasis word recognition model. Each piece of speech data after labeling contains a matching text, an emotion label, and an emphasis label.

3. A fine-grained emphasis controllable emotional speech synthesis method according to claim 1, characterized in that: Speech synthesis model with multi-scale prosody control. In the diffusion model, forward SDE describes the process of gradually disturbing the original data into noise. The formula is: ， in, is the target distribution for regression during the generation process; It is with time The noise-related scheduling parameters, over time As the value of increases, the noise gradually increases; Reverse SDE is used to gradually restore data from noise. Through the learning denoising process, a mel-spectrogram with a distribution similar to the original data is restored from Gaussian noise. The reverse denoising process also introduces emotional information through the emotion classifier to make the generated speech conform to the specified emotion category. The specific reverse SDE formula is: ， Among them, in addition to the denoising term in the conventional diffusion model , and an additional global rhythmic gradient term , used to guide the injection of emotional information by adjusting the weight parameters of the emotional gradient , the intensity of emotion expression in the generated speech can be controlled.

4. The method for synthesizing emotional speech with fine-grained emphasis and controllable according to claim 1, characterized in that: In the model training process of step 3, the sentiment classifier is pre-trained using cross entropy loss, and a loss function consisting of basic prediction error loss, emphasis word variance loss, diffusion loss, and prior loss is used for back propagation to learn network parameters. The expression is: ， in, is the basic prediction error loss, which is the mean square error of the duration, fundamental frequency, and energy predicted in the local prosody predictor; To emphasize word variance loss, it is the mean square error that emphasizes word variance features; Diffusion loss is used to minimize the difference between the scores generated by the diffusion model and the scores of the real data, ensuring that the diffusion model can gradually restore high-quality Mel-spectrograms from noise; It is the prior loss term in the diffusion model, which is used to constrain the generated Mel-spectrogram to be closer to a given prior distribution; , , is the weight coefficient, which is used to balance the weights of different loss items in the total loss.

5. A fine-grained emphasis controllable emotional speech synthesis method according to claim 4, characterized in that: In the model training process, the variance features of emphasized words used include pitch variance and phoneme duration variance. Variance of phoneme duration Defined as: ， ， in, and are the logarithmic pitch distributions in words and sentences, respectively, and are the average duration of each phoneme in words and sentences, respectively.

6. A fine-grained emphasis controllable emotional speech synthesis method according to claim 5, characterized in that: During the model training process, diffusion loss is the main loss function in the diffusion model. It is derived from the training objective of the diffusion model and is used to estimate the process of restoring the Mel spectrum from noise: ， in, is the score function used to estimate the diffusion process; is a time step scheduling factor, which is used to control the weight of losses at different time steps; The prior loss term in the diffusion model is defined as: ， in, Indicates that the generated Mel-spectrogram has a mean value of , the covariance is the unit matrix The degree of match between the Gaussian distributions.

Images