Multi-rhythm-guided emotion intensity controllable speech synthesis controller and control method
By constructing a multi-prosodic guided speech synthesis controller with controllable emotional intensity, and using pitch and energy predictors to explicitly model and predict speech prosodic features, the problem of continuous control of emotional intensity in existing technologies is solved, achieving continuous and stable control of emotional intensity and improving the quality and expressiveness of speech synthesis.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-03-05
- Publication Date
- 2026-05-15
AI Technical Summary
Existing speech synthesis controllers struggle to achieve continuous control of emotional intensity, and directly manipulating emotional embeddings can lead to a decline in speech quality. They also ignore the intrinsic relationship between emotional intensity and speech prosodic features and lack effective prosodic modeling schemes.
A multi-prosodic guided speech synthesis controller with controllable emotional intensity is constructed. Pitch and energy features are explicitly modeled and predicted through pitch predictors and energy predictors. A predictive modulation module is introduced to map the target emotion embedding and generate modulation bias terms. The processed features are then fused with text features through pitch modulation and energy modulation modules. Finally, the scale is adjusted by adjusting control parameters to achieve continuous control of emotional intensity.
It achieves continuous, flexible and stable control of emotional intensity while ensuring the naturalness and structural stability of speech, thereby improving the subtlety and overall quality of emotional expression in synthesized speech, enhancing the generalization ability and robustness of the model, and avoiding speech distortion problems.
Smart Images
Figure CN122050356A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to a speech synthesis controller in the field of speech synthesis technology, and more particularly to a speech synthesis controller with multi-prosodic guided emotion intensity controllable, and also to a speech synthesis control method with multi-prosodic guided emotion intensity controllable. Background Technology
[0002] With the development of deep learning technology, speech synthesis has made significant progress in terms of naturalness and intelligibility, and is gradually evolving from traditional neutral speech synthesis to emotional speech synthesis. Emotional speech synthesis introduces emotional information into synthesized speech, making the generated speech closer to real human speech in terms of intonation, rhythm, and expressiveness, which is of great value in applications such as intelligent voice assistants, human-computer interaction, and virtual characters. Most existing emotional speech synthesis methods control the emotional category of synthesized speech by introducing emotional tags or emotional embedding vectors. However, in practical applications, emotions not only have category attributes but also continuously changing intensity, i.e., emotional strength. How to achieve refined and continuous control of emotional strength while ensuring speech naturalness and emotional consistency remains a problem that urgently needs to be solved in current technologies.
[0003] Currently, some technical solutions achieve control by manually labeling emotional intensity levels or constructing emotional intensity tags. However, these methods rely on a large amount of manually labeled data, resulting in high labeling costs and strong subjectivity, making it difficult to cover the range of emotional changes among different speakers and in different contexts. Another approach attempts to directly scale, interpolate, or weight the emotional embedding vector to adjust emotional intensity. However, since the emotional embedding space is usually automatically learned by the model, its internal structure is complex and lacks clear physical or phonetic meaning. Directly manipulating the emotional embedding can easily disrupt the original emotional representation structure, leading to problems such as emotional distortion, decreased speech quality, or instability in synthesized speech. Furthermore, existing methods often ignore the intrinsic relationship between emotional intensity and prosodic features (such as pitch and energy), lacking an effective solution that can achieve controllable adjustment of emotional intensity through prosodic modeling without directly interfering with emotional representation. Summary of the Invention
[0004] To address the technical problems of existing speech synthesis controllers, such as difficulty in continuous control and low quality, this invention provides a speech synthesis controller and control method with multi-prosodic guided emotion intensity controllability.
[0005] This invention is achieved using the following technical solution: a multi-prosodic guided speech synthesis controller with controllable emotional intensity, whose input includes target emotion embedding and text features fused with emotional information, and further includes: A pitch predictor is used to model and predict pitch features to generate pitch prediction results for continuously adjusting emotional intensity. An energy predictor is used to model and predict energy characteristics, generating energy prediction results for continuous adjustment of emotion intensity. A prediction modulation module is used to map the target emotion embedding, generate modulation bias terms, and add them to the pitch prediction result and the energy prediction result, respectively. The pitch modulation module is used to fuse the pitch features processed by the prediction modulation module with the text features, and to perform feature alignment and amplitude constraint. An energy modulation module is used to fuse the energy features processed by the prediction modulation module with the text features, and to perform feature alignment and amplitude constraint. Specifically, by adjusting the pitch emotion intensity control parameter and the energy emotion intensity control parameter, the outputs of the pitch modulation module and the energy modulation module are scaled to achieve continuous control of the synthesized speech emotion intensity.
[0006] This invention explicitly models and predicts pitch and energy features by constructing pitch predictors and energy predictors. It introduces a prediction modulation module to map the target emotion embedding to generate modulation bias terms, which are added to the pitch and energy prediction results. Then, the processed pitch and energy features are fused with text features and aligned and constrained by pitch modulation and energy modulation modules. Finally, the output is scaled by adjusting the pitch and energy emotion intensity control parameters. This invention solves the problems of difficulty in continuously controlling emotion intensity, easy degradation of speech quality due to direct manipulation of emotion embedding, and strong dependence on artificial emotion intensity labels in the prior art. It achieves continuous, flexible and stable control of emotion intensity while ensuring the naturalness and structural stability of speech, which significantly improves the subtlety of emotional expression and the overall quality of synthesized speech.
[0007] As a further improvement to the above scheme, both the pitch predictor and the energy predictor adopt a conditional variational autoencoder structure, and the loss function of the conditional variational autoencoder is: In the formula, To predict target features; As a conditional variable, it is composed of target sentiment embeddings; As latent variables; The posterior distribution of the encoder output. For decoder, It is the prior distribution; Kullback-Leibler divergence is used to measure... and The differences between them.
[0008] As a further improvement to the above scheme, the scale scaling is achieved through the following formula: In the formula, The pitch emotional intensity control parameter, The energy emotion intensity control parameter, To predict the obtained pitch features, To predict the obtained energy characteristics, This is the scaled pitch characteristic. This represents the scaled energy characteristics.
[0009] As a further improvement to the above scheme, the loss functions of the pitch predictor and the energy predictor during the training process include: pitch prediction mean square error loss, energy prediction mean square error loss, KL divergence of the pitch latent variable distribution, and KL divergence of the energy latent variable distribution.
[0010] As a further improvement to the above scheme, the prediction modulation module maps the target emotion embedding into pitch modulation bias and energy modulation bias through a linear transformation layer.
[0011] As a further improvement to the above scheme, both the pitch modulation module and the energy modulation module include a linear projection layer and a nonlinear activation function.
[0012] As a further improvement to the above scheme, the pitch prediction result is based on a pitch spectrogram, which is obtained by performing a continuous wavelet transform on the pitch profile after logarithmic normalization.
[0013] As a further improvement to the above scheme, the pitch emotion intensity control parameter and the energy emotion intensity control parameter are adjustable scalars, and the emotion intensity is continuously controlled by scaling the pitch features and energy features obtained from the prediction during the inference stage.
[0014] As a further improvement to the above scheme, the pitch features and energy features are aligned with the text sequence at the phoneme level using a pre-trained forced alignment tool before training.
[0015] This invention also provides a multi-prosodic guided speech synthesis control method with controllable emotional intensity, which applies any of the multi-prosodic guided speech synthesis controllers with controllable emotional intensity described above. The control method includes the following steps: Modeling and predicting pitch and energy features to generate pitch prediction results and energy prediction results for continuously adjusting emotional intensity. The target emotion embedding is mapped to generate a modulation bias term, which is then added to the pitch prediction result and the energy prediction result, respectively. The modulated pitch and energy features are fused with the text features, and feature alignment and amplitude constraints are performed. By adjusting the pitch emotion intensity control parameter and the energy emotion intensity control parameter respectively, the outputs of the pitch modulation module and the energy modulation module are scaled to achieve continuous control of the synthesized speech emotion intensity.
[0016] Compared with existing speech synthesis controllers and methods, the multi-prosodic guided emotion intensity controllable speech synthesis controller and control method of the present invention have the following beneficial effects: 1. This multi-prosodic guided speech synthesis controller with controllable emotional intensity explicitly models and predicts pitch and energy features by constructing pitch and energy predictors. A predictive modulation module maps the target emotion embedding to generate modulation bias terms, which are added to the pitch and energy prediction results. The processed pitch and energy features are then fused with text features and aligned and constrained by pitch and energy modulation modules. Finally, the output is scaled by adjusting the pitch and energy emotion intensity control parameters. This staged modulation strategy effectively enhances the system's adaptability under different emotion categories, intensities, and content conditions, significantly reduces dependence on specific training data distributions, improves the model's generalization ability and practical application value, and solves the problems of continuous emotional intensity control, direct manipulation of emotion embedding leading to speech quality degradation, and strong dependence on artificial emotion intensity labels in existing technologies. It achieves continuous, flexible, and stable control of emotional intensity while ensuring speech naturalness and structural stability, significantly improving the emotional expression subtlety and overall quality of synthesized speech.
[0017] 2. This multi-prosodic guided emotion intensity controllable speech synthesis controller introduces an emotion intensity controller into the speech synthesis system, explicitly applying emotion intensity to two key prosodic dimensions: pitch and energy features. Adjustable control parameters scale the prosodic features, thereby achieving continuous adjustment of emotion intensity without altering the emotion category. Compared to existing methods that only support discrete emotion label switching, this method can more precisely express the differences in emotion at different intensity levels and accurately control the intensity levels within the same emotion, greatly enriching the emotional expressiveness of synthesized speech.
[0018] 3. This multi-prosodic guided speech synthesis controller with controllable emotional intensity incorporates a conditional variational autoencoder structure in the pitch and energy prediction processes. Utilizing the characteristics of its continuous latent space, it models prosodic features, resulting in a smooth transition between predicted pitch and energy and changes in emotional intensity. When adjusting the pitch and energy emotional intensity control parameters, the synthesized speech's emotional expression can continuously change from flat to full, effectively avoiding problems such as abrupt changes in pitch or energy and abnormal amplification caused by abrupt emotional embedding in traditional methods. This significantly improves the naturalness and stability of the synthesized speech.
[0019] 4. The multi-prosodic guided emotion intensity controllable speech synthesis controller modulates and constrains prosodic features in stages through a prediction modulation module, a pitch modulation module, and an energy modulation module. This ensures that the prosodic features are limited to a reasonable range before entering the speech synthesis model, thereby enhancing the robustness of the system under different emotion categories, different emotion intensities, and different speech content conditions, and reducing the model's dependence on specific data distributions.
[0020] 5. This multi-prosodic guided speech synthesis controller with controllable emotional intensity decouples complex emotional expressions in speech into two independent dimensions—pitch features and energy features—for explicit modeling and prediction by constructing pitch predictors and energy predictors. By introducing a predictive modulation module, it utilizes the target emotion embedding to generate modulation bias terms, which are then added to the pitch and energy prediction results, achieving precise intervention of emotional intensity on underlying acoustic features. Compared to existing methods that directly manipulate mixed emotion embeddings, this controller's decoupling control mechanism avoids speech distortion caused by feature coupling, significantly improving the clarity and naturalness of synthesized speech, and possesses a refined prosodic decoupling and independent control mechanism.
[0021] 6. The beneficial effects of this multi-prosodic guided emotion intensity controllable speech synthesis method are the same as those of the controller mentioned above, and will not be elaborated here. Attached Figure Description
[0022] Figure 1 This is a schematic diagram of the overall model structure of the speech synthesis system containing the multi-prosodic guided emotion intensity controllable speech synthesis controller of Embodiment 1 of the present invention. Detailed Implementation
[0023] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.
[0024] Example 1 Emotional intensity in speech is mainly reflected through prosodic features such as the amplitude of pitch and energy changes, and these prosodic features exhibit continuous changes over time. If only the emotion category is discretely controlled, it is often difficult to accurately express the emotional differences at different intensity levels, which can easily lead to monotonous or unnatural emotional expression in speech.
[0025] For the above issues, please refer to Figure 1 This embodiment provides a multi-prosodic guided speech synthesis controller with controllable emotional intensity. This controller is also an emotional intensity control module that can be applied to emotional speech synthesis systems to explicitly model and regulate emotional prosodic features during speech synthesis. The controller takes as input text features with target emotional embeddings and fused emotional information, and predicts and models the two key prosodic attributes of pitch and energy required in the speech synthesis process. The controller includes a pitch predictor, an energy predictor, a predictive modulation module, a pitch modulation module, and an energy modulation module. These modules work together to achieve flexible adjustment of emotional intensity while ensuring the naturalness and structural stability of the speech.
[0026] In this model, text features are used to describe the semantic content and basic prosodic structure of the speech to be synthesized, while target sentiment embeddings are used to represent the target sentiment category and its corresponding sentiment intensity. By explicitly incorporating sentiment information into the prosodic prediction process, the model can controllably model sentiment intensity during speech generation.
[0027] The pitch predictor is used to model and predict pitch features, generating pitch prediction results for continuously adjusting emotional intensity. The pitch prediction results are based on a pitch spectrogram, which is obtained by performing a continuous wavelet transform on a logarithmically normalized pitch profile. Specifically, firstly, the pitch profile F0(t) is extracted from the original audio using PyWorldVocoder, and then padded, logarithmically transformed, and normalized. Subsequently, a continuous wavelet transform (CWT) is applied to the processed pitch sequence to obtain a pitch spectrogram that more finely represents the continuous pitch variation characteristics; this spectrogram serves as the prediction target for the pitch predictor.
[0028] The energy predictor is used to model and predict energy features, generating energy prediction results for continuously adjusting emotional intensity. The energy features here employ a preprocessing and modeling approach consistent with pitch to unify the prosodic modeling framework.
[0029] In this embodiment, pitch and energy features are aligned with the text sequence at the phoneme level before training using a pre-trained forced alignment tool. Specifically, prosodic features are modeled at the phoneme level. To avoid the uncertainty introduced by the random duration predictor within the model affecting prosodic alignment, a pre-trained Montreal Forced Alignment (MFA) tool is used to align pitch, energy, and text sequence, thereby ensuring the consistency of prosodic features and text features in the temporal dimension.
[0030] During the inference phase, the predicted pitch spectrogram is restored to a pitch profile through inverse continuous wavelet transform and used in speech synthesis. First, pitch and energy features are extracted from the original emotional speech audio as supervision signals for the pitch and energy predictors. The pitch and energy predictors are used to model and predict the pitch and energy features, respectively. Both the pitch and energy predictors can employ a conditional variational autoencoder (CDAE) structure. CDAEs have a natural advantage in modeling continuously changing features; their training objective is to maximize the marginal log-likelihood given conditional variables, and their loss function can be expressed as: In the formula, To predict target features; As a conditional variable, it is composed of target sentiment embeddings; As latent variables; The posterior distribution of the encoder output. For decoder, It is the prior distribution; Kullback-Leibler divergence is used to measure... and The differences between them.
[0031] By modeling pitch and energy features using a conditional variational autoencoder, and leveraging the continuous latent space of this autoencoder, the predicted prosodic features can smoothly change with emotional conditions, thus avoiding the prosodic abrupt change problem common in traditional methods. Furthermore, by introducing emotional embeddings as conditional variables into the predictor, the model can generate prosodic features with corresponding emotional trends based on the target emotion.
[0032] During training, this embodiment optimizes the pitch predictor and energy predictor by calculating the mean squared error loss between predicted pitch features and true pitch features, and the mean squared error loss between predicted energy features and true energy features. Simultaneously, to constrain the distribution of the latent space, Kullback-Leibler divergence is calculated for the latent variable distributions in the pitch predictor and energy predictor, respectively, in the following form: Through the above training methods, the pitch predictor and energy predictor are able to generate stable and continuous prosodic prediction results under emotional condition constraints.
[0033] It should be noted that, in some embodiments, the loss functions of the pitch predictor and the energy predictor during the training process may include: pitch prediction mean squared error loss, energy prediction mean squared error loss, KL divergence of the pitch latent variable distribution, and KL divergence of the energy latent variable distribution.
[0034] After the pitch predictor and energy predictor output their prediction results, the prediction modulation module maps the target emotion embedding, generates modulation bias terms, and adds them to the pitch prediction result and energy prediction result respectively, thereby explicitly modulating the emotion information at the prosodic level. The prediction modulation module can map the target emotion embedding to pitch modulation bias and energy modulation bias through a linear transformation layer.
[0035] The pitch modulation module is used to fuse the pitch features processed by the predictive modulation module with the text features, and to perform feature alignment and amplitude constraints. Specifically, in this embodiment, the pitch features modulated by the predictive modulation module are added element-wise with the text features to obtain the pitch feature representation of the fused text information. The fused pitch features are then input to the pitch modulation module, where linear projection and a nonlinear activation function are used to perform feature alignment and amplitude constraints on the pitch features.
[0036] The energy modulation module is used to fuse the energy features processed by the prediction modulation module with the text features, and to perform feature alignment and amplitude constraints. Similar to the above, this embodiment adds the energy features modulated by the prediction modulation module to the text features element-wise to obtain the energy feature representation of the fused text information. The fused energy features are then input to the energy modulation module to complete the mapping and constraint of the energy features in the same way.
[0037] It should be noted that both the pitch modulation module and the energy modulation module can include a linear projection layer and a nonlinear activation function. Through this modulation process, abnormal amplification or unreasonable fluctuations in pitch and energy features during emotional intensity adjustment can be effectively avoided, thus ensuring the stability of the speech prosodic structure under different emotional intensity conditions.
[0038] Specifically, the controller scales the outputs of the pitch modulation and energy modulation modules by adjusting the pitch and energy emotion intensity control parameters, respectively, to achieve continuous control over the emotion intensity of the synthesized speech. This results in continuous, flexible, and stable control over the emotion intensity of the final synthesized speech. Finally, the prosodic features processed by the energy modulation module are used as overall conditions input into the speech synthesis model to generate a speech signal with the target emotion category and controllable emotion intensity.
[0039] In this embodiment, the pitch-emotion intensity control parameter and the energy-emotion intensity control parameter are adjustable scalars. During the inference phase, continuous control of the emotion intensity is achieved by scaling the predicted pitch and energy features. The scaling is implemented using the following formula: In the formula, This is a parameter for controlling pitch emotional intensity. This is a parameter for controlling the intensity of emotional energy. To predict the obtained pitch features, To predict the obtained energy characteristics, This is the scaled pitch characteristic. This represents the scaled energy characteristics.
[0040] In this way, by adjusting and The value of can be continuously adjusted to control the intensity of emotion without changing the emotion category. Since the pitch and energy features have been constrained and aligned by the modulation module before entering the speech synthesis model, the stability of the overall speech structure and the naturalness of the synthesized speech can still be maintained under different emotion intensities.
[0041] In summary, compared with existing speech synthesis controllers, the multi-prosodic guided emotion intensity controllable speech synthesis controller of this embodiment has the following advantages: 1. This multi-prosodic guided speech synthesis controller with controllable emotional intensity explicitly models and predicts pitch and energy features by constructing pitch and energy predictors. A predictive modulation module maps the target emotion embedding to generate modulation bias terms, which are added to the pitch and energy prediction results. The processed pitch and energy features are then fused with text features and aligned and constrained by pitch and energy modulation modules. Finally, the output is scaled by adjusting the pitch and energy emotion intensity control parameters. This staged modulation strategy effectively enhances the system's adaptability under different emotion categories, intensities, and content conditions, significantly reduces dependence on specific training data distributions, improves the model's generalization ability and practical application value, and solves the problems of continuous emotional intensity control, direct manipulation of emotion embedding leading to speech quality degradation, and strong dependence on artificial emotion intensity labels in existing technologies. It achieves continuous, flexible, and stable control of emotional intensity while ensuring speech naturalness and structural stability, significantly improving the emotional expression subtlety and overall quality of synthesized speech.
[0042] 2. This multi-prosodic guided emotion intensity controllable speech synthesis controller introduces an emotion intensity controller into the speech synthesis system, explicitly applying emotion intensity to two key prosodic dimensions: pitch and energy features. Adjustable control parameters scale the prosodic features, thereby achieving continuous adjustment of emotion intensity without altering the emotion category. Compared to existing methods that only support discrete emotion label switching, this method can more precisely express the differences in emotion at different intensity levels and accurately control the intensity levels within the same emotion, greatly enriching the emotional expressiveness of synthesized speech.
[0043] 3. This multi-prosodic guided speech synthesis controller with controllable emotional intensity incorporates a conditional variational autoencoder structure in the pitch and energy prediction processes. Utilizing the characteristics of its continuous latent space, it models prosodic features, resulting in a smooth transition between predicted pitch and energy and changes in emotional intensity. When adjusting the pitch and energy emotional intensity control parameters, the synthesized speech's emotional expression can continuously change from flat to full, effectively avoiding problems such as abrupt changes in pitch or energy and abnormal amplification caused by abrupt emotional embedding in traditional methods. This significantly improves the naturalness and stability of the synthesized speech.
[0044] 4. The multi-prosodic guided emotion intensity controllable speech synthesis controller modulates and constrains prosodic features in stages through a prediction modulation module, a pitch modulation module, and an energy modulation module. This ensures that the prosodic features are limited to a reasonable range before entering the speech synthesis model, thereby enhancing the robustness of the system under different emotion categories, different emotion intensities, and different speech content conditions, and reducing the model's dependence on specific data distributions.
[0045] 5. This multi-prosodic guided speech synthesis controller with controllable emotional intensity decouples complex emotional expressions in speech into two independent dimensions—pitch features and energy features—for explicit modeling and prediction by constructing pitch predictors and energy predictors. By introducing a predictive modulation module, it utilizes the target emotion embedding to generate modulation bias terms, which are then added to the pitch and energy prediction results, achieving precise intervention of emotional intensity on underlying acoustic features. Compared to existing methods that directly manipulate mixed emotion embeddings, this controller's decoupling control mechanism avoids speech distortion caused by feature coupling, significantly improving the clarity and naturalness of synthesized speech, and possesses a refined prosodic decoupling and independent control mechanism.
[0046] Example 2 This embodiment provides a multi-prosodic guided speech synthesis control method with controllable emotional intensity, which is applied to the multi-prosodic guided speech synthesis controller with controllable emotional intensity in Embodiment 1. This control method explicitly models and regulates emotional prosodic features during speech synthesis, achieving continuous controllability of emotional intensity through staged modulation and scaling. The control method of this embodiment includes the following steps.
[0047] (1) Model and predict pitch features and energy features to generate pitch prediction results and energy prediction results for continuously adjusting emotional intensity. (2) Map the target emotion embedding, generate modulation bias terms, and add them to the pitch prediction result and energy prediction result respectively; (3) The pitch and energy features after modulation are fused with the text features, and feature alignment and amplitude constraints are performed; (4) The outputs of the pitch modulation module and the energy modulation module are scaled by adjusting the pitch emotion intensity control parameter and the energy emotion intensity control parameter respectively, so as to achieve continuous control of the synthesized speech emotion intensity.
[0048] The specific steps of this method will be described in detail below, combining the training phase and the inference phase.
[0049] I. Training Phase During the training phase, pitch and energy features are first extracted from the raw emotional speech data as supervision signals for subsequent prediction. Specifically, PyWorldVocoder is used to extract the pitch contour F0(t) from the raw audio, and then it is padded, logarithmically transformed, and normalized. Subsequently, continuous wavelet transform (CWT) is applied to the normalized pitch sequence to obtain a pitch spectrogram that can accurately represent the continuous pitch variation characteristics. The energy features adopt the same preprocessing flow as the pitch features to unify the prosodic modeling framework.
[0050] To ensure consistency between prosodic features and text features over time, a pre-trained Montreal Forced Alignment (MFA) tool is used to align pitch, energy, and text sequences at the phoneme level, avoiding alignment uncertainty introduced by the random duration predictor within the model.
[0051] After data preprocessing, the text features infused with sentiment information and the target sentiment embedding are used as input to the controller. The control method includes the following core steps: 1. Prosodic Feature Modeling and Prediction: Pitch and energy features are modeled and predicted using pitch and energy predictors, respectively. Both the pitch and energy predictors employ a conditional variational autoencoder (CVAE) structure. Their training objective is to maximize the marginal log-likelihood given conditional variables. The loss function consists of reconstruction loss and KL divergence. (1) Reconstruction loss: Calculate the mean square error (MSE) between the predicted pitch feature and the true pitch feature, and the mean square error between the predicted energy feature and the true energy feature; (2) KL divergence: Calculate the KL divergence for the distribution of latent variables of pitch and energy respectively, constrain the shape of the latent spatial distribution, and make it close to the prior distribution.
[0052] Through the above training, the pitch predictor and energy predictor can generate stable and continuous prosodic prediction results under emotional condition constraints, laying the foundation for subsequent emotional intensity regulation.
[0053] 2. Emotion Embedding Modulation: The target emotion is embedded into the prediction modulation module and mapped to pitch modulation bias and energy modulation bias through a linear transformation layer, which are then added to the pitch prediction result and energy prediction result, respectively. This step explicitly injects emotional information into the prosodic features, achieving preliminary intervention of emotional intensity on the underlying acoustic features.
[0054] 3. Prosody-Text Fusion and Constraints: The modulated pitch features are added element-wise to the text features to obtain the pitch feature representation of the fused text information, which is then input into the pitch modulation module. Similarly, the modulated energy features are added to the text features and input into the energy modulation module. Both the pitch modulation and energy modulation modules contain linear projection layers and nonlinear activation functions to perform feature alignment and amplitude constraints on the fused features, ensuring that the prosodic features fluctuate within a reasonable range and avoiding abnormal amplification during subsequent adjustment.
[0055] 4. Joint Optimization of Training Losses: During training, the following loss functions are jointly optimized: mean squared error loss for pitch prediction, mean squared error loss for energy prediction, KL divergence of the pitch latent variable distribution, and KL divergence of the energy latent variable distribution. The parameters of each module in the controller are updated through backpropagation, enabling the model to learn the mapping relationship between sentiment categories and prosodic features.
[0056] II. Reasoning Stage During the inference phase, given the target sentiment embedding and text features infused with sentiment information, the control method generates speech prosodic features with controllable sentiment intensity according to the following steps: 1. Prosody Feature Prediction: Using a trained pitch predictor and energy predictor, initial pitch prediction results and initial energy prediction results are generated based on the target sentiment embedding. The pitch prediction results exist in the form of a pitch spectrogram, which needs to be subsequently restored to a pitch profile through inverse continuous wavelet transform.
[0057] 2. Emotion Embedding Modulation: The target emotion is embedded into the prediction modulation module, generating a modulation bias term and adding it to the pitch and energy prediction results, so that the prosodic features carry the target emotion information.
[0058] 3. Prosody-Text Fusion and Constraints: The modulated pitch features are added to the text features and input into the pitch modulation module for feature alignment and amplitude constraints; energy features are processed similarly. This step ensures that the prosodic features are within a stable range before emotional intensity adjustment.
[0059] 4. Emotional Intensity Scaling: Adjustable scalar control parameters are introduced: pitch emotional intensity control parameter α and energy emotional intensity control parameter β. The outputs of the pitch modulation module and energy modulation module are scaled, calculated using the following formula: In the formula, This is a parameter for controlling pitch emotional intensity. This is a parameter for controlling the intensity of emotional energy. To predict the obtained pitch features, To predict the obtained energy characteristics, This is the scaled pitch characteristic. This represents the scaled energy characteristics.
[0060] 5. Speech generation: The scaled energy features are input as overall conditions into the speech synthesis model (such as a neural network-based acoustic model and vocoder). At the same time, the scaled pitch features are restored to pitch contours through inverse continuous wavelet transform and also input into the synthesis model. Finally, a speech signal with target emotion category and controllable emotion intensity is generated.
[0061] In summary, this embodiment achieves continuous and stable control of emotional intensity through steps such as explicit prosodic feature modeling, emotion embedding modulation, prosodic-text fusion constraints, and scale scaling. This effectively solves the problems of difficulty in finely adjusting emotional intensity and easy degradation of speech quality in existing technologies. This method has advantages such as simple implementation, flexible control, and strong robustness, and can be widely applied in scenarios such as intelligent voice interaction and virtual digital humans.
[0062] Example 3 This embodiment provides a speech synthesis system, which includes the multi-prosodic guided emotion intensity controllable speech synthesis controller described in Embodiment 1, and realizes speech synthesis with continuously adjustable emotion intensity based on the controller. By deeply integrating emotion intensity control with the speech generation process, this system can generate natural, stable, and emotionally expressive speech in different application scenarios.
[0063] The speech synthesis system mainly includes a text front-end module, an emotion intensity controller (i.e., the controller described in Example 1), an acoustic model, and a vocoder. The emotion intensity controller is the core module, used to explicitly model and regulate pitch and energy features to achieve continuous control of emotion intensity.
[0064] The text front-end module receives input text and performs linguistic analysis to generate basic text features. These features include phoneme sequences, syllable boundaries, and parts of speech. Simultaneously, this module receives externally input target sentiment tags (such as "happy" or "sad") and sentiment intensity control parameters. The text front-end module fuses the basic text features with the sentiment information to generate text features infused with sentiment information, which are then output to the sentiment intensity controller.
[0065] The emotional intensity controller is the core of this system, and its structure and working principle have been described in detail in Example 1. The controller takes text features incorporating emotional information and the target emotional embedding as input. Through internal pitch predictors, energy predictors, prediction modulation modules, pitch modulation modules, and energy modulation modules, it models, predicts, modulates, and constrains the pitch and energy features, ultimately outputting prosodic features (including pitch and energy features) scaled by emotional intensity. The pitch and energy emotional intensity control parameters are dynamically adjusted by the user or an external system to achieve continuous emotional intensity changes from mild to full.
[0066] The acoustic model receives prosodic features output from the emotion intensity controller and text features infused with emotional information, using these as conditional inputs to generate corresponding acoustic feature parameters (such as Mel spectrograms). The acoustic model can employ a neural network-based architecture (such as Tacotron, FastSpeech, etc.), learning the mapping relationship from text and prosodic features to acoustic features through training.
[0067] A vocoder converts the acoustic features generated by an acoustic model into the final speech waveform. Vocoders can employ classical algorithms (such as Griffin-Lim) or neural vocoders based on neural networks (such as WaveNet, HiFi-GAN, etc.) to synthesize high-quality, highly natural speech signals.
[0068] During the process, the user first inputs the text to be synthesized and specifies the desired emotion category (such as "happy") and emotion intensity (via...). and (Parameter adjustment). The text front-end module converts the text into text features incorporating emotional information, and sends them along with the target emotional embedding to the emotional intensity controller. The controller adjusts the parameters according to the settings. and The pitch and energy features are scaled to generate prosodic features with the target emotional intensity. Subsequently, the acoustic model generates acoustic parameters based on these features, and finally, a vocoder synthesizes the speech output.
[0069] This system decouples emotion intensity control from the speech generation process, using an independent controller module to finely adjust prosodic features. This not only avoids the speech quality degradation caused by directly modifying the emotion embedding but also achieves continuous and stable control of emotion intensity. Experiments show that the system can generate natural, fluent, and emotionally expressive speech across various emotion categories and intensity levels, making it widely applicable to scenarios requiring emotional voice interaction, such as intelligent customer service, audiobooks, virtual anchors, and game character voice-overs.
[0070] It should be noted that the acoustic model and vocoder in this embodiment can adopt mature architectures in existing technologies. The focus of this embodiment is on the introduction of the emotion intensity controller and its collaboration with other modules of the system. By integrating the emotion intensity controller as an independent module into the speech synthesis system, continuous and controllable emotion intensity can be achieved without changing the original main structure of the system. This approach has advantages such as simple implementation, strong scalability, and wide applicability.
[0071] Example 4 This embodiment provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the multi-prosodic guided emotion intensity controllable speech synthesis control method of Embodiment 2.
[0072] The method in Example 2 can be applied in software form, such as by designing it as a standalone program and installing it on a computer terminal, which can be a computer, smartphone, control system, or other IoT devices. Alternatively, the method in Example 2 can be designed as an embedded program and installed on a computer terminal, such as on a microcontroller.
[0073] Example 5 This embodiment provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, it implements the steps of the multi-prosodic guided emotion intensity controllable speech synthesis control method of Embodiment 2.
[0074] When applying the method of Example 2, it can be applied in the form of software, such as by designing it as a program that can run independently on a computer-readable storage medium. The computer-readable storage medium can be a USB flash drive, designed as a USB security token, and the program can be designed to start the entire method through an external trigger.
[0075] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A speech synthesis controller with multi-prosodic guided emotion intensity controllable, characterized in that, Its input includes target sentiment embeddings and text features fused with sentiment information, and includes: A pitch predictor is used to model and predict pitch features to generate pitch prediction results for continuously adjusting emotional intensity. An energy predictor is used to model and predict energy characteristics, generating energy prediction results for continuous adjustment of emotion intensity. A prediction modulation module is used to map the target emotion embedding, generate modulation bias terms, and add them to the pitch prediction result and the energy prediction result, respectively. The pitch modulation module is used to fuse the pitch features processed by the prediction modulation module with the text features, and to perform feature alignment and amplitude constraint. An energy modulation module is used to fuse the energy features processed by the prediction modulation module with the text features, and to perform feature alignment and amplitude constraint. Specifically, by adjusting the pitch emotion intensity control parameter and the energy emotion intensity control parameter, the outputs of the pitch modulation module and the energy modulation module are scaled to achieve continuous control of the synthesized speech emotion intensity.
2. The multi-prosodic guided emotion intensity controllable speech synthesis controller as described in claim 1, characterized in that, Both the pitch predictor and the energy predictor employ a conditional variational autoencoder structure, and the loss function of the conditional variational autoencoder is: ； In the formula, To predict target features; As a conditional variable, it is composed of target sentiment embeddings; As latent variables; The posterior distribution of the encoder output. For decoder, It is the prior distribution; Kullback-Leibler divergence is used to measure... and The differences between them.
3. The multi-prosodic guided emotion intensity controllable speech synthesis controller as described in claim 1, characterized in that, The scaling is achieved using the following formula: ； ； In the formula, The pitch emotional intensity control parameter, The energy emotion intensity control parameter, To predict the obtained pitch features, To predict the obtained energy characteristics, This is the scaled pitch characteristic. This represents the scaled energy characteristics.
4. The multi-prosodic guided emotion intensity controllable speech synthesis controller as described in claim 1, characterized in that, The loss functions of the pitch predictor and the energy predictor during the training process include: pitch prediction mean square error loss, energy prediction mean square error loss, KL divergence of the pitch latent variable distribution, and KL divergence of the energy latent variable distribution.
5. The multi-prosodic guided emotion intensity controllable speech synthesis controller as described in claim 1, characterized in that, The predictive modulation module uses a linear transformation layer to embed and map the target emotion into a pitch modulation bias and an energy modulation bias.
6. The multi-prosodic guided emotion intensity controllable speech synthesis controller as described in claim 1, characterized in that, Both the pitch modulation module and the energy modulation module include a linear projection layer and a nonlinear activation function.
7. The multi-prosodic guided emotion intensity controllable speech synthesis controller as described in claim 1, characterized in that, The pitch prediction result is based on the pitch spectrogram, which is obtained by performing continuous wavelet transform on the pitch profile after logarithmic normalization.
8. The multi-prosodic guided emotion intensity controllable speech synthesis controller as described in claim 1, characterized in that, The pitch emotion intensity control parameter and the energy emotion intensity control parameter are adjustable scalars. During the inference phase, the emotion intensity is continuously controlled by scaling the pitch features and energy features obtained from the prediction.
9. The multi-prosodic guided emotion intensity controllable speech synthesis controller as described in claim 1, characterized in that, The pitch and energy features are aligned with the text sequence at the phoneme level using a pre-trained forced alignment tool before training.
10. A method for controlling speech synthesis with controllable emotional intensity guided by multi-prosodic rhythm, characterized in that, It is applied in a multi-prosodic guided emotion intensity controllable speech synthesis controller as described in any one of claims 1-9, wherein the control method includes the following steps: Modeling and predicting pitch and energy features to generate pitch prediction results and energy prediction results for continuously adjusting emotional intensity. The target emotion embedding is mapped to generate a modulation bias term, which is then added to the pitch prediction result and the energy prediction result, respectively. The modulated pitch and energy features are fused with the text features, and feature alignment and amplitude constraints are performed. By adjusting the pitch emotion intensity control parameter and the energy emotion intensity control parameter respectively, the outputs of the pitch modulation module and the energy modulation module are scaled to achieve continuous control of the synthesized speech emotion intensity.