Emotional voice conversion method and device based on rhythm prediction, equipment and medium

Abstract

The invention relates to the technical field of artificial intelligence and the field of financial science and technology, and discloses an emotional speech conversion method based on rhythm prediction, which comprises the following steps: extracting phoneme sequences, acoustic features and emotional dimension representations in input text data and source audio data, and extracting language information by using a text encoder; frame-level rhythm changes and textual rhythm representations are generated based on the language information and the sentiment dimension representations. The method comprises the following steps: generating acoustic rhythm representation in combination with speaker characteristics and emotional dimension representation, calculating alignment loss by aligning text rhythm representation and acoustic rhythm representation, jointly training a rhythm prediction module and an acoustic modeling module to generate comprehensive rhythm representation, and finally generating converted voice waveforms and evaluating the converted voice waveforms through a discriminator. And optimizing each module parameter based on the comprehensive loss, and generating a target emotional voice. According to the method, accurate control of rhythm fine granularity is realized through implicit modeling, and the problem of mismatching during training and operation is avoided, so that the naturalness and emotion richness of voice are greatly improved.

Description

Technical Field

[0001] The present invention relates to the fields of artificial intelligence technology and financial technology, and in particular to an emotional speech conversion method, device, equipment and storage medium based on rhythm prediction. Background Art

[0002] Existing emotional voice conversion (EVC) technology faces significant challenges in application, mainly in terms of content naturalness and emotional richness. First, low content naturalness is one of the core problems of existing technologies. The converted speech lacks fluency and naturalness in listening, especially in the process of emotional voice conversion, often resulting in stiff sound, unnatural pauses, etc., which cannot simulate the natural pronunciation of real-person voices, seriously affecting the user experience.

[0003] Secondly, the existing EVC technology also faces the dilemma of insufficient emotional expression. The emotional expression of the converted voice is often too simple and difficult to convey complex and delicate emotions, resulting in a significant gap compared with the emotional expression of real voice. This deficiency limits the application of emotional voice conversion in actual scenarios, especially in occasions that need to convey rich emotions (such as voice assistants, virtual customer service, etc.).

[0004] In addition, the architecture of existing EVC models usually adopts a cascade approach, that is, the acoustic converter and the vocoder work separately. This approach has inconsistencies in prosody modeling during training and actual operation, resulting in the inability of the converted speech to accurately present emotional prosody changes during operation. This disconnect between training and actual application further reduces audio quality and affects the naturalness of the content and the perceived emotional experience.

[0005] At the same time, existing models lack the ability to model prosodic changes in a fine-grained manner. Prosodic information is an important component of emotional expression, but existing technologies are unable to fully capture and present these changes during speech conversion, resulting in inaccurate emotional conversion and a lack of emotional hierarchy.

[0006] In summary, the existing emotional speech conversion technology has major deficiencies in terms of content naturalness, emotional richness, and rhythm modeling. These problems seriously affect the actual application effect of emotional speech conversion technology. Summary of the invention

[0007] The main purpose of the present invention is to provide an emotional speech conversion method, device, equipment and storage medium based on prosody prediction, aiming to solve the technical problem that the existing emotional speech conversion technology cannot achieve high-precision prosody modeling during the processing process, resulting in the generated speech having obvious deficiencies in naturalness and emotional expression.

[0008] To achieve the above object, the present invention provides an emotional speech conversion method based on prosody prediction, comprising:

[0009] Process the input text data and source audio data to extract phoneme sequences, acoustic features, and emotional dimension representations;

[0010] extracting language information from the phoneme sequence using a text encoder;

[0011] Using a prosody prediction module to predict frame-level prosody changes and duration of emotional speech based on the language information and the emotional dimension representation, and generating a text prosody representation based on the frame-level prosody changes;

[0012] Extracting speaker features from the source audio data using a speaker encoder, and generating an acoustic prosody representation corresponding to the acoustic features based on the speaker features and the emotional dimension representation using an emotion descriptor;

[0013] Determine an alignment loss between the text prosody representation and the acoustic prosody representation, and jointly train the prosody prediction module and the acoustic modeling module based on the alignment loss, wherein the acoustic modeling module includes the speaker encoder and the emotion descriptor;

[0014] generating a comprehensive prosodic representation based on the acoustic prosodic representation, the text prosodic information and the emotional dimension representation using a prosodic integrator;

[0015] Based on the comprehensive prosodic representation and the duration prediction results of the emotional speech, using the emotional speech synthesis module to generate a converted speech waveform, using the discriminator to evaluate the speech waveform and determine the comprehensive loss;

[0016] Based on the comprehensive loss, the model parameters of the prosody prediction module, the acoustic modeling module and the emotional speech synthesis module are optimized by back propagation, and the target emotional speech is generated using the optimized modules.

[0017] Furthermore, to achieve the above object, the present invention provides an emotional speech conversion device based on prosody prediction, comprising:

[0018] A data processing module is used to process input text data and source audio data, extract phoneme sequences, acoustic features, and emotional dimension representations;

[0019] A text encoder module, for extracting language information from the phoneme sequence using a text encoder;

[0020] A prosody prediction module, configured to predict frame-level prosody changes and duration of emotional speech based on the language information and the emotional dimension representation using the prosody prediction module, and generate a text prosody representation based on the frame-level prosody changes;

[0021] An emotion descriptor module, configured to extract speaker features from the source audio data using a speaker encoder, and generate an acoustic prosody representation corresponding to the acoustic features based on the speaker features and the emotion dimension representation using an emotion descriptor;

[0022] A joint training module, used for determining an alignment loss between the text prosody representation and the acoustic prosody representation, and jointly training the prosody prediction module and the acoustic modeling module based on the alignment loss, wherein the acoustic modeling module includes the speaker encoder and the emotion descriptor;

[0023] A prosody integration module, for generating a comprehensive prosody representation based on the acoustic prosody representation, text prosody information and emotional dimension representation using a prosody integrator;

[0024] An emotional speech synthesis module, for generating a converted speech waveform using the emotional speech synthesis module based on the comprehensive prosody representation and the duration prediction result of the emotional speech, and evaluating the speech waveform using a discriminator to determine a comprehensive loss;

[0025] The model optimization and emotional speech generation module is used to optimize the model parameters of the prosody prediction module, the acoustic modeling module and the emotional speech synthesis module through back propagation based on the comprehensive loss, and use the optimized modules to generate target emotional speech.

[0026] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a computer device, which includes a memory, a processor, and an emotional speech conversion program based on prosody prediction stored in the memory and executable on the processor, wherein the emotional speech conversion program based on prosody prediction, when executed by the processor, implements the steps of the emotional speech conversion method based on prosody prediction as described above.

[0027] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, on which is stored an emotional speech conversion program based on prosody prediction, and when the emotional speech conversion program based on prosody prediction is executed by a processor, the steps of the emotional speech conversion method based on prosody prediction as described above are implemented.

[0028] Beneficial effects: The present invention relates to the fields of artificial intelligence technology and financial technology, and discloses an emotional speech conversion method based on rhythm prediction. By extracting the phoneme sequence, acoustic features and emotional dimension representation from the input text data and source audio data, a text encoder is used to extract language information, and frame-level rhythm changes and text rhythm representation are generated based on the language information and emotional dimension representation. The acoustic rhythm representation is generated by combining the speaker characteristics and the emotional dimension representation, and the alignment loss is calculated by aligning the text rhythm representation and the acoustic rhythm representation. The rhythm prediction module and the acoustic modeling module are jointly trained to generate a comprehensive rhythm representation. Finally, the converted speech waveform is generated and evaluated by the discriminator. The parameters of each module are optimized based on the comprehensive loss to generate the target emotional speech. The present invention achieves precise control of the fine-grained rhythm through implicit modeling, avoids the problem of mismatch between training and runtime, thereby greatly improving the naturalness and emotional richness of the speech, and improving the listening experience and application experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The present invention will be further described below with reference to the accompanying drawings and embodiments, in which:

[0030] Figure 1 A schematic diagram of an application environment of an emotional speech conversion method based on prosody prediction in one embodiment of the present invention;

[0031] Figure 2 It is a flow chart of an embodiment of an emotional speech conversion method based on prosody prediction of the present invention;

[0032] Figure 3 It is a functional module diagram of a preferred embodiment of the emotional speech conversion device based on prosody prediction of the present invention;

[0033] Figure 4 A schematic diagram of the structure of a computer device in one embodiment of the present invention;

[0034] Figure 5 FIG. 4 is another schematic diagram of the structure of a computer device in one embodiment of the present invention. DETAILED DESCRIPTION

[0035] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0036] The emotional speech conversion method based on prosody prediction provided by the embodiment of the present invention can be applied in the following aspects: Figure 1In an application environment, the user end communicates with the server end through a network. The server end can extract the phoneme sequence, acoustic features and emotional dimension representation in the input text data and the source audio data through the user end, extract the language information using the text encoder, and generate frame-level rhythm changes and text rhythm representation based on the language information and emotional dimension representation. The acoustic rhythm representation is generated by combining the speaker characteristics and the emotional dimension representation, and the alignment loss is calculated by aligning the text rhythm representation and the acoustic rhythm representation. The rhythm prediction module and the acoustic modeling module are jointly trained to generate a comprehensive rhythm representation, and finally the converted speech waveform is generated and evaluated by the discriminator, and the parameters of each module are optimized based on the comprehensive loss to generate the target emotional speech. The present invention realizes precise control of the rhythm fine-grainedness through implicit modeling, avoids the problem of mismatch between training and runtime, thereby greatly improving the naturalness and emotional richness of the speech, and improving the listening experience and application experience. Among them, the user end can be but not limited to various personal computers, laptops, smart phones, tablet computers and portable wearable devices. The server end can be implemented by an independent server or a server cluster composed of multiple servers. The present invention is described in detail below through specific embodiments.

[0037] See also Figure 2 , Figure 2 This is a flow chart of an embodiment of the method for emotional speech conversion based on prosody prediction provided by the present invention. It should be noted that although the logical sequence is shown in the flow chart, in some cases, the steps shown or described may be performed in a different order than that here.

[0038] like Figure 2 As shown, the emotional speech conversion method based on prosody prediction proposed by the present invention comprises the following steps:

[0039] S10, processing the input text data and source audio data, extracting phoneme sequences, acoustic features, and emotional dimension representations;

[0040] In this embodiment, the input text data is preliminarily cleaned, such as removing redundant symbols, punctuation marks, and spaces. Subsequently, the text is segmented or cut to convert it into a structured form suitable for subsequent processing. The text data is processed using natural language processing tools (such as spaCy, NLTK, etc.) to ensure that the cleaned text can be further used for speech conversion.

[0041] Convert the preprocessed text into a phoneme sequence. Phonemes are the smallest pronunciation units of a language and can accurately represent the pronunciation characteristics of a language. This process requires decomposing each word into several phonemes so that the speech can be better restored during speech generation. Text can be converted into a phoneme sequence based on a pronunciation dictionary (such as the International Phonetic Alphabet IPA or a language-specific pronunciation dictionary). Modern speech conversion models can also generate phonemes directly from text through deep learning language models.

[0042] Extract acoustic features such as fundamental frequency (F0), spectrum, duration, etc. from the source audio. These features are used to represent the physical structure and rhythmic changes of the audio. Use techniques such as short-time Fourier transform (STFT) and Mel frequency cepstral coefficients (MFCC) to extract the spectrum and other acoustic features of the audio. Software tools such as Librosa can easily extract these features.

[0043] Analyze the emotional information in the audio and extract emotional dimension representations, such as arousal, valence, etc. These emotional dimensions are used to guide emotional speech conversion. By training an emotion recognition model, usually based on a deep neural network (such as a convolutional neural network or a recurrent neural network), the acoustic features in the audio are analyzed and the corresponding emotional dimension representations are generated.

[0044] In one specific implementation, a standard word segmentation tool is used to segment financial terms and related texts, and the text data is converted into a phoneme sequence in combination with a traditional phoneme dictionary (such as IPA). The spectral features of the audio are extracted based on the short-time Fourier transform (STFT), and further acoustic features are extracted in combination with the Mel frequency cepstral coefficient (MFCC). A convolutional neural network (CNN) is used to train an emotion recognition model to analyze the emotional state of the customer.

[0045] Example description: Financial companies often use intelligent voice customer service systems to handle a large number of customer requests. In order to improve customer experience, these systems need to adjust the voice response according to the customer's emotional changes to ensure that the voice is natural and emotional. For example, when a customer is nervous or uneasy, the system can use emotional speech conversion technology to convert the voice into a more soothing and gentle tone to respond to the customer's questions. The system converts the text content entered by the customer (such as conversations in online customer service) into a phoneme sequence. Intelligent customer service extracts acoustic features from historical customer voices and extracts the customer's current emotional state, such as whether the customer is excited, calm or dissatisfied. Combined with sentiment analysis in financial transactions, the system can convert the customer's voice data into arousal and valence information through the emotion recognition model, judge the customer's emotional state, and generate corresponding emotional speech.

[0046] By accurately processing text data and source audio data, it is possible to more efficiently extract phoneme sequences in text and acoustic features in audio, and improve the naturalness of speech conversion and the richness of emotional expression through emotional dimension representation. In financial applications.

[0047] S20, extracting language information from the phoneme sequence using a text encoder;

[0048] In this embodiment, a phoneme sequence is generated through a text preprocessing step, which represents the smallest speech unit of each word in the text. The phoneme sequence is input into a text encoder, which is responsible for capturing the grammatical, semantic, and speech features in the text. The text is converted into a phoneme sequence through a standard phoneme dictionary (such as the International Phonetic Alphabet IPA) and input into the encoder. Modern speech processing systems can generate phoneme sequences directly from text through deep learning networks (such as RNN, LSTM, or Transformer).

[0049] The core function of a text encoder is to convert the input phoneme sequence into a higher-dimensional vector representation. This representation contains grammatical information (such as syntactic structure), semantic information (such as contextual relations), and retains the phoneme information related to speech. The text encoder can adopt a structure based on a recurrent neural network (RNN) or a long short-term memory network (LSTM) to capture the temporal dependencies and contextual relations in the phoneme sequence. Modern encoders (such as Transformer) can further improve the ability to capture global semantics through the self-attention mechanism.

[0050] The vector representation generated by the encoder extracts the language information of the text, including semantics, grammar, and pronunciation order of phonemes. This information is crucial for subsequent prosody modeling and can help the system generate natural and fluent speech. The Transformer-based encoder can capture the language information in the entire text sequence without relying on order. At the same time, through the multi-dimensional feature extraction of multi-layer neural networks, the complex semantic and syntactic structures in the text can be obtained.

[0051] Example description: When a customer asks over the phone, "How much is the remaining outstanding balance of my loan?", the system generates a response text based on the customer's question, such as "Your loan still has 10,000 yuan outstanding." The text encoder then converts the generated text into a phoneme sequence and extracts language information from it to generate speech output. The system uses the text encoder to process the generated phoneme sequence and extract the language information in the text. This includes extracting syntactic structure, semantic information, etc. to ensure that the generated speech can naturally convey the information that the customer cares about. A phoneme sequence is generated from the text (such as "loan balance outstanding" is converted into phonemes), and the system processes it according to the semantics.

[0052] By using a text encoder to extract language information from a phoneme sequence, the system can better understand the semantics and grammar of the text during speech conversion, thereby generating more natural and fluent speech expressions. Advanced models such as Transformer can capture complex language patterns in a short period of time, further improving the quality and naturalness of generated speech.

[0053] S30, using a prosody prediction module to predict frame-level prosody changes and duration of emotional speech based on the language information and the emotional dimension representation, and generating a text prosody representation based on the frame-level prosody changes;

[0054] In this embodiment, the role of the prosody prediction module is to predict the prosody changes of speech in each time frame based on language information (grammar, semantics, etc. from the text encoder) and emotional dimension representation (such as arousal and valence). Prosody features include pitch, rhythm, stress, pauses, and duration. These features are crucial to generating natural and emotional speech.

[0055] Extract language information from the text encoder, such as the syntactic structure, semantic association, context, etc. of words. The emotional dimension representation is the emotional features (such as arousal and valence) generated by the emotional model, which reflects the intensity and type of emotion that should be in the speech. Prosodic changes refer to the changes in the pitch, duration, rhythm and other characteristics of the speech within each time frame. Frame-level prediction can capture the prosodic changes in a small time period, making speech generation more refined. Use a recurrent neural network (RNN) or a multi-layer perceptron (MLP) to process the language information and emotional dimensions frame by frame, and output the prosodic features of each frame.

[0056] The pronunciation duration of each syllable or speech segment determines the speed and rhythm of speech, which is particularly important when expressing emotions. Duration prediction can determine the effects of pauses, prolonged sounds, and fast rhythms in sentences. Time series models (such as LSTM) or convolutional networks are used to analyze the pronunciation duration of syllables in speech to ensure that the output speech conforms to the speed and rhythm of natural language.

[0057] Integrate frame-level prosodic changes (including pitch, duration, etc.) into a prosodic representation of the entire text. This representation is one of the inputs for subsequent speech generation, ensuring natural expression and emotional delivery of speech. Use a multi-layer perceptron (MLP) or other nonlinear models to combine the prosodic features of each frame to generate a text-level prosodic representation for subsequent speech synthesis modules.

[0058] In one specific implementation, a Transformer model is used to process the frame-level prosodic features of the entire text in parallel through a self-attention mechanism. Transformer can capture global dependencies in language information and is particularly suitable for processing long texts and complex sentences. Nonlinear prosodic prediction is achieved by using a multi-layer perceptron (MLP), especially in scenarios with strong emotional expressions, such as anger, excitement, etc., to generate rich prosodic changes. Combining the attention mechanism with duration modeling, natural speech duration and speech speed changes are generated to ensure that the speech output meets the emotional expression expected by the user.

[0059] Example description: When a customer asks "How is the progress of my loan approval?", the system can generate a reply based on the loan information and adjust the rhythm and tone of the voice through the rhythm prediction module. For example, the system can use a soft and peaceful tone to calm the customer's anxiety, predict the pitch and duration changes of each frame, and make the voice sound more stable and natural. The system extracts grammatical and semantic information from the generated text. The system predicts the rhythm changes of each frame, such as pitch, rhythm, and duration, through the rhythm prediction module to generate a natural speech rhythm. According to the customer's emotional state (anxiety, confusion, etc.), the system incorporates the emotional dimension representation into the generated speech.

[0060] By using the prosody prediction module, the system can generate speech output with natural emotions and smooth intonation. Frame-level prosody change prediction improves the expressiveness of speech details and makes the speech more authentic and credible. Combined with duration prediction, the system can not only control speech speed and pauses, but also express a variety of intonation and rhythm changes through the emotional dimension, improving the quality of speech conversion, especially the performance effect in emotional speech conversion.

[0061] S40, extracting speaker features from the source audio data using a speaker encoder, and generating an acoustic prosody representation corresponding to the acoustic features based on the speaker features and the emotional dimension representation using an emotion descriptor;

[0062] In this embodiment, the task of the speaker encoder is to extract speaker features from the source audio data. These features include the speaker's voice characteristics, such as timbre, pitch, range, etc. Speaker features help ensure that the generated speech retains the speaker's identity and that the synthesized speech is consistent with the sound characteristics of the original audio. These features reflect the unique sound elements in the speech, such as pitch, voice quality, speaking speed, etc. The source audio data is processed by a deep neural network (DNN) or a convolutional neural network (CNN) to extract the speaker's audio features. These features will be used in the subsequent speech synthesis process to ensure that the generated speech matches the speaker's original voice.

[0063] The task of the emotion descriptor is to generate an acoustic prosodic representation by combining speaker features and emotion dimension representation. The emotion dimension representation reflects the emotional state in the speech (such as arousal, valence, etc.), while the speaker features ensure that the speech retains the uniqueness of the speaker. The resulting acoustic prosodic representation contains features such as pitch, rhythm, duration, etc., and matches the emotion and speaker features.

[0064] Emotional features are usually determined based on an emotion model (e.g., generated by an emotion recognition system). Emotional dimension representation may include emotional features such as anger, happiness, and calmness in speech. The audio representation that combines speaker characteristics with emotional features is used for subsequent speech generation to ensure that the speech can express emotions while maintaining the uniqueness of the speaker.

[0065] Through the multimodal fusion model, the speaker characteristics and emotional dimension representation are fused into a unified representation to generate the final acoustic prosody representation. The fused prosody features are processed with a nonlinear activation function (such as ReLU) to ensure that the generated speech is rich in emotion and delicate in speech changes.

[0066] Example description: When the bank customer service system receives and generates the customer's answer voice, the system uses the speaker encoder to extract the customer service's voice features, and at the same time generates an acoustic prosodic representation with emotional expression through the emotion descriptor based on the customer's emotional state (such as anxiety or confusion). These acoustic prosodic features ensure that the speech output can convey emotions while maintaining the uniqueness of the customer service voice. The system extracts speaker features (such as pitch, speaking speed, etc.) from the customer service's voice to ensure that the customer service's voice characteristics are maintained when the voice is generated. The system combines the customer's emotional state to generate a prosodic representation with appropriate emotional expression to ensure that the generated speech is consistent with the emotional context of the customer interaction.

[0067] By using the speaker encoder to extract speaker features and the emotion descriptor to generate prosodic representation, the system can generate speech that is consistent with the original speaker's voice quality and has emotional expression. This ensures that the generated speech can not only express delicate emotions, but also retain the speaker's unique timbre and voice characteristics, improving the naturalness of speech synthesis and the ability to convey emotions.

[0068] S50, determining an alignment loss between the text prosody representation and the acoustic prosody representation, and jointly training the prosody prediction module and the acoustic modeling module based on the alignment loss, wherein the acoustic modeling module includes the speaker encoder and the emotion descriptor;

[0069] In this embodiment, the text prosodic representation is generated based on language information and emotional dimensions, reflecting the changes in the rhythm, intonation, and duration of the text content during speech generation. The acoustic prosodic representation combines speaker characteristics and emotional characteristics, and is a manifestation of the prosodic features in actual speech. The alignment loss is calculated to ensure that the text prosodic representation and the acoustic prosodic representation maintain consistency and fluency when generating speech, avoiding unnatural intonation or rhythm misalignment in the generated speech.

[0070] The text prosodic representation is generated by the text encoder and the prosodic prediction module, reflecting the language prosodic features of the text content. The acoustic prosodic representation is generated by the speaker encoder and the emotion descriptor, reflecting the prosodic features of the actual speech, including pitch, rhythm, speaking speed, etc.

[0071] Kullback-Leibler divergence (KL divergence) is often used to measure the difference between two probability distributions. In this step, the difference between the text prosody representation and the acoustic prosody representation can be calculated using KL divergence as the alignment loss. Through the alignment loss function, the system can evaluate whether the generated text prosody and acoustic prosody are consistent. The goal is to minimize the alignment loss to ensure that the generated speech is consistent with the expected input text prosody.

[0072] Through the feedback of alignment loss, the system will jointly train the prosody prediction module (responsible for predicting the prosody features of the text) and the acoustic modeling module (including the speaker encoder and the emotion descriptor). The goal of joint training is to optimize these two modules so that the generated speech achieves consistency between the text prosody and the acoustic prosody, thereby improving the naturalness of the speech and the accuracy of emotional expression. The generated text prosody features are optimized through joint training to make them closer to the acoustic prosody. By jointly optimizing the speaker encoder and the emotion descriptor, it is ensured that the generated speech not only conforms to the speaker's voice characteristics, but also expresses appropriate emotions.

[0073] The two modules (rhythm prediction module and acoustic modeling module) are integrated into the same training process. The training process adjusts the weight parameters of each module through the feedback of alignment loss. Through the gradient descent method and back propagation algorithm, the system optimizes the neural network parameters of the two modules, reduces the alignment loss, and thus improves the performance of the overall system.

[0074] Example description: When a banking system generates a voice response for a customer, the system needs to ensure that the text prosody (description of the loan status) is consistent with the acoustic prosody (intonation and rhythm in the speech). By calculating the alignment loss between the text prosody and the acoustic prosody, the banking system can ensure that the generated speech conveys accurate information while also having a natural voice performance. The system ensures that the generated speech matches the bank's professional tone in terms of intonation and rhythm by calculating the intonation and speed of the text describing the loan information. Based on the customer's emotional state (such as anxiety or eagerness), acoustic prosody features with appropriate emotional expressions are generated, so that the generated speech not only has a natural and fluent intonation, but also conveys appropriate emotional expressions.

[0075] Through the feedback of alignment loss, the system can ensure that the language information of the text is consistent with the actual generated speech rhythm in terms of pitch and rhythm, improving the naturalness of the generated speech and the accuracy of emotional expression. Joint training of the rhythm prediction module and the acoustic modeling module enables the system to optimize the feature expression of both text and audio, reduce the unnaturalness in the speech output, and ensure the integrity of the speaker's characteristics and emotional expression.

[0076] S60, using a prosody integrator to generate a comprehensive prosody representation based on the acoustic prosody representation, text prosody information, and emotional dimension representation;

[0077] In this embodiment, the task of the prosody integrator is to integrate the information from different modules, namely, the acoustic prosody representation, the text prosody information and the emotional dimension representation, to generate a comprehensive prosody representation. This comprehensive prosody representation contains all the necessary prosody features in the speech and is used for subsequent speech synthesis to ensure that the generated speech is natural, fluent and accurately conveys the desired emotion.

[0078] The acoustic prosody representation is extracted from the speaker encoder and the emotion descriptor, and represents the speaker's prosodic features such as pitch, rhythm, and speaking rate. The text prosodic information is generated from the prosody prediction module, reflecting the characteristics of the text content such as intonation, rhythm, and pauses. The emotion dimension represents the emotions conveyed in the speech (such as anger, happiness, etc.).

[0079] The prosodic integrator fuses the prosodic information from the acoustic, text, and sentiment modules in a weighted manner. Different weight assignments depend on the needs in the scenario, for example, in speech with stronger emotional expressions, the weight of the emotional dimension may be higher. The integrated information is processed by nonlinear activation functions (such as ReLU, Tanh) to generate a natural comprehensive prosodic representation. This helps to maintain the natural changes in emotion and prosody in speech.

[0080] The prosodic integrator receives information from three different modules and integrates them together to generate a comprehensive prosodic representation. This representation preserves the speaker's voice characteristics and emotional expression capabilities while meeting the grammatical and semantic requirements of the text. The acoustic prosodic representation is based on the acoustic features of the source audio to ensure that the speaker's voice quality and pitch are consistent. The textual prosodic information is based on the language information of the text encoder to ensure that the speech output is consistent with the grammar and rhythm of the text. The emotional dimension representation ensures that the emotional expression of the speech is as expected, such as increasing the speaking rate in tense scenes and lowering the pitch in calm scenes.

[0081] The prosody integrator combines the acoustic prosody, text prosody, and emotional dimension information through weighted summation. The weight of each module can be dynamically adjusted according to the emotional intensity of the speech. In order to ensure the balance between the prosodic information, the integrator normalizes the input data to ensure that the final generated comprehensive prosodic representation is stable.

[0082] Example description: A customer asks over the phone, "How is my loan approval?" The system needs to generate a voice response with appropriate emotion and stable tone. The prosody integrator receives acoustic prosody, text prosody, and emotional information, and fuses this information to ensure that the generated voice is consistent with the professional tone of financial services in terms of intonation and rhythm. The timbre and rhythm of the voice customer service are extracted from the system to ensure that the generated voice matches the actual customer service voice. Based on the loan approval information, the rhythm and pause information in the text is generated to ensure that the language structure of the voice is correct. Combined with the emotion descriptor, a suitable tone is generated to ensure that the voice expresses calm and stable emotions.

[0083] By using a prosody integrator, the system can generate a comprehensive prosody representation to ensure the naturalness of speech and the accuracy of emotional expression. When integrating multiple prosodic information, the prosody integrator can ensure the coordination of the speaker's voice quality, the language structure of the text, and the emotional expression. The final generated speech is not only consistent with the text and audio in terms of intonation and rhythm, but also conveys the expected emotion, significantly improving the quality of speech generation.

[0084] S70, based on the comprehensive prosody representation and the duration prediction result of the emotional speech, using the emotional speech synthesis module to generate a converted speech waveform, using a discriminator to evaluate the speech waveform, and determining a comprehensive loss;

[0085] In this embodiment, the comprehensive prosodic representation is the result of integrating the acoustic prosodic representation, text prosodic information and emotional dimension representation, representing all the features that need to be expressed in speech, such as pitch, rhythm, pauses and emotional expression. The duration prediction result determines the pronunciation duration of each syllable, word or phrase in the speech, thereby controlling the speech rate and pauses. Based on these inputs, the emotional speech synthesis module is responsible for generating the final speech waveform.

[0086] The comprehensive prosody representation integrates the pitch, rhythm and emotional information of speech and is the basic input for speech generation. The duration prediction result determines the pronunciation duration of each speech segment to ensure smooth speech output.

[0087] The emotional speech synthesis module uses speech synthesis models such as generative adversarial networks (GAN) or WaveNet to generate speech waveforms based on comprehensive prosody representation and duration prediction results. During training, the model learns the impact of different emotions and prosody on speech. The synthesis module not only generates the basic waveform of the speech, but also adjusts the fundamental frequency (pitch) and spectral features of the speech to match the prosody and duration information.

[0088] The task of the discriminator is to evaluate the difference between the generated speech waveform and the real speech, and to determine whether the speech is natural and fluent and whether it meets the expected emotional expression. The discriminator is usually a module in the Generative Adversarial Network (GAN), which is used to distinguish the generated speech waveform from the actual target speech. By comparing the generated speech with the target real speech, it is evaluated whether the generated speech has achieved the expected effect in terms of emotion, rhythm and naturalness. The discriminator in the Generative Adversarial Network (GAN) learns how to distinguish between real speech and generated speech. Through multiple rounds of training, the generated speech waveform will gradually approach the characteristics of real speech, thereby improving the naturalness of speech synthesis. The discriminator evaluates the speech waveform based on multiple indicators, such as the naturalness of the speech, the accuracy of emotional expression, and whether the changes in pitch are smooth.

[0089] During speech generation, the system calculates a comprehensive loss to evaluate the difference between the generated speech and the target speech. The comprehensive loss usually consists of multiple parts, such as reconstruction loss, adversarial loss, and sentiment classification loss. By optimizing these losses, the system can continuously improve the quality of generated speech.

[0090] The reconstruction loss measures the difference between the generated speech waveform and the real speech waveform to ensure that the speech waveform meets the target. The adversarial loss is the loss when the discriminator judges whether the generated speech is close to the real speech, which is used to optimize the adversarial nature between the generator and the discriminator. The sentiment classification loss ensures that the generated speech is consistent with the expected emotions (such as anger, happiness, etc.) in terms of emotional expression. The system optimizes the comprehensive loss through the back-propagation algorithm, and continuously updates the weights of the generator (emotional speech synthesis module) so that it can generate more natural and emotionally satisfactory speech. The reconstruction loss, adversarial loss, and sentiment classification loss are weighted and combined to form a comprehensive loss, and the model is optimized based on this loss.

[0091] By using the emotional speech synthesis module, the system can generate natural emotional speech and further optimize the speech quality in combination with the discriminator evaluation results. The calculation and optimization of the comprehensive loss helps the system generate more fluent and natural speech, and can accurately express emotions. The final speech output can be applied to various scenarios that require emotional interaction, such as intelligent customer service, voice assistants, etc.

[0092] S80, optimizing the model parameters of the prosody prediction module, the acoustic modeling module and the emotional speech synthesis module through back propagation based on the comprehensive loss, and using the optimized modules to generate target emotional speech.

[0093] In this embodiment, the comprehensive loss is composed of multiple loss functions, including reconstruction loss, adversarial loss, sentiment classification loss, etc. These losses reflect the difference between the generated speech and the target real speech. By calculating the comprehensive loss, the system can determine the quality of the generated speech and use the back propagation algorithm to optimize the model parameters of the prosody prediction module, acoustic modeling module, and emotional speech synthesis module.

[0094] The comprehensive loss is a weighted sum of multiple losses, including: reconstruction loss, which reflects the difference between the generated speech and the target speech; adversarial loss, which is provided by the discriminator in the generative adversarial network (GAN) and reflects the authenticity of the generated speech; and sentiment classification loss, which ensures that the generated speech meets the expected emotional expression.

[0095] By calculating the gradient of each layer of the model and propagating the gradient to the previous layer, the weight parameters of the model are gradually updated to reduce the comprehensive loss. By minimizing the loss function, the model will gradually generate a speech waveform that is more in line with the expected one.

[0096] The optimization of the comprehensive loss is not just for one module, but for the three modules of prosody prediction module, acoustic modeling module and emotional speech synthesis module. The back propagation algorithm gradually adjusts the parameters in these modules so that they can jointly generate natural and emotional speech output. The prosody prediction module is responsible for predicting the prosodic features of the text, such as pitch, rhythm, and speaking speed. The acoustic modeling module generates prosodic features related to the acoustic features to ensure that the speaker's characteristics and emotional expression are consistent. The emotional speech synthesis module integrates all the information to generate the final speech waveform.

[0097] These three modules are jointly trained in the same training process. The parameters of each module are updated according to the gradient in the back-propagation algorithm, making the whole system more coordinated and efficient. Through multi-task learning, the model learns multiple objectives (such as rhythm, acoustic features, emotional expression) at the same time, and improves the overall performance through joint optimization.

[0098] The optimized model can generate more natural speech that meets the requirements of emotional expression. The optimized prosody prediction module, acoustic modeling module and emotional speech synthesis module can generate accurate pitch, rhythm, and speech speed, and the speech waveform has emotional expression. The target emotional speech means that the generated speech can not only reflect the text content, but also convey a specific emotional state (such as pleasure, anger, calmness, etc.).

[0099] When generating speech, the optimized emotional speech synthesis module uses the trained prosodic and acoustic features to generate the final speech waveform, which accurately reflects the semantics and emotion of the input text.

[0100] Through back propagation optimization based on comprehensive loss, the system can generate more natural and fluent emotional speech. After optimization, the rhythm prediction module, acoustic modeling module and emotional speech synthesis module can work better in coordination, and the generated speech waveform can achieve higher quality in rhythm, speaking speed and emotional expression. The final generated speech can not only accurately convey the text content, but also effectively express emotions and improve user experience.

[0101] The present invention relates to the fields of artificial intelligence technology and financial technology, and discloses an emotional speech conversion method based on rhythm prediction. By extracting phoneme sequences, acoustic features and emotional dimension representations from input text data and source audio data, a text encoder is used to extract language information, and frame-level rhythm changes and text rhythm representations are generated based on the language information and emotional dimension representations. Acoustic rhythm representations are generated by combining speaker characteristics and emotional dimension representations, and the alignment loss is calculated by aligning the text rhythm representation and the acoustic rhythm representation. The rhythm prediction module and the acoustic modeling module are jointly trained to generate a comprehensive rhythm representation. Finally, a converted speech waveform is generated and evaluated by a discriminator, and the parameters of each module are optimized based on the comprehensive loss to generate the target emotional speech. The present invention achieves precise control of rhythm fine-grainedness through implicit modeling, avoids the problem of mismatch between training and runtime, thereby greatly improving the naturalness and emotional richness of speech, and improving the listening experience and application experience.

[0102] In one embodiment, the above S10 includes:

[0103] S101, performing word segmentation processing on text data, and converting the text data after word segmentation into a phoneme sequence;

[0104] S102, extracting a linear spectrogram from the source audio data using a short-time Fourier transform;

[0105] S103, extracting fundamental frequency and duration information from the source audio data, and using the linear spectrogram, fundamental frequency and duration information as acoustic features;

[0106] S104, inputting the acoustic features into an emotion recognition model, and generating an emotion dimension representation by analyzing the acoustic features, wherein the emotion dimension representation includes the arousal degree and valence of the emotion, which are used to represent the activation degree of the emotion and the positive and negative attributes of the emotion.

[0107] In this embodiment, text segmentation is the process of decomposing input text data into multiple words or phrases. The task of converting the segmented text data into a phoneme sequence is to extract the phonemes corresponding to each word for subsequent use in speech generation. The phoneme sequence is the basis of speech generation and represents the pronunciation information of each word in the speech. The text is segmented according to the grammatical structure of the language to obtain each word or phrase. The segmented text is converted into phonemes, which are the smallest pronunciation units in speech.

[0108] You can use a rule-based word segmentation method or a machine learning-based word segmentation model, such as HMM (Hidden Markov Model) or CRF (Conditional Random Field). By looking up a dictionary or a pre-trained speech model, each word in the text is mapped to a corresponding phoneme sequence.

[0109] Short-time Fourier transform (STFT) is a method for converting audio signals into time-frequency domain representation. Through STFT, the audio signal can be decomposed into different frequency components and a spectrogram of the audio can be generated. The linear spectrogram is one of the outputs of STFT, which reflects the frequency changes of the audio signal in different time periods and is used for subsequent acoustic feature extraction. The linear spectrogram represents the distribution of the source audio in the frequency domain and is often used in audio signal processing and speech generation. The source audio data is divided into multiple windows, and Fourier transform is performed on each window to generate a short-time spectrum. The linear spectrogram is generated through the converted spectral information and input into the next model as an acoustic feature.

[0110] Fundamental frequency is the basic frequency of speech, which is usually related to the speaker's tone. Duration information refers to the length of time each syllable or word is pronounced, which is a key parameter when generating natural speech. By extracting fundamental frequency and duration information, the prosodic characteristics of speech can be further captured. Fundamental frequency (F0) reflects the pitch changes in speech and is one of the core features for generating natural speech. Duration information reflects the duration of pronunciation of each syllable, which affects the speed and rhythm of speech. The fundamental frequency can be extracted using the Pitch tracking algorithm based on the autocorrelation method. The duration of each syllable is predicted by Viterbi decoding or HMM and used as the duration feature input.

[0111] The extracted acoustic features are analyzed by the emotion recognition model to generate an emotion dimension representation. The emotion dimension representation usually includes arousal and valence, which represent the degree of emotion activation and the positive and negative attributes of emotion respectively. Arousal describes the intensity of emotion, such as excitement or calmness; valence describes the polarity of emotion, such as happiness or sadness. The emotion recognition model predicts the emotional state of speech by analyzing the input acoustic features (linear spectrogram, fundamental frequency, duration, etc.). The emotion recognition model can use convolutional neural network (CNN) or recurrent neural network (RNN) to classify acoustic features and generate emotion dimension representation. The emotion dimension representation includes the arousal and valence of emotion, which are used to describe the emotional state in speech.

[0112] By processing text data and source audio data, the system of this embodiment can extract detailed phoneme sequences, acoustic features and emotional dimension representations to ensure natural tone and emotional expression when generating speech.

[0113] In one embodiment, in the above S50, determining the alignment loss between the text prosody representation and the acoustic prosody representation, and jointly training the prosody prediction module and the acoustic modeling module based on the alignment loss, comprises:

[0114] S501, using Kullback-Leibler divergence to determine the distribution difference between the text prosody representation and the acoustic prosody representation, and using the distribution difference as an alignment loss for joint training;

[0115] S502, determining the gradients of the prosody prediction module and the acoustic modeling module through back propagation based on the alignment loss;

[0116] S503, using a gradient descent method to update parameters of a prosody prediction module and an acoustic modeling module according to the gradient;

[0117] S504, alternately train the rhythm prediction module and the acoustic modeling module, re-determine the alignment loss after each training, and gradually optimize the module parameters.

[0118] In this embodiment, the text prosodic representation is generated based on language information, reflecting the prosodic features of the text, such as the tone and rhythm; the acoustic prosodic representation combines speaker characteristics and emotional dimension representation, reflecting the prosodic features of the actual speech. By using the Kullback-Leibler divergence (KL divergence), the difference in probability distribution between the two prosodic representations can be calculated. This distribution difference reflects the degree of alignment between the generated text prosody and the acoustic prosody. The KL divergence is used to measure the difference between two probability distributions. The smaller the KL divergence, the better the alignment between the text prosodic representation and the acoustic prosodic representation. KL divergence, as a loss function, reflects the degree of alignment between the text and the acoustic prosody. By comparing the probability distributions of the text prosodic representation and the acoustic prosodic representation, the KL divergence between the two is calculated to generate an alignment loss. The result of the KL divergence is used as the target loss function for subsequent training to adjust the parameters of the model.

[0119] Backpropagation is an algorithm used to optimize neural network models. Based on the alignment loss calculated by KL divergence, the system can calculate the gradients of the rhythm prediction module and the acoustic modeling module through the backpropagation algorithm, and then decide how to update their parameters. The gradient represents the sensitivity of the loss function to the model parameters. Through the gradient descent method, the system can reduce the alignment loss. The gradient characterizes the impact of the change in the loss function on each model parameter. The backpropagation algorithm calculates the gradient of each module layer by layer through the chain derivation rule for adjusting the parameters. Based on the result of the alignment loss function, the backpropagation algorithm is used to determine the gradient of the rhythm prediction module and the acoustic modeling module. The gradient of each layer of the model is calculated to obtain the update direction of the parameters of each layer.

[0120] Through the gradient descent algorithm, the model gradually updates the parameters of the rhythm prediction module and the acoustic modeling module to minimize the alignment loss. Gradient descent is an iterative optimization method that gradually adjusts the direction of the parameters so that the model output gradually approaches the target distribution. Based on the gradient information, the model parameters are updated to reduce the alignment loss. Common gradient descent methods include standard gradient descent, stochastic gradient descent (SGD), Adam, etc. In each iteration, the model uses the gradient descent method to update the parameters of the rhythm prediction module and the acoustic modeling module. Set the learning rate to determine the magnitude of the parameter adjustment to ensure the stability and convergence of the optimization process.

[0121] Alternating training means training the prosody prediction module and the acoustic modeling module separately at different stages. After each training session, the system recalculates the alignment loss of the text prosody representation and the acoustic prosody representation to further optimize the parameters of the two modules. This alternating training method ensures that the two modules can coordinately improve the naturalness and emotional expression of the overall speech generation. The prosody prediction module and the acoustic modeling module are optimized successively, and the alignment loss is re-evaluated after each training session to ensure that the model parameters are continuously optimized. Optimize the parameters of the prosody prediction module in one training phase and the parameters of the acoustic modeling module in another training phase. Ensure the coordinated optimization of the two modules through phased training. After each training session, the system recalculates the alignment loss and adjusts the training strategy.

[0122] By using KL divergence as the alignment loss, the system can effectively measure the difference between text prosody and acoustic prosody. Jointly training the prosody prediction module and the acoustic modeling module can ensure that the generated speech is more natural in prosody and emotion. The alternating training method further improves the stability and convergence of the model and optimizes the quality of the final generated speech.

[0123] In one embodiment, the above S60 includes:

[0124] S601, performing weighted fusion of the acoustic prosody representation, text prosody information and emotional dimension representation through a prosody integrator to generate a preliminary fusion representation;

[0125] S602, using a feature extraction module in the prosody integrator to perform feature extraction on the preliminary fusion representation to generate a fusion feature representation;

[0126] S603: Input the fused feature representation into a nonlinear activation function for processing to generate a final comprehensive rhythm representation.

[0127] In this embodiment, the first step of the prosody integrator is to perform weighted fusion of acoustic prosody representation, text prosody information and emotional dimension representation. This process is to fuse prosody feature information from different sources to ensure that the generated comprehensive prosody representation can simultaneously reflect the characteristics of the speaker, the grammatical structure of the text and the emotional expression. The acoustic prosody representation is based on speaker characteristics and reflects the characteristics of the actual speech, such as tone and rhythm. The text prosody information is a prosody feature generated based on the text content, reflecting the intonation and pauses in the grammar and language structure. The emotional dimension representation represents the intensity and positive and negative attributes of the emotion in the speech. The prosody integrator dynamically adjusts the weights of each feature according to different input sources and fuses them by weighted summation. For example, when the emotion is strong, the weight of the emotional dimension will increase, while in neutral speech, the text prosody information may occupy a higher weight. Depending on the application scenario, the weight can be obtained through training and learning, or it can be manually set in the specific implementation.

[0128] After weighted fusion, the prosody integrator will further extract features from the generated preliminary fusion representation. The purpose of feature extraction is to enhance the expressiveness of the representation and capture detailed changes in the prosody, such as the rise and fall of intonation and the stress of syllables. The feature extraction module can be a model based on a multi-layer neural network, such as a convolutional neural network (CNN) or a multi-layer perceptron (MLP), which extracts higher-level prosodic features by processing the preliminary fusion representation. These features will more finely represent the emotional and prosodic changes in the speech. Feature enhancement techniques, such as batch normalization or activation functions, may be introduced in the feature extraction process to ensure the stability and robustness of the feature expression.

[0129] After feature extraction, the generated fused feature representation is processed through a nonlinear activation function. The purpose of this process is to introduce nonlinear mapping so that the model can learn the complex relationship between prosody and emotion. Nonlinear processing enables the final comprehensive prosodic representation to better reflect the expression of different emotional levels in speech. Nonlinear activation functions such as ReLU, Sigmoid, or Tanh are used to convert linear features into nonlinear features to increase the expressive power of the model. The fused prosodic features are processed nonlinearly through the activation function to generate the final comprehensive prosodic representation. Nonlinear processing can introduce more prosodic changes and emotional details, making the generated speech more natural and emotionally expressive.

[0130] In this embodiment, through weighted fusion, feature extraction and nonlinear processing of the prosody integrator, the system can generate a comprehensive prosody representation to ensure that the generated speech can not only accurately convey the text content but also express rich emotional characteristics.

[0131] In one embodiment, the above S70 includes:

[0132] S701, inputting the comprehensive prosody representation and the duration prediction result of the emotional speech into an emotional speech synthesis module, and generating a speech waveform after emotion conversion through the emotional speech synthesis module;

[0133] S702, using a discriminator to evaluate the speech waveform, evaluate the naturalness, authenticity and accuracy of emotional expression of the speech waveform, and generate a corresponding evaluation result;

[0134] S703, determining the reconstruction loss between the speech waveform and the target real speech;

[0135] S704, determining the authenticity and naturalness loss of the speech waveform based on the evaluation result;

[0136] S705, determining the difference between the emotion expression of the speech waveform and the expected emotion label based on the emotion dimension representation, and determining the emotion classification loss;

[0137] S706, combining the reconstruction loss, adversarial loss and sentiment classification loss to determine a comprehensive loss.

[0138] In this embodiment, the task of the emotional speech synthesis module is to generate a converted speech waveform based on the input comprehensive prosodic representation and the duration prediction result of the emotional speech. The comprehensive prosodic representation provides information such as pitch, rhythm, emotion, etc. in the speech, and the duration prediction result determines the pronunciation duration of each syllable, word or phrase to ensure the natural fluency of the speech. The comprehensive prosodic representation provides rhythmic features such as rhythm, pitch, emotion, etc. in the speech. The duration prediction result determines the pronunciation duration of each syllable in the speech to ensure the rhythm and fluency of the speech. The emotional speech synthesis module generates speech waveforms through models such as generative adversarial networks (GAN) or WaveNet. These models use comprehensive prosodic representation and duration prediction as input to gradually generate realistic speech output. The synthesis module generates speech signals frame by frame and adjusts the length of each syllable according to the duration prediction.

[0139] The task of the discriminator is to evaluate the generated speech waveform to determine whether it meets the standards in terms of naturalness, authenticity, and accuracy of emotional expression. Through the feedback of the discriminator, the system can understand the gap between the generated speech and the target real speech. Naturalness evaluation is used to determine whether the speech is fluent and whether there are intermittent or unnatural tones. Authenticity evaluation is used to evaluate whether the generated speech sounds like real speech. Emotional expression evaluation is used to determine whether the generated speech accurately conveys the expected emotion. In the Generative Adversarial Network (GAN), the discriminator learns how to distinguish between generated speech and real speech. The generator and the discriminator gradually generate more natural speech through repeated adversarial training. The discriminator gives evaluation results based on naturalness, authenticity, and accuracy of emotional expression.

[0140] To optimize the generated speech, the system calculates three key loss values: reconstruction loss, adversarial loss, and sentiment classification loss. Through these losses, the system is able to measure the difference between the generated speech and the target speech and optimize the model through backpropagation. The reconstruction loss measures the difference between the generated speech waveform and the real speech. The adversarial loss is fed back through the GAN discriminator to determine how close the generated speech is to the real speech. The sentiment classification loss ensures that the generated speech emotion is consistent with the target emotion label. The reconstruction loss uses the L2 loss function or cross entropy loss to calculate the difference between the generated speech and the target speech. The adversarial loss calculates the difference between the generator and the real speech through the discriminator in the GAN to optimize the generator. The sentiment classification loss uses predefined emotion labels (such as happy, angry) as a reference, and calculates whether the generated speech accurately conveys the emotion through the classification loss.

[0141] The system performs a weighted combination of reconstruction loss, adversarial loss, and sentiment classification loss to form a comprehensive loss. The comprehensive loss is the goal of the entire model optimization. By minimizing the comprehensive loss, the system can generate more natural and emotionally accurate speech. A weighted combination is performed based on the importance of each loss. For example, for scenes with strong emotional needs, the weight of sentiment classification loss may be higher. Through the back-propagation algorithm, the model updates parameters layer by layer to reduce the comprehensive loss and gradually optimize the quality of the generated speech.

[0142] This embodiment generates speech waveforms based on comprehensive prosody representation and duration prediction results, and combines the evaluation of the discriminator and the optimization of multiple losses. The system can generate natural, realistic and emotional speech output. The naturalness of the generated speech and the accuracy of emotional expression are improved, and it can adapt to a variety of application scenarios, especially those that require high emotional expression, such as intelligent customer service and voice assistants.

[0143] In one embodiment, in the above S40, using an emotion descriptor to generate an acoustic prosody representation corresponding to the acoustic feature based on the speaker feature and the emotion dimension representation includes:

[0144] S401, weighted fusion of the speaker characteristics and the emotion dimension representation by the emotion descriptor to generate preliminary emotion prosody information;

[0145] S402, using a feature extraction module in the emotion descriptor to process the fused prosody information to extract higher-level features of the emotion prosody;

[0146] S403, generating a final acoustic prosody representation based on higher-level prosody features.

[0147] In this embodiment, the first step of the emotion descriptor is to perform weighted fusion of speaker features and emotion dimension representation to generate preliminary emotion prosody information. This process aims to combine the speaker's personal characteristics (such as timbre, pitch, etc.) with emotional expression, so that the generated speech not only conforms to the speaker's characteristics, but also accurately reflects the current emotional state. Speaker features reflect the speaker's personalized timbre and prosody characteristics. Emotional dimension representation represents the arousal (Arousal) and valence (Valence) of emotion, which is used to express the intensity and positive and negative polarity of emotion. The emotion descriptor weights the speaker features and the emotion dimension representation. The weight setting depends on different situations. For example, in some scenes with high emotional expression, the weight of the emotion dimension may be higher, while in professional voice scenes, the weight of the speaker features is greater. Preliminary emotion prosody information can be generated by linear weighted fusion or by learning the optimal fusion weight through a neural network.

[0148] After generating preliminary emotional prosody information, the emotional descriptor inputs this information into the feature extraction module for higher-level feature processing. This step aims to extract more fine-grained features from the preliminary fused emotional prosody information, such as subtle changes in pitch, rhythm, pauses, etc., to enhance the quality of speech generation. The feature extraction module usually uses a deep neural network (such as a convolutional neural network CNN or a multi-layer perceptron MLP) to process the prosody information and extract high-level emotional features. Through a multi-layer network structure, prosody features at different levels are extracted to ensure that speech generation can capture complex emotional changes and delicate speaker characteristics.

[0149] After feature extraction, the system generates the final acoustic prosodic representation. This representation not only contains speaker characteristics, but also conveys emotional information in speech. The acoustic prosodic representation will be used as the input of the subsequent speech synthesis module to ensure that the generated speech is consistent with the target speech in both rhythm and emotion. The acoustic prosodic representation is a high-level prosodic feature that is a fusion of speaker characteristics and emotional dimension representation, reflecting the pitch, rhythm and emotional changes in speech. Based on the extracted high-level features, the acoustic prosodic representation is generated through a neural network. This representation is used to guide the generator in emotional expression and pitch control when generating speech. The acoustic prosodic representation can be used in a variety of emotional speech generation scenarios, such as intelligent customer service, virtual voice assistants, etc., to ensure that the speech not only conforms to the speaker characteristics, but also can accurately express emotions.

[0150] In this embodiment, the system can generate a delicate and natural acoustic prosody representation through the weighted fusion and high-level feature extraction of speaker characteristics and emotional dimension representation by the emotion descriptor. This representation can not only accurately reflect the characteristics of the speaker, but also convey delicate emotional changes, making the generated speech more natural and emotionally expressive.

[0151] In one embodiment, in the above S30, generating a text prosody representation based on the frame-level prosody change includes:

[0152] S301, extracting pitch features, duration features and rhythm features from the frame-level rhythmic changes;

[0153] S302, generating intonation changes based on the tone feature, determining the duration and pause position of each word or phrase based on the duration feature, and generating stress and rhythm changes in the text based on the rhythm feature;

[0154] S303: Integrate the intonation change, the duration and pause position of each word or phrase, and the stress and rhythm changes into a text prosody representation.

[0155] In this embodiment, the frame-level prosodic changes represent the dynamic changes of speech in the time frame, including features such as pitch, duration and rhythm. By extracting these features, the prosodic information of the speech can be further generated, so that the generated speech sounds more natural. The pitch feature reflects the high and low changes in the speech, and determines the tone and emotional expression. The duration feature reflects the pronunciation duration of each syllable or word, and determines the rhythm and fluency of the speech. Rhythmic features: capture the stress and rhythm changes in the speech to ensure the fluency and coherence of the speech. The pitch, duration and rhythm features of each frame are extracted through a recursive neural network (RNN) or a convolutional neural network (CNN). The audio on each time frame is processed, and these features are extracted and recorded frame by frame to form complete prosodic information.

[0156] The pitch feature reflects the high and low changes in speech. Generating intonation changes is to ensure that the generated speech can convey appropriate emotions and semantics. Intonation changes are generated by the ups and downs of the pitch, ensuring that the speech has a natural intonation. Use the pitch feature to adjust the pitch of each syllable or word, and generate intonation in the speech to make it conform to the grammatical and emotional requirements of the language.

[0157] Duration features determine the rhythm and pauses of speech. Based on the duration features, the duration of each word or phrase can be generated and the location of pauses in the sentence can be determined to enhance the fluency of speech. Duration control ensures that the pronunciation time of each syllable, word or phrase conforms to natural language expression. Based on the extracted duration features, the system determines the duration of each word or phrase. Based on the duration features, natural pauses are automatically generated between sentences or phrases to ensure a balanced speaking speed.

[0158] Rhythm features are used to generate stress and rhythm changes in speech, ensuring that the generated speech meets the requirements of language expression in terms of rhythm and enhancing the coherence and naturalness of speech. Stress and rhythm make speech more natural by controlling the stress position and rhythm changes in sentences. The system generates stress and rhythm changes in sentences based on rhythm features, ensuring that the speech feels natural and has a smooth rhythm.

[0159] The generated text prosodic representation is an integration of pitch, duration, pause, and rhythm changes, which is used to guide the speech generation module to ensure that the speech has natural rhythm and emotional expression. The text prosodic representation is the core input for generating speech, ensuring that the generated speech can meet the natural expression and emotional requirements of the language. The system integrates intonation changes, duration and pause positions, stress and rhythm changes into a unified prosodic representation as the input for subsequent speech generation.

[0160] This embodiment generates a text prosodic representation based on frame-level prosodic changes, and the system can generate natural, fluent and emotional speech output. The precise control of the prosodic representation makes the generated speech not only conform to the grammatical structure of the text, but also express appropriate emotions and pitch changes, greatly improving the naturalness and emotional transmission ability of the generated speech.

[0161] In one embodiment, an emotional speech conversion device based on prosody prediction is provided, and the emotional speech conversion device based on prosody prediction corresponds one-to-one to the emotional speech conversion method based on prosody prediction in the above embodiment. Figure 3 , Figure 3 This is a functional module diagram of a preferred embodiment of the emotional speech conversion device based on prosody prediction of the present invention. Data processing module 10, text encoder module 20, prosody prediction module 30, emotional descriptor module 40, joint training module 50, prosody integration module 60, emotional speech synthesis module 70 and model optimization and emotional speech generation module 80. The functional modules are described in detail as follows:

[0162] The data processing module 10 is used to process the input text data and source audio data, and extract the phoneme sequence, acoustic features and emotional dimension representation;

[0163] A text encoder module 20, for extracting language information from the phoneme sequence using a text encoder;

[0164] A prosody prediction module 30, configured to predict frame-level prosody changes and duration of emotional speech based on the language information and the emotional dimension representation using a prosody prediction module, and generate a text prosody representation based on the frame-level prosody changes;

[0165] An emotion descriptor module 40, configured to extract speaker features from the source audio data using a speaker encoder, and to generate an acoustic prosody representation corresponding to the acoustic features based on the speaker features and the emotion dimension representation using an emotion descriptor;

[0166] A joint training module 50, configured to determine an alignment loss between the text prosody representation and the acoustic prosody representation, and to jointly train the prosody prediction module and the acoustic modeling module based on the alignment loss, wherein the acoustic modeling module includes the speaker encoder and the emotion descriptor;

[0167] A prosody integration module 60, for generating a comprehensive prosody representation based on the acoustic prosody representation, text prosody information and emotional dimension representation using a prosody integrator;

[0168] An emotional speech synthesis module 70, for generating a converted speech waveform using the emotional speech synthesis module based on the comprehensive prosody representation and the duration prediction result of the emotional speech, and evaluating the speech waveform using a discriminator to determine a comprehensive loss;

[0169] The model optimization and emotional speech generation module 80 is used to optimize the model parameters of the prosody prediction module, the acoustic modeling module and the emotional speech synthesis module through back propagation based on the comprehensive loss, and use the optimized modules to generate target emotional speech.

[0170] In one embodiment, the data processing module 10 is specifically used for:

[0171] Perform word segmentation on text data and convert the segmented text data into a phoneme sequence;

[0172] Extracting a linear spectrogram from the source audio data using a short-time Fourier transform;

[0173] Extracting fundamental frequency and duration information from the source audio data, and using the linear spectrogram, fundamental frequency and duration information as acoustic features;

[0174] The acoustic features are input into an emotion recognition model, and an emotion dimension representation is generated by analyzing the acoustic features. The emotion dimension representation includes the arousal degree and valence of the emotion, which are used to represent the activation degree of the emotion and the positive and negative attributes of the emotion.

[0175] In one embodiment, the joint training module 50 is specifically configured to:

[0176] Using Kullback-Leibler divergence to determine the distribution difference between the text prosody representation and the acoustic prosody representation, and using the distribution difference as the alignment loss for joint training;

[0177] Based on the alignment loss, determining the gradients of the prosody prediction module and the acoustic modeling module through back propagation;

[0178] Use the gradient descent method to update the parameters of the rhythm prediction module and the acoustic modeling module according to the gradient

[0179] By alternately training the prosody prediction module and the acoustic modeling module, the alignment loss is re-determined after each training and the module parameters are gradually optimized.

[0180] In one embodiment, the rhythm integration module 60 is specifically used to:

[0181] Performing weighted fusion of the acoustic prosody representation, text prosody information and emotional dimension representation through a prosody integrator to generate a preliminary fusion representation;

[0182] Using the feature extraction module in the prosody integrator to extract features from the preliminary fusion representation to generate a fusion feature representation;

[0183] The fused feature representation is input into a nonlinear activation function for processing to generate a final comprehensive rhythmic representation.

[0184] In one embodiment, the emotional speech synthesis module 70 is specifically used for:

[0185] Inputting the comprehensive prosody representation and the duration prediction result of the emotional speech into an emotional speech synthesis module, and generating a speech waveform after emotion conversion through the emotional speech synthesis module;

[0186] Using a discriminator to evaluate the speech waveform, evaluate the naturalness, authenticity and accuracy of emotional expression of the speech waveform, and generate corresponding evaluation results;

[0187] Determining a reconstruction loss between the speech waveform and a target real speech;

[0188] Based on the evaluation result, determining the authenticity and naturalness loss of the speech waveform;

[0189] Based on the emotion dimension representation, determining the difference between the emotion expression of the speech waveform and the expected emotion label, and determining the emotion classification loss;

[0190] The reconstruction loss, adversarial loss and sentiment classification loss are combined to determine the comprehensive loss.

[0191] In one embodiment, the emotion descriptor module 40 is specifically configured to:

[0192] The speaker characteristics and the emotional dimension representation are weightedly integrated by the emotional descriptor to generate preliminary emotional prosody information;

[0193] The feature extraction module in the emotion descriptor is used to process the fused rhythm information and extract higher-level features of the emotion rhythm;

[0194] Based on higher-level prosodic features, the final acoustic prosodic representation is generated.

[0195] In one embodiment, the prosody prediction module 30 is specifically configured to:

[0196] Extracting pitch features, duration features and rhythm features from the frame-level rhythmic changes;

[0197] Generate intonation changes based on the pitch feature, determine the duration and pause position of each word or phrase based on the duration feature, and generate stress and rhythm changes in the text based on the rhythm feature;

[0198] The intonation changes, the duration and pause positions of each word or phrase, and the stress and rhythm changes are integrated into a prosodic representation of the text.

[0199] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 4 As shown. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external user terminal through a network connection. When the computer program is executed by the processor, it realizes the functions or steps of a service-side method of emotional speech conversion based on rhythm prediction.

[0200] In one embodiment, a computer device is provided. The computer device may be a user terminal, and its internal structure diagram may be as follows: Figure 5 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server through a network connection. When the computer program is executed by the processor, the functions or steps on the user side of a method for emotional speech conversion based on prosody prediction are realized.

[0201] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:

[0202] Process the input text data and source audio data to extract phoneme sequences, acoustic features, and emotional dimension representations;

[0203] extracting language information from the phoneme sequence using a text encoder;

[0204] Using a prosody prediction module to predict frame-level prosody changes and duration of emotional speech based on the language information and the emotional dimension representation, and generating a text prosody representation based on the frame-level prosody changes;

[0205] Extracting speaker features from the source audio data using a speaker encoder, and generating an acoustic prosody representation corresponding to the acoustic features based on the speaker features and the emotional dimension representation using an emotion descriptor;

[0206] Determine an alignment loss between the text prosody representation and the acoustic prosody representation, and jointly train the prosody prediction module and the acoustic modeling module based on the alignment loss, wherein the acoustic modeling module includes the speaker encoder and the emotion descriptor;

[0207] generating a comprehensive prosodic representation based on the acoustic prosodic representation, the text prosodic information and the emotional dimension representation using a prosodic integrator;

[0208] Based on the comprehensive prosodic representation and the duration prediction results of the emotional speech, using the emotional speech synthesis module to generate a converted speech waveform, using the discriminator to evaluate the speech waveform and determine the comprehensive loss;

[0209] Based on the comprehensive loss, the model parameters of the prosody prediction module, the acoustic modeling module and the emotional speech synthesis module are optimized by back propagation, and the target emotional speech is generated using the optimized modules.

[0210] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0211] Process the input text data and source audio data to extract phoneme sequences, acoustic features, and emotional dimension representations;

[0212] extracting language information from the phoneme sequence using a text encoder;

[0213] Using a prosody prediction module to predict frame-level prosody changes and duration of emotional speech based on the language information and the emotional dimension representation, and generating a text prosody representation based on the frame-level prosody changes;

[0214] Extracting speaker features from the source audio data using a speaker encoder, and generating an acoustic prosody representation corresponding to the acoustic features based on the speaker features and the emotional dimension representation using an emotion descriptor;

[0215] Determine an alignment loss between the text prosody representation and the acoustic prosody representation, and jointly train the prosody prediction module and the acoustic modeling module based on the alignment loss, wherein the acoustic modeling module includes the speaker encoder and the emotion descriptor;

[0216] generating a comprehensive prosodic representation based on the acoustic prosodic representation, the text prosodic information and the emotional dimension representation using a prosodic integrator;

[0217] Based on the comprehensive prosodic representation and the duration prediction results of the emotional speech, using the emotional speech synthesis module to generate a converted speech waveform, using the discriminator to evaluate the speech waveform and determine the comprehensive loss;

[0218] Based on the comprehensive loss, the model parameters of the prosody prediction module, the acoustic modeling module and the emotional speech synthesis module are optimized by back propagation, and the target emotional speech is generated using the optimized modules.

[0219] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can refer to the relevant descriptions on the server side and the user side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0220] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0221] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0222] It should be noted that if software tools or components other than those of the Company appear in the embodiments of the present application, they are only used for illustration and do not represent actual use. The above-described embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the above-mentioned embodiments, a person of ordinary skill in the art should understand that the technical solutions described in the above-mentioned embodiments can still be modified, or some of the technical features thereof can be replaced by equivalents; and these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A method for emotional speech conversion based on prosody prediction, characterized in that: The following steps are involved: Process the input text data and source audio data to extract phoneme sequences, acoustic features, and emotional dimension representations; extracting language information from the phoneme sequence using a text encoder; Using a prosody prediction module to predict frame-level prosody changes and duration of emotional speech based on the language information and the emotional dimension representation, and generating a text prosody representation based on the frame-level prosody changes; Extracting speaker features from the source audio data using a speaker encoder, and generating an acoustic prosody representation corresponding to the acoustic features based on the speaker features and the emotional dimension representation using an emotion descriptor; Determine an alignment loss between the text prosody representation and the acoustic prosody representation, and jointly train the prosody prediction module and the acoustic modeling module based on the alignment loss, wherein the acoustic modeling module includes the speaker encoder and the emotion descriptor; generating a comprehensive prosodic representation based on the acoustic prosodic representation, the text prosodic information and the emotional dimension representation using a prosodic integrator; Based on the comprehensive prosodic representation and the duration prediction results of the emotional speech, using the emotional speech synthesis module to generate a converted speech waveform, using the discriminator to evaluate the speech waveform and determine the comprehensive loss; Based on the comprehensive loss, the model parameters of the prosody prediction module, the acoustic modeling module and the emotional speech synthesis module are optimized by back propagation, and the target emotional speech is generated using the optimized modules.

2. The method for converting emotional speech based on prosody prediction as claimed in claim 1, characterized in that: Process the input text data and source audio data, extract phoneme sequences, acoustic features, and emotional dimension representations, including: Perform word segmentation on text data and convert the segmented text data into a phoneme sequence; Extracting a linear spectrogram from the source audio data using a short-time Fourier transform; Extracting fundamental frequency and duration information from the source audio data, and using the linear spectrogram, fundamental frequency and duration information as acoustic features; The acoustic features are input into an emotion recognition model, and an emotion dimension representation is generated by analyzing the acoustic features. The emotion dimension representation includes the arousal degree and valence of the emotion, which are used to represent the activation degree of the emotion and the positive and negative attributes of the emotion.

3. The emotional speech conversion method based on prosody prediction as claimed in claim 1, characterized in that: Determining an alignment loss between the text prosody representation and the acoustic prosody representation, and jointly training the prosody prediction module and the acoustic modeling module based on the alignment loss, comprising: Using Kullback-Leibler divergence to determine the distribution difference between the text prosody representation and the acoustic prosody representation, and using the distribution difference as the alignment loss for joint training; Based on the alignment loss, determining the gradients of the prosody prediction module and the acoustic modeling module through back propagation; Using a gradient descent method, updating parameters of a prosody prediction module and an acoustic modeling module according to the gradient; The rhythm prediction module and the acoustic modeling module are trained alternately, the alignment loss is re-determined after each training, and the module parameters are gradually optimized.

4. The emotional speech conversion method based on prosody prediction as described in claim 1 is characterized in that: A prosody integrator is used to generate a comprehensive prosody representation based on the acoustic prosody representation, text prosody information, and emotional dimension representation, including: Performing weighted fusion of the acoustic prosody representation, text prosody information and emotional dimension representation through a prosody integrator to generate a preliminary fusion representation; Using the feature extraction module in the prosody integrator to extract features from the preliminary fusion representation to generate a fusion feature representation; The fused feature representation is input into a nonlinear activation function for processing to generate a final comprehensive rhythm representation.

5. The method for converting emotional speech based on prosody prediction as claimed in claim 1, characterized in that: Based on the comprehensive prosodic representation and the duration prediction results of the emotional speech, the emotional speech synthesis module is used to generate a converted speech waveform, and the discriminator is used to evaluate the speech waveform to determine the comprehensive loss, including: Inputting the comprehensive prosody representation and the duration prediction result of the emotional speech into an emotional speech synthesis module, and generating a speech waveform after emotion conversion through the emotional speech synthesis module; Using a discriminator to evaluate the speech waveform, evaluate the naturalness, authenticity and accuracy of emotional expression of the speech waveform, and generate corresponding evaluation results; Determining a reconstruction loss between the speech waveform and a target real speech; Based on the evaluation result, determining the loss of authenticity and naturalness of the speech waveform; Based on the emotion dimension representation, determining the difference between the emotion expression of the speech waveform and the expected emotion label, and determining the emotion classification loss; The reconstruction loss, adversarial loss and sentiment classification loss are combined to determine the comprehensive loss.

6. The emotional speech conversion method based on prosody prediction as claimed in claim 1, characterized in that: Generating an acoustic prosody representation corresponding to the acoustic feature based on the speaker feature and the emotional dimension representation using an emotional descriptor, comprising: The speaker characteristics and the emotional dimension representation are weightedly integrated by the emotional descriptor to generate preliminary emotional prosody information; The feature extraction module in the emotion descriptor is used to process the fused rhythm information and extract higher-level features of the emotion rhythm; Based on higher-level prosodic features, the final acoustic prosodic representation is generated.

7. The emotional speech conversion method based on prosody prediction as claimed in claim 1, characterized in that: Generating a text prosody representation based on the frame-level prosody change includes: Extracting pitch features, duration features and rhythm features from the frame-level rhythmic changes; Generate intonation changes based on the pitch feature, determine the duration and pause position of each word or phrase based on the duration feature, and generate stress and rhythm changes in the text based on the rhythm feature; The intonation changes, the duration and pause positions of each word or phrase, and the stress and rhythm changes are integrated into a prosodic representation of the text.

8. An emotional speech conversion device based on prosody prediction, characterized in that: The emotional speech conversion device based on prosody prediction comprises: A data processing module is used to process input text data and source audio data, extract phoneme sequences, acoustic features, and emotional dimension representations; A text encoder module, configured to extract language information from the phoneme sequence using a text encoder; A prosody prediction module, configured to predict frame-level prosody changes and duration of emotional speech based on the language information and the emotional dimension representation using the prosody prediction module, and generate a text prosody representation based on the frame-level prosody changes; An emotion descriptor module, configured to extract speaker features from the source audio data using a speaker encoder, and generate an acoustic prosody representation corresponding to the acoustic features based on the speaker features and the emotion dimension representation using an emotion descriptor; A joint training module, used for determining an alignment loss between the text prosody representation and the acoustic prosody representation, and jointly training the prosody prediction module and the acoustic modeling module based on the alignment loss, wherein the acoustic modeling module includes the speaker encoder and the emotion descriptor; A prosody integration module, for generating a comprehensive prosody representation based on the acoustic prosody representation, text prosody information and emotional dimension representation using a prosody integrator; An emotional speech synthesis module, for generating a converted speech waveform using the emotional speech synthesis module based on the comprehensive prosody representation and the duration prediction result of the emotional speech, and evaluating the speech waveform using a discriminator to determine a comprehensive loss; The model optimization and emotional speech generation module is used to optimize the model parameters of the prosody prediction module, the acoustic modeling module and the emotional speech synthesis module through back propagation based on the comprehensive loss, and use the optimized modules to generate target emotional speech.

9. A computer device, characterized in that: The computer device includes a memory, a processor, and an emotional speech conversion program based on prosody prediction stored in the memory and executable on the processor. When the emotional speech conversion program based on prosody prediction is executed by the processor, the steps of the emotional speech conversion method based on prosody prediction as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The storage medium stores an emotional speech conversion program based on prosody prediction, and when the emotional speech conversion program based on prosody prediction is executed by the processor, the steps of the emotional speech conversion method based on prosody prediction as described in any one of claims 1-7 are implemented.

Images