Emotional speech synthesis method, device, computer equipment and readable storage medium

Abstract

The present invention relates to the field of artificial intelligence and speech synthesis technology, and discloses an emotional speech synthesis method, apparatus, computer equipment, and readable storage medium, comprising: obtaining a target text to be speech synthesized, identifying an emotional vector representation of the target text; predicting the emotional style of the emotional vector representation to obtain an emotional style weight; determining the phoneme embedding sequence of the target text based on a pre-trained acoustic model, and controlling the acoustic model to adjust the phoneme embedding sequence using the emotional style weight so that the acoustic model generates the emotional speech of the target text using the adjusted phoneme embedding sequence. Even in the absence of professional audio material or emotional annotation, speech output with high emotional expressiveness can be obtained, thereby improving the generalization ability of the model.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence and speech synthesis technology, and in particular to an emotional speech synthesis method, device, computer equipment and readable storage medium. Background Art

[0002] As a key innovation in the field of artificial intelligence, emotional speech synthesis technology is demonstrating broad application prospects across multiple industries, including healthcare and finance. In healthcare settings, this technology imbues personalized virtual assistants with emotionality, making interactions with patients more approachable and humane, significantly enhancing the patient experience. Furthermore, emotional speech synthesis technology can be applied to emotional AI therapists. By simulating the therapist's emotional expressions, it provides patients with a more emotionally supportive conversational experience, thereby enhancing treatment effectiveness.

[0003] In the related art, the emotional speech synthesis framework relies on manually annotated emotion tags, so that when synthesizing emotional speech, it is necessary to input emotion tags for the text of the speech to be generated, so that the emotional speech synthesis model can synthesize emotional speech according to the input emotion tags. However, the applicant recognizes that the accuracy of the emotion tags annotated during model training is often affected by the subjectivity of the annotator, and there may be significant differences between different annotators. In addition, the reliance on tags limits the generalization ability of the model, making it impossible for the trained emotional speech synthesis model to accurately identify emotions when faced with unseen emotion tags or emotion expression tags in natural conversations, making it difficult to generate natural and realistic emotional speech. Summary of the Invention

[0004] The present invention provides an emotional speech synthesis method, apparatus, computer equipment and readable storage medium to solve the technical problem that the generalization ability of the model is limited, so that the trained emotional speech synthesis model cannot accurately recognize emotions when faced with unseen emotion labels or emotion expression labels in natural conversations, and it is difficult to generate natural and realistic emotional speech.

[0005] In a first aspect, a method for emotional speech synthesis is provided, comprising:

[0006] Obtaining a target text to be subjected to speech synthesis, and identifying a sentiment vector representation of the target text;

[0007] Performing emotional style prediction on the emotional vector representation to obtain an emotional style weight;

[0008] Based on a pre-trained acoustic model, a phoneme embedding sequence of the target text is determined, and the acoustic model is controlled to adjust the phoneme embedding sequence using the emotional style weight so that the acoustic model generates the emotional speech of the target text using the adjusted phoneme embedding sequence.

[0009] In a second aspect, an emotional speech synthesis device is provided, comprising:

[0010] A recognition module is used to obtain a target text to be subjected to speech synthesis and identify a sentiment vector representation of the target text;

[0011] A prediction module, configured to perform emotional style prediction on the emotional vector representation to obtain an emotional style weight;

[0012] A generation module is used to determine the phoneme embedding sequence of the target text based on a pre-trained acoustic model, and control the acoustic model to adjust the phoneme embedding sequence using the emotional style weight so that the acoustic model generates the emotional speech of the target text using the adjusted phoneme embedding sequence.

[0013] In a third aspect, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned emotional speech synthesis method when executing the computer program.

[0014] In a fourth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned emotional speech synthesis method are implemented.

[0015] In the scheme implemented by the above-mentioned emotional speech synthesis method, device, computer equipment and readable storage medium, the target text to be speech synthesized is obtained, the emotional vector representation of the target text is identified, the emotional style of the emotional vector representation is predicted, and the emotional style weight is obtained. Based on the pre-trained acoustic model, the phoneme embedding sequence of the target text is determined, and the acoustic model is controlled to adjust the phoneme embedding sequence using the emotional style weight, so that the acoustic model uses the adjusted phoneme embedding sequence to generate the emotional speech of the target text. The process of synthesizing emotional speech does not require any additional reference audio or emotional labels as an aid. Even in the absence of professional audio materials or emotional annotations, speech output with high emotional expressiveness can be obtained, which improves the generalization ability of the model, so that the trained emotional speech synthesis model can accurately identify emotions and generate natural and realistic emotional speech when faced with unseen emotional labels or emotional expression labels in natural conversations. It can provide strong technical support for speech synthesis in any scenario. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0017] Figure 1 2 is a schematic diagram of an application environment of an emotional speech synthesis method according to an embodiment of the present invention;

[0018] Figure 2 is a flow chart of an emotional speech synthesis method according to an embodiment of the present invention;

[0019] Figure 3 This is a flowchart of a specific implementation method of the emotional speech synthesis method;

[0020] Figure 4 This is a flowchart of a specific implementation method of the emotional speech synthesis method;

[0021] Figure 5 This is a flowchart of a specific implementation method of the emotional speech synthesis method;

[0022] Figure 6 This is a flowchart of a specific implementation method of the emotional speech synthesis method;

[0023] Figure 7 This is a flowchart of a specific implementation method of the emotional speech synthesis method;

[0024] Figure 8 This is a flowchart of a specific implementation method of the emotional speech synthesis method;

[0025] Figure 9 This is a flowchart of a specific implementation method of the emotional speech synthesis method;

[0026] Figure 10 is a structural diagram of an emotional speech synthesis device according to an embodiment of the present invention;

[0027] Figure 11 is a structural diagram of a computer device in one embodiment of the present invention;

[0028] Figure 12 FIG. 2 is another structural diagram of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0029] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0030] The emotional speech synthesis method provided by the embodiment of the present invention can be applied in Figure 1 In an application environment, the client communicates with the server via a network. The server can obtain the target text to be synthesized, identify the target text's emotional vector representation, predict the emotional style of the emotional vector representation, obtain the emotional style weight, determine the target text's phoneme embedding sequence based on a pre-trained acoustic model, and control the acoustic model to adjust the phoneme embedding sequence using the emotional style weight so that the acoustic model generates the target text's emotional speech using the adjusted phoneme embedding sequence, and feed the generated emotional speech back to the client. In the present invention, for medical scenarios, the process of synthesizing emotional speech does not require any additional reference audio or emotional labels as an aid. Even in the absence of professional audio materials or emotional annotations, speech output with high emotional expressiveness can be obtained, which not only lowers the threshold for emotional speech synthesis but also greatly broadens its application scenarios. Whether it is a warm reminder from medical intelligent customer service, product explanations in financial software, or even personalized dubbing for entertainment products, it can provide strong technical support. The client can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented as a standalone server or a server cluster consisting of multiple servers. The present invention is described in detail below through specific embodiments.

[0031] In order to achieve the synthesis of emotional speech, the present application embodiment constructs a style adaptation model and an acoustic model. The following describes the training process of these two models. Figure 2 As shown, Figure 2 A schematic diagram of a model training process of the emotional speech synthesis method provided in an embodiment of the present invention includes the following steps:

[0032] S10: Acquire multiple sample texts.

[0033] In this embodiment of the application, in order to train the model, the server collects multiple sample texts. Each sample text corresponds to a sample voice and a sample style weight. The sample voice is the recording corresponding to the sample text, and the sample style weight reflects the required expression style strength of the sample text, such as gentle, serious, or encouraging.

[0034] In medical scenarios, the collected sample texts can be doctors' diagnosis instructions and rehabilitation suggestions for patients. For a sample text about postoperative rehabilitation suggestions, if you want the voice to sound more inspiring, you can assign it a higher encouragement style weight. This provides rich and clearly labeled data resources for subsequent model training, helping the model learn speech features in different styles.

[0035] S20: extracting a target sample text from the multiple sample texts.

[0036] In the embodiment of the present application, the server will use the fine-tuned DistilROBERTa language model to predict the probability of the text's emotional category, and select text-speech pairs with obvious emotional tendencies as target sample texts, so that the model trained using the target sample texts will be more accurate. Figure 3 As shown, extracting a target sample text from a plurality of sample texts includes the following steps S21 to S22:

[0037] S21: Use the fine-tuned pre-trained language model to predict the probability of the sentiment category corresponding to each sample text.

[0038] In this embodiment of the present application, the server first uses a fine-tuned DistilBERTa language model to predict the sentiment category probabilities of a series of sample texts. DistilBERTa is a lightweight variant of the BERT model that uses knowledge distillation to reduce the model size and computational complexity while maintaining most of the performance of the original BERT model. Fine-tuning refers to training the pre-trained DistilBERTa model on a specific sentiment classification task dataset to enable it to better identify and classify the sentiment tendency of a given text.

[0039] In practice, the server inputs each sample text into a fine-tuned DistilBERTa model. For example, the sample text may include patient comments such as "I am very satisfied with the results of this treatment" or "I felt a lot of pain and discomfort after the surgery." The fine-tuned DistilBERTa model analyzes the words, phrases, and context in these texts and outputs a probability distribution for each text belonging to different sentiment categories (e.g., positive, negative, neutral).

[0040] By fine-tuning the DistilBERTa model, the server can more accurately capture the unique emotional expressions in medical texts, such as patient satisfaction with treatment outcomes and perceived pain levels. It efficiently obtains the emotional category probability for each sample text, providing reliable data support for subsequent emotional screening and model training. Furthermore, the lightweight nature of the DistilBERTa model makes this process highly feasible and practical even in resource-constrained environments.

[0041] S22: Obtain a preset probability threshold, and extract sample texts whose corresponding emotion category probabilities are greater than the probability threshold from multiple sample texts as target sample texts.

[0042] In an embodiment of the present application, a preset probability threshold is set in the server. After obtaining the sentiment category probability of each sample text, the server obtains the preset probability threshold and extracts the sample texts whose corresponding sentiment category probability is greater than the probability threshold from multiple sample texts as target sample texts, thereby screening out texts with clear sentiment expression as target sample texts. The preset probability threshold is set based on actual application needs and the requirements for sentiment classification accuracy, and can be determined through experience or experimentation.

[0043] Taking the medical scenario as an example, the server may set a high probability threshold (such as 0.8 or higher) to ensure that the selected text has a strong and clear sentiment. This way, only patient feedback that the model determines to be highly positive or highly negative is selected as target sample text. For example, if the model predicts that the sentence "I am very satisfied with the results of this treatment" has a probability of 0.95 of belonging to the positive sentiment category, then this sentence will be selected as the target sample text because it exceeds the preset probability threshold. Conversely, if the sentiment category probabilities of a sentence are more dispersed or uniformly low, it indicates that the emotional expression is not clear enough and will therefore not be selected. In this way, by setting a reasonable probability threshold and filtering sample text with sentiment category probabilities above the threshold, the server can effectively reduce noisy data and improve the accuracy and efficiency of subsequent model training. In medical scenarios, the server can focus more on text that clearly indicates the patient's emotional state, thereby training a model that is better able to understand and respond to patients' emotional needs.

[0044] S30. Use the fine-tuned pre-trained language model to extract the sample sentiment vector of the target sample text, and use the sample sentiment vector and the target sample style weight corresponding to the target sample text to train the style adaptation model.

[0045] In this embodiment of the present application, the server uses a fine-tuned pre-trained language model to analyze the target sample text and extract a sentiment vector that represents the emotional expression of the target sample text. The purpose of fine-tuning the pre-trained language model is to adapt it to the task of sentiment analysis of medical text. The fine-tuned pre-trained language model is consistent with that described in step S21 above and will not be further described here.

[0046] Subsequently, through the style adaptation model framework, the sentiment vector and style weights are used as input, and the model is trained to output stylized sentiment features. The style adaptation model framework can be a GST network. In this way, a style adaptation model is trained by combining the style weights corresponding to the text. This style adaptation model can learn how to combine the sentiment vector and style weights to generate voice instructions with a specific emotional style and intensity. For example, for a text encouraging a child to take medication on time, the style adaptation model will adjust the voice parameters such as intonation and rhythm based on its sentiment vector and "encouragement" style weight to make it sound more positive.

[0047] In this way, by introducing emotion vectors and style weights, the trained style adaptation model can generate more delicate speech that meets the needs of medical scenarios, enhancing the emotional expression ability and personalized features of speech synthesis. Figure 4 As shown, the process of training the style adaptation model includes the following steps S31 to S32:

[0048] S31. Use the fine-tuned pre-trained language model to extract the sample sentiment vector of the target sample text, take the sample sentiment vector as input and the target sample style weight as the target value, use a multi-category classification task to train the model, and obtain the initial adaptation model.

[0049] In this embodiment of the present application, the server first uses a fine-tuned pre-trained language model to extract the sentiment vector of the target sample text. The fine-tuned pre-trained language model can deeply understand the semantics and sentiment of the text, thereby accurately extracting the sample sentiment vector. The fine-tuned pre-trained language model is consistent with the one mentioned in step S21 above and will not be repeated here.

[0050] Next, the server takes the extracted sample sentiment vector as input and the target sample style weight as the target value. The target sample style weight (GST weight) is defined based on actual application requirements and represents the weight distribution of different sentiment styles, such as optimism, pessimism, and neutrality. In medical scenarios, these style weights correspond to different expectations and attitudes of patients or doctors regarding treatment outcomes.

[0051] To train the model, the server uses a multi-class classification task to enable the model to classify the input sample sentiment vector into one of the predefined GST weight categories. Through training, the model gradually establishes a mapping relationship between sentiment vectors and GST weights, thus obtaining an initial adaptation model.

[0052] For example, consider a healthcare scenario. Suppose the server has a set of texts about cancer treatment that include patient sentiment. Using a fine-tuned pre-trained language model, the server extracts sentiment vectors from these texts and maps them to the style weights of target samples (e.g., optimism vs. pessimism). Through training, the model learns to identify the style weights represented by different sentiment vectors, providing a foundation for subsequent sentiment-based style adaptation.

[0053] S32. Use the classification cross entropy loss function to calculate the loss value of the initial adaptation model, and adjust the model parameters of the initial adaptation model. Continue to train the adjusted initial adaptation model using the sample emotion vector as input and the target sample style weight as the target value, and recalculate the loss value until the calculated loss value reaches the first preset condition. The initial adaptation model that has completed the model parameter adjustment is used as the style adaptation model.

[0054] After obtaining the initial adaptation model, the server needs to further optimize the model's performance. To this end, the server uses the categorical cross entropy loss function to calculate the model's loss value. The categorical cross entropy loss function is a commonly used loss function that measures the difference between the probability distribution predicted by the model and the true label distribution. The specific function formula is as follows:

[0055] Formula 1: L = -∑i = 1Ny i log(p i )

[0056] Where L represents the calculated loss value; N represents the total number of sample sentiment vectors; yi represents the target sample style weight corresponding to the i-th target sample text; pi represents the predicted sample style weight predicted by the initial adaptation model for the sample sentiment vector of the i-th target sample text. In this way, through the above formula 1, the server calculates the predicted sample style weight p of each target sample text i and the target sample style weight y i The cross entropy loss between them is calculated, and then the loss values ​​of all samples are added together to obtain the total loss (that is, the loss value). The total loss value reflects the current mastery of the model on the emotional style adaptation task.

[0057] To reduce the loss, the server uses a backpropagation algorithm to calculate the gradient of the loss with respect to the model parameters and adjusts the model parameters using an optimization algorithm (such as Adam or SGD). The adjusted model is then trained again, using the same sample sentiment vector as input and the target sample style weights as the target value. During training, the server continuously calculates the loss and adjusts the model parameters to minimize it.

[0058] This process continues until the calculated loss value reaches a first pre-set condition. This first pre-set condition can be a small threshold or an upper limit on the number of training rounds, indicating that the model has been fully trained and has achieved the expected performance level. When the first pre-set condition is met, the server uses the initial adaptation model with the model parameter adjustment as the style adaptation model.

[0059] S40: Train an acoustic model using the multiple sample texts and the sample speech corresponding to the multiple sample texts.

[0060] In the embodiment of the present application, after obtaining rich sample text and its corresponding voice data, the server will use this data to train the acoustic model, which is responsible for converting the text into the corresponding voice waveform and generating high-quality voice output. Figure 5 As shown, the process includes the following steps S41 to S43:

[0061] S41: Using a preset reference encoder, a preset text encoder, a preset variational adapter and a decoder, each sample text and the sample speech corresponding to each sample text are processed to generate a predicted mel-spectrogram for the sample speech corresponding to each sample text.

[0062] In the embodiment of the present application, the server prepares a complete processing framework in advance, which includes a preset reference encoder, a preset text encoder, a preset variational adapter and a decoder. These components work together to convert the sample text and its corresponding sample speech into a mel-spectrogram, an acoustic feature representation. When generating a predicted mel-spectrogram for the sample speech corresponding to each sample text, such as Figure 6 As shown, the following steps S411 to S414 are included:

[0063] S411: For each sample text, perform phoneme conversion on the sample text to obtain a sample phoneme sequence of the sample text.

[0064] In this embodiment of the present application, the server first processes the sample text and converts it into a phoneme sequence. Phonemes are the smallest units that make up language sounds. Through phoneme conversion, the server can convert text information into a lower-level acoustic representation, providing a foundation for subsequent steps.

[0065] Specifically, the server can use text analysis software or algorithms to preprocess the sample text, including word segmentation, part-of-speech tagging, etc., and then use a phoneme dictionary or phoneme recognition model to convert each word or syllable in the text into the corresponding phoneme, and finally obtain a sample phoneme sequence of the sample text.

[0066] Taking the medical scenario as an example, suppose the server has a set of medical records about heart disease diagnosis. Through the processing of this step, the server can convert these texts into phoneme sequences, providing basic data for the subsequent generation of acoustic features.

[0067] S412: Encode the sample speech corresponding to the sample text using a preset reference encoder to obtain a sample style vector of the sample speech, and adjust the preset text encoder using the sample style vector.

[0068] In an embodiment of the present application, the server uses a preset reference encoder to encode the sample speech corresponding to the sample text and extracts the style features therein. The style features reflect the personalized characteristics of the speech, such as speaking speed, intonation, pronunciation habits, etc.

[0069] Specifically, the preset reference encoder is a pre-trained neural network model that extracts and encodes features from the input speech signal, outputting a vector representing the speech style. The server then uses this sample style vector to adjust the preset text encoder to better adapt it to the stylistic characteristics of the current speech sample.

[0070] For example, when processing a recording of a conversation between a doctor and a patient, the speech information in the recording can be used to generate a sample style vector and use it to adjust the text encoder to more accurately capture the doctor's speech characteristics when expressing medical terms or diagnostic opinions.

[0071] S413: Using the adjusted preset text encoder, convert the sample phoneme sequence into a sample latent sequence.

[0072] In this embodiment of the present application, the server uses a modified preset text encoder to convert the sample phoneme sequence into a sample latent sequence. The latent sequence is the internal representation obtained by the text encoder after encoding the input phoneme sequence. It is a high-dimensional, abstract feature representation that contains comprehensive information about the sample text and the sample speech style.

[0073] Specifically, the preset text encoder is a neural network model that encodes an input phoneme sequence and outputs a hidden sequence representing the text. The encoding process ensures continuity and smoothness in the transformation to preserve the key features of the original information. By adjusting the preset text encoder, the server can better capture the deep features in the text, providing stronger support for subsequent acoustic feature generation.

[0074] S414: Using a preset variational adapter to adjust the sample hidden sequence, and using a decoder to convert the adjusted sample hidden sequence into a mel-spectrogram, to obtain a predicted mel-spectrogram corresponding to the sample text.

[0075] In this embodiment of the present application, the server further adjusts and optimizes the sample hidden sequence. Specifically, the server uses a preset variational adapter to transform the sample hidden sequence to generate a representation that better matches the target acoustic features. The variational adapter has nonlinear transformation capabilities and can flexibly adjust the feature distribution and interrelationships in the hidden sequence.

[0076] The server then uses a decoder to convert the adjusted sample hidden sequence into a mel-spectrogram. Mel-spectrograms are a commonly used acoustic feature representation method that reflects the acoustic characteristics of audio signals by converting them into energy distributions across a series of frequency bands. Through the decoder's conversion process, the server obtains the predicted mel-spectrogram corresponding to the sample text.

[0077] Taking the medical scenario as an example, assuming there is a set of sample texts about heart disease diagnosis and corresponding doctor voice samples, a predicted mel-spectrogram corresponding to its content can be generated for each voice sample, providing acoustic feature support for subsequent analysis and diagnosis.

[0078] S42: Obtain a sample Mel spectrogram associated with the sample speech corresponding to each sample text, and calculate a mean square error loss value between the predicted Mel spectrogram generated for each sample text and its associated sample Mel spectrogram.

[0079] To evaluate the accuracy of the predicted mel-spectrogram, the server needs to obtain the actual sample mel-spectrogram (also known as the sample mel-spectrogram) corresponding to each sample text. The server then calculates the mean squared error (MSE) loss between the predicted mel-spectrogram and the actual sample mel-spectrogram to quantify the difference between the two. The smaller the MSE loss value, the closer the predicted mel-spectrogram is to the actual sample mel-spectrogram, which means the model's prediction performance is better. By comparing the MSE loss values ​​of the predicted mel-spectrogram and the actual sample mel-spectrogram, the server can evaluate the model's performance and optimize it.

[0080] S43: Adjust the preset text encoder, preset variational adapter and decoder according to the calculated mean square error loss value, and use the adjusted preset text encoder, adjusted preset variational adapter and adjusted decoder to regenerate the predicted Mel-spectrogram for each sample speech and calculate the new mean square error loss value, continue to adjust the preset text encoder, preset variational adapter and decoder according to the new mean square error loss value until the calculated mean square error loss value reaches the preset second condition, integrate the currently adjusted preset text encoder, preset variational adapter and decoder to obtain the acoustic training model.

[0081] After obtaining the mean squared error (MSE) loss, the server uses the backpropagation algorithm to adjust the parameters of the preset text encoder, preset variational adapter, and decoder. Through iterative optimization, the server continuously reduces the difference between the predicted mel-spectrogram and the true sample mel-spectrogram until the MSE loss value reaches the second preset condition. The setting strategy for the second preset condition is the same as the setting strategy for the first preset condition mentioned above and will not be repeated here.

[0082] Finally, the server integrates the adjusted preset text encoder, preset variational adapter, and decoder to form a complete acoustic model. In this acoustic model, the preset text encoder converts the phoneme embedding sequence into a hidden sequence. The preset variational adapter adjusts the hidden sequence to control prosodic features such as pitch, duration, and energy. The decoder converts the adjusted hidden sequence into a mel-spectrogram. In practical applications, FastSpeech2 can be used as the basic framework for training acoustic models.

[0083] The above training model does not involve the annotation of emotional labels. In the training process, no emotional labels are relied upon. Instead, the emotional information of the text content itself is used to learn emotional expression. Moreover, based on the text emotion perception of the language model, the pre-trained BERT language model is used to convert the text content into emotional embedding, thereby establishing a connection between text emotion and speech rhythm. After completing the training of the required model, the server can use the trained model to realize the synthesis of emotional speech. Specifically, Figure 7 As shown, the process includes the following steps S50 to S70:

[0084] S50: Obtain the target text to be speech synthesized, and identify the emotion vector representation of the target text.

[0085] In an embodiment of the present application, the server obtains the target text to be speech synthesized, uses a fine-tuned pre-trained language model to perform sentiment analysis on the target text, and identifies the sentiment vector representation of the target text. The fine-tuned pre-trained language model is described in the above content and will not be repeated here. In the medical scenario, the target text can be a doctor's order, patient comfort words, etc.; the sentiment vector is a multidimensional vector that can capture the emotional features in the text, such as positive, negative, comforting, nervous, etc. For example, when identifying the medical order text "Please rest assured, your condition is getting better", the server can capture the positive and comforting emotions therein and generate the corresponding sentiment vector. In this way, the recognition of the sentiment vector provides a basis for subsequent emotional style prediction, so that the generated speech can be closer to the real emotional expression of humans.

[0086] S60: Perform emotional style prediction on the emotional vector representation to obtain the emotional style weight.

[0087] In an embodiment of the present application, after obtaining the emotion vector, the server will use the style adaptation model trained in the above process to further analyze the emotion vector and predict its emotion style weight. The emotion style weight reflects the proportion of different emotions in the synthesized speech and determines the emotional tendency of the final speech. For example, when processing the soothing text "Please remain calm, we will arrange surgery for you as soon as possible", the emotion style prediction can ensure that the generated speech has enough soothing and calming emotions to reduce the patient's tension. In this way, through the introduction of emotion style weights, the server can more finely control the emotional expression of the synthesized speech, thereby improving the applicability of speech synthesis in various scenarios.

[0088] S70: Based on the pre-trained acoustic model, determine the phoneme embedding sequence of the target text, and control the acoustic model to adjust the phoneme embedding sequence using the emotional style weight, so that the acoustic model generates the emotional speech of the target text using the adjusted phoneme embedding sequence.

[0089] In the embodiment of the present application, after obtaining the emotional style weight, the server will determine the phoneme embedding sequence of the target text based on the pre-trained acoustic model, and control the acoustic model to adjust the phoneme embedding sequence using the emotional style weight, so that the acoustic model uses the adjusted phoneme embedding sequence to generate the emotional speech of the target text. Specifically, when generating the emotional speech of the target text, Figure 8 As shown, the process includes the following steps S71 to S74:

[0090] S71: Perform phoneme conversion on the target text to obtain a target phoneme sequence of the target text.

[0091] First, the server converts the target text into a phoneme sequence. Phonemes are the smallest units of speech. Through phoneme conversion, text content can be converted into a format that can be processed by the acoustic model. For example, when converting "Please rest after surgery" into a phoneme sequence, the server decomposes each Chinese character into its corresponding pinyin phoneme.

[0092] S72: Input the target phoneme sequence into the acoustic model, so that the preset text encoder in the acoustic model converts the target phoneme sequence into a target hidden sequence.

[0093] Next, the server inputs the target phoneme sequence into the pre-trained acoustic model. The preset text encoder in the acoustic model can convert the target phoneme sequence into a target hidden sequence, which contains the acoustic features of the speech.

[0094] S73: Based on a preset variational adapter in the acoustic model, adjusting the target hidden sequence, and converting the adjusted target hidden sequence into a target mel-spectrogram based on a decoder in the acoustic model.

[0095] After obtaining the target hidden sequence, the server uses the preset variational adapter in the acoustic model to adjust the hidden sequence based on the emotional style weights, aiming to integrate emotional features into the acoustic features and thus generate emotionally charged speech. After adjustment, the target hidden sequence is converted into a mel-spectrogram through the acoustic model's decoder. The mel-spectrogram is a frequency-domain representation that reflects the spectral characteristics of speech.

[0096] S74: Calling a vocoder to convert the target mel-spectrogram, and using the converted speech as the emotional speech of the target text.

[0097] Finally, the server invokes a vocoder (such as the Griffin-Lim algorithm or WaveGlow) to convert the mel-spectrogram into the final speech output. The vocoder converts the mel-spectrogram into a time-domain signal, audible to humans. This process incorporates emotional features, so the generated speech is imbued with appropriate emotional overtones. For example, when processing the joyful news "The operation was successful, congratulations!", the server can generate a joyful voice message to convey to the patient and their family.

[0098] In summary, the logical process of the technical solution of this application is summarized as follows: Figure 9The processing of this application is divided into stage 1 and stage 2. In stage 1, first, suitable emotional data is selected from the available emotional data set, and the emotional data contains text and corresponding speech in different emotional states. Then, the MFA alignment operation is performed on the text content in the emotional data to obtain the phoneme vector of the text, and embedded into the phoneme embedding space through the text encoder. The GST network uses the reference encoder to encode the speech corresponding to the text into a style embedding, and uses the style embedding to adjust the text encoder so that the model can generate different speech styles. The text encoder converts the phoneme embedding sequence into a hidden sequence, the variational adapter adjusts the hidden sequence to control rhythmic features such as pitch, duration and energy, and the decoder converts the adjusted hidden sequence into a mel spectrogram. In stage 2, suitable emotional data is selected from the available emotional data set, and the emotional data contains text and corresponding speech in different emotional states. On the one hand, the selected emotional speech is input into the reference encoder, and the speech encoding output by the reference encoder is transmitted to the GST network, so that the GST network uses the reference encoder to encode the speech corresponding to the text into a style embedding and perform its subsequent operations. On the other hand, the selected emotional text is input into the language model to generate the corresponding emotional text vector. The emotional text vector is used as input, the GST weight is used as the target value, and the adaptation module is trained using a multi-category classification task to map the emotional embedding into the emotional style space to realize the style space exploration guided by the language model.

[0099] It can be seen that in the above scheme, the target text to be speech synthesized is obtained, the emotion vector representation of the target text is identified, the emotion style is predicted for the emotion vector representation, and the emotion style weight is obtained. Based on the pre-trained acoustic model, the phoneme embedding sequence of the target text is determined, and the acoustic model is controlled to adjust the phoneme embedding sequence using the emotion style weight, so that the acoustic model uses the adjusted phoneme embedding sequence to generate the emotional speech of the target text. The process of synthesizing emotional speech does not require any additional reference audio or emotion labels as an aid. Even in the absence of professional audio materials or emotion annotations, speech output with high emotional expressiveness can be obtained, which improves the generalization ability of the model, so that the trained emotional speech synthesis model can accurately identify emotions and generate natural and realistic emotional speech when faced with unseen emotion labels or emotion expression labels in natural conversations. It can provide strong technical support for speech synthesis in any scenario.

[0100] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0101] In one embodiment, an emotional speech synthesis device is provided, which corresponds to the emotional speech synthesis method in the above embodiment. Figure 10 As shown, the emotional speech synthesis device includes a recognition module 101, a prediction module 102 and a generation module 103. The functional modules are described in detail as follows:

[0102] The recognition module 101 is used to obtain a target text to be subjected to speech synthesis and recognize an emotion vector representation of the target text;

[0103] The prediction module 102 is used to perform emotional style prediction on the emotional vector representation to obtain an emotional style weight;

[0104] The generation module 103 is used to determine the phoneme embedding sequence of the target text based on the pre-trained acoustic model, and control the acoustic model to adjust the phoneme embedding sequence using the emotional style weight so that the acoustic model uses the adjusted phoneme embedding sequence to generate the emotional speech of the target text.

[0105] In one embodiment, the emotional speech synthesis device further includes:

[0106] A training module is configured to obtain a plurality of sample texts, each of which corresponds to a sample speech and a sample style weight; extract a target sample text from the plurality of sample texts, extract a sample emotion vector of the target sample text using a fine-tuned pre-trained language model, and train a style adaptation model using the sample emotion vector and the target sample style weight corresponding to the target sample text; and train an acoustic model using the plurality of sample texts and the sample speech corresponding to the plurality of sample texts;

[0107] Accordingly, the recognition module 101 is configured to recognize the sentiment vector representation of the target text using the fine-tuned pre-trained language model;

[0108] Accordingly, the prediction module 102 is configured to use the style adaptation model to predict the emotional style weight represented by the emotional vector.

[0109] In one embodiment, the training module is used to use the fine-tuned pre-trained language model to predict the probability of the emotion category corresponding to each of the sample texts; obtain a preset probability threshold, and extract the sample text whose corresponding emotion category probability is greater than the probability threshold from the multiple sample texts as the target sample text.

[0110] In one embodiment, the training module is used to use the fine-tuned pre-trained language model to extract the sample emotion vector of the target sample text, use the sample emotion vector as input and the target sample style weight as the target value, and use a multi-category classification task to perform model training to obtain an initial adaptation model; use the classification cross entropy loss function to calculate the loss value of the initial adaptation model, and adjust the model parameters of the initial adaptation model, continue to use the sample emotion vector as input and the target sample style weight as the target value to train the adjusted initial adaptation model, and recalculate the loss value until the calculated loss value reaches the first preset condition, and use the initial adaptation model that has completed the model parameter adjustment as the style adaptation model.

[0111] In one embodiment, the training module is used to process each of the sample texts and the sample speech corresponding to each of the sample texts using a preset reference encoder, a preset text encoder, a preset variational adapter, and a decoder, and generate a predicted Mel-spectrogram for the sample speech corresponding to each of the sample texts; obtain a sample Mel-spectrogram associated with the sample speech corresponding to each of the sample texts, and calculate a mean square error loss value between the predicted Mel-spectrogram generated for each of the sample texts and the sample Mel-spectrogram associated with it; adjust the preset text encoder, the preset variational adapter, and the decoder according to the calculated mean square error loss value, and use the adjusted preset text encoder, the adjusted preset variational adapter, and the adjusted decoder to regenerate a predicted Mel-spectrogram for each of the sample speech and calculate a new mean square error loss value; continue to adjust the preset text encoder, the preset variational adapter, and the decoder according to the new mean square error loss value until the calculated mean square error loss value reaches a second preset condition, and integrate the currently adjusted preset text encoder, the preset variational adapter, and the decoder to obtain the acoustic model.

[0112] In one embodiment, the training module is used to perform phoneme conversion on each of the sample texts to obtain a sample phoneme sequence of the sample text; use the preset reference encoder to encode the sample speech corresponding to the sample text to obtain a sample style vector of the sample speech, and use the sample style vector to adjust the preset text encoder; use the adjusted preset text encoder to convert the sample phoneme sequence into a sample hidden sequence; use a preset variational adapter to adjust the sample hidden sequence, and use the decoder to convert the adjusted sample hidden sequence into a mel-spectrogram to obtain a predicted mel-spectrogram corresponding to the sample text.

[0113] In one embodiment, the generation module 103 is used to perform phoneme conversion on the target text to obtain a target phoneme sequence of the target text; input the target phoneme sequence into the acoustic model so that the preset text encoder in the acoustic model converts the target phoneme sequence into a target hidden sequence; adjust the target hidden sequence based on the preset variational adapter in the acoustic model, and convert the adjusted target hidden sequence into a target Mel-spectrogram based on the decoder in the acoustic model; call the vocoder to convert the target Mel-spectrogram, and use the converted speech as the emotional speech of the target text.

[0114] The present invention provides an emotional speech synthesis device, which obtains a target text to be speech synthesized, identifies an emotional vector representation of the target text, predicts an emotional style for the emotional vector representation, obtains an emotional style weight, determines a phoneme embedding sequence of the target text based on a pre-trained acoustic model, and controls the acoustic model to adjust the phoneme embedding sequence using the emotional style weight, so that the acoustic model generates the emotional speech of the target text using the adjusted phoneme embedding sequence. The process of synthesizing emotional speech does not require any additional reference audio or emotional labels as an aid, and can obtain speech output with high emotional expressiveness even in the absence of professional audio materials or emotional annotations, thereby improving the generalization ability of the model, so that the trained emotional speech synthesis model can accurately identify emotions and generate natural and realistic emotional speech when faced with unseen emotional labels or emotional expression labels in natural conversations, and can provide strong technical support for speech synthesis in any scenario.

[0115] For the specific definition of the emotional speech synthesis device, please refer to the definition of the emotional speech synthesis method above, which will not be repeated here. The various modules in the above-mentioned emotional speech synthesis device can be implemented in whole or in part by software, hardware and their combination. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0116] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 11As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external client via a network connection. When the computer program is executed by the processor, it realizes the functions or steps on the server side of an emotional speech synthesis method.

[0117] In one embodiment, a computer device is provided. The computer device may be a client, and its internal structure diagram may be as follows: Figure 12 As shown. The computer device includes a processor, memory, network interface, display screen and input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it realizes the functions or steps of the client side of an emotional speech synthesis method

[0118] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the following steps are performed:

[0119] Obtaining a target text to be subjected to speech synthesis, and identifying a sentiment vector representation of the target text;

[0120] Performing emotional style prediction on the emotional vector representation to obtain an emotional style weight;

[0121] Based on a pre-trained acoustic model, a phoneme embedding sequence of the target text is determined, and the acoustic model is controlled to adjust the phoneme embedding sequence using the emotional style weight so that the acoustic model generates the emotional speech of the target text using the adjusted phoneme embedding sequence.

[0122] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:

[0123] Obtaining a target text to be subjected to speech synthesis, and identifying a sentiment vector representation of the target text;

[0124] Performing emotional style prediction on the emotional vector representation to obtain an emotional style weight;

[0125] Based on a pre-trained acoustic model, a phoneme embedding sequence of the target text is determined, and the acoustic model is controlled to adjust the phoneme embedding sequence using the emotional style weight so that the acoustic model generates the emotional speech of the target text using the adjusted phoneme embedding sequence.

[0126] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can be found in the relevant descriptions of the server side and the client side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0127] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0128] Those skilled in the art will clearly understand that for the sake of convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0129] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention, and should all be included in the scope of protection of the present invention.

Claims

1. An emotional speech synthesis method, characterized in that: include: Obtaining a target text to be subjected to speech synthesis, and identifying a sentiment vector representation of the target text; Performing emotional style prediction on the emotional vector representation to obtain an emotional style weight, wherein, before obtaining a target text to be subjected to speech synthesis and identifying the emotional vector representation of the target text, obtaining a plurality of sample texts, wherein each of the sample texts corresponds to a sample speech and a sample style weight, extracting a target sample text from the plurality of sample texts, extracting a sample emotional vector of the target sample text using a fine-tuned pre-trained language model, training a style adaptation model using the sample emotional vector and the target sample style weight corresponding to the target sample text, and using the style adaptation model to predict the emotional style weight of the emotional vector representation; Based on a pre-trained acoustic model, a phoneme embedding sequence of the target text is determined, and the acoustic model is controlled to adjust the phoneme embedding sequence using the emotional style weight so that the acoustic model generates the emotional speech of the target text using the adjusted phoneme embedding sequence.

2. The method according to claim 1, characterized in that Before obtaining the target text to be speech synthesized and identifying the emotion vector representation of the target text, the method further includes: Training an acoustic model using the multiple sample texts and sample speech corresponding to the multiple sample texts; Correspondingly, the identifying the sentiment vector representation of the target text includes: using the fine-tuned pre-trained language model to identify the sentiment vector representation of the target text.

3. The method according to claim 2, characterized in that Extracting a target sample text from the plurality of sample texts includes: Using the fine-tuned pre-trained language model, predicting the probability of the sentiment category corresponding to each of the sample texts; A preset probability threshold is obtained, and sample texts whose corresponding emotion category probabilities are greater than the probability threshold are extracted from the multiple sample texts as the target sample texts.

4. The method according to claim 2, characterized in that The method of extracting a sample sentiment vector of the target sample text using a fine-tuned pre-trained language model and training a style adaptation model using the sample sentiment vector and a target sample style weight corresponding to the target sample text includes: Extracting a sample sentiment vector of the target sample text using the fine-tuned pre-trained language model, taking the sample sentiment vector as input and the target sample style weight as a target value, and performing model training using a multi-category classification task to obtain an initial adaptation model; The classification cross entropy loss function is used to calculate the loss value of the initial adaptation model, and the model parameters of the initial adaptation model are adjusted. The sample emotion vector is continued to be used as input and the target sample style weight is used as the target value to train the adjusted initial adaptation model, and the loss value is recalculated until the calculated loss value reaches the first preset condition. The initial adaptation model that has completed the model parameter adjustment is used as the style adaptation model.

5. The method according to claim 2, characterized in that The method of training an acoustic model by using the plurality of sample texts and sample speech corresponding to the plurality of sample texts includes: Using a preset reference encoder, a preset text encoder, a preset variational adapter, and a decoder, each of the sample texts and the sample speech corresponding to each of the sample texts is processed to generate a predicted mel-spectrogram for the sample speech corresponding to each of the sample texts; Obtaining a sample Mel spectrogram associated with the sample speech corresponding to each of the sample texts, and calculating a mean square error loss value between the predicted Mel spectrogram generated for each of the sample texts and its associated sample Mel spectrogram; The preset text encoder, the preset variational adapter and the decoder are adjusted according to the calculated mean square error loss value, and the adjusted preset text encoder, the adjusted preset variational adapter and the adjusted decoder are used to regenerate a predicted Mel-spectrogram for each sample speech and calculate a new mean square error loss value. The preset text encoder, the preset variational adapter and the decoder are continued to be adjusted according to the new mean square error loss value until the calculated mean square error loss value reaches a second preset condition. The currently adjusted preset text encoder, the preset variational adapter and the decoder are integrated to obtain the acoustic model.

6. The method according to claim 5, characterized in that The method of processing each sample text and the sample speech corresponding to each sample text by using a preset reference encoder, a preset text encoder, and a preset variational adapter, and generating a predicted mel-spectrogram for the sample speech corresponding to each sample text, includes: For each of the sample texts, performing phoneme conversion on the sample text to obtain a sample phoneme sequence of the sample text; Encoding the sample speech corresponding to the sample text using the preset reference encoder to obtain a sample style vector of the sample speech, and adjusting the preset text encoder using the sample style vector; Using the adjusted preset text encoder, converting the sample phoneme sequence into a sample latent sequence; The sample hidden sequence is adjusted using a preset variational adapter, and the adjusted sample hidden sequence is converted into a mel-spectrogram using the decoder to obtain a predicted mel-spectrogram corresponding to the sample text.

7. The method according to claim 1, characterized in that The method of determining the phoneme embedding sequence of the target text based on the pre-trained acoustic model, and controlling the acoustic model to adjust the phoneme embedding sequence using the emotional style weight so that the acoustic model generates the emotional speech of the target text using the adjusted phoneme embedding sequence, includes: Performing phoneme conversion on the target text to obtain a target phoneme sequence of the target text; Inputting the target phoneme sequence into the acoustic model so that a preset text encoder in the acoustic model converts the target phoneme sequence into a target hidden sequence; Adjusting the target hidden sequence based on a preset variational adapter in the acoustic model, and converting the adjusted target hidden sequence into a target mel-spectrogram based on a decoder in the acoustic model; A vocoder is called to convert the target mel-spectrogram, and the converted speech is used as the emotional speech of the target text.

8. An emotional speech synthesis device, characterized in that: include: A recognition module is used to obtain a target text to be subjected to speech synthesis and identify a sentiment vector representation of the target text; A prediction module is used to predict the emotional style of the emotional vector representation to obtain an emotional style weight; wherein the device further includes: a training module, used to obtain multiple sample texts, wherein each sample text corresponds to a sample speech and a sample style weight; extract a target sample text from the multiple sample texts, use a fine-tuned pre-trained language model to extract a sample emotional vector of the target sample text, and use the sample emotional vector and the target sample style weight corresponding to the target sample text to train a style adaptation model; accordingly, the prediction module is used to use the style adaptation model to predict the emotional style weight represented by the emotional vector; A generation module is used to determine the phoneme embedding sequence of the target text based on a pre-trained acoustic model, and control the acoustic model to adjust the phoneme embedding sequence using the emotional style weight so that the acoustic model generates the emotional speech of the target text using the adjusted phoneme embedding sequence.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the emotional speech synthesis method according to any one of claims 1 to 7 are implemented.

10. A readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the emotional speech synthesis method according to any one of claims 1 to 7 are implemented.

Images