Emotional speech synthesis method and device, computer equipment and readable storage medium

Abstract

The invention relates to the technical field of artificial intelligence and speech synthesis, and discloses an emotional speech synthesis method and device, computer equipment and a readable storage medium, and the method comprises the steps: obtaining a target text to be subjected to speech synthesis, and recognizing the emotional vector representation of the target text; performing emotion style prediction on the emotion vector representation to obtain an emotion style weight; and based on a pre-trained acoustic model, determining a phoneme embedding sequence of the target text, and controlling the acoustic model to adjust the phoneme embedding sequence by using the emotional style weight, so that the acoustic model generates emotional speech of the target text by using the adjusted phoneme embedding sequence. Even under the condition that no professional audio material or emotion label exists, voice output with high emotion expressive power can be obtained, and the generalization ability of the model is improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence and speech synthesis technology, and in particular to an emotional speech synthesis method, device, computer equipment and readable storage medium. Background Art

[0002] As an important innovation in the field of artificial intelligence, emotional speech synthesis technology is gradually showing its wide application prospects in multiple industries, such as in the medical field and the financial field. In medical scenarios, this technology gives emotional colors to personalized virtual assistants, making them more friendly and humane when interacting with patients, greatly improving patients' medical experience. At the same time, emotional speech synthesis technology can also be applied to emotional artificial intelligence therapists. By simulating the emotional expression of therapists, it can provide patients with a more emotionally supportive conversation experience, thereby enhancing the treatment effect.

[0003] In the related art, the emotional speech synthesis framework relies on manually annotated emotion tags, so that when synthesizing emotional speech, it is necessary to input emotion tags for the text of the speech to be generated, so that the emotional speech synthesis model can synthesize emotional speech according to the input emotion tags. However, the applicant recognizes that the accuracy of the emotion tags annotated when training the model is often affected by the subjectivity of the annotator, and there may be significant differences between different annotators. In addition, the reliance on tags limits the generalization ability of the model, so that the trained emotional speech synthesis model cannot accurately identify emotions when faced with unseen emotion tags or emotion expression tags in natural conversations, and it is difficult to generate natural and realistic emotional speech. Summary of the invention

[0004] The present invention provides an emotional speech synthesis method, apparatus, computer equipment and readable storage medium to solve the technical problem that the generalization ability of the model is limited, so that the trained emotional speech synthesis model cannot accurately recognize emotions when faced with unseen emotional labels or emotional expression labels in natural conversations, and it is difficult to generate natural and realistic emotional speech.

[0005] In a first aspect, a method for synthesizing emotional speech is provided, comprising:

[0006] Acquire a target text to be subjected to speech synthesis, and identify a sentiment vector representation of the target text;

[0007] Performing emotional style prediction on the emotional vector representation to obtain an emotional style weight;

[0008] Based on the pre-trained acoustic model, the phoneme embedding sequence of the target text is determined, and the acoustic model is controlled to adjust the phoneme embedding sequence using the emotional style weight, so that the acoustic model generates the emotional speech of the target text using the adjusted phoneme embedding sequence.

[0009] In a second aspect, an emotional speech synthesis device is provided, comprising:

[0010] A recognition module is used to obtain a target text to be subjected to speech synthesis and recognize a sentiment vector representation of the target text;

[0011] A prediction module, used to predict the emotional style of the emotional vector representation to obtain the emotional style weight;

[0012] A generation module is used to determine the phoneme embedding sequence of the target text based on a pre-trained acoustic model, and control the acoustic model to adjust the phoneme embedding sequence using the emotional style weight so that the acoustic model generates the emotional speech of the target text using the adjusted phoneme embedding sequence.

[0013] In a third aspect, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned emotional speech synthesis method when executing the computer program.

[0014] In a fourth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned emotional speech synthesis method are implemented.

[0015] In the scheme implemented by the above-mentioned emotional speech synthesis method, device, computer equipment and readable storage medium, the target text to be speech synthesized is obtained, the emotional vector representation of the target text is identified, the emotional style prediction is performed on the emotional vector representation, and the emotional style weight is obtained. Based on the pre-trained acoustic model, the phoneme embedding sequence of the target text is determined, and the acoustic model is controlled to adjust the phoneme embedding sequence using the emotional style weight, so that the acoustic model uses the adjusted phoneme embedding sequence to generate the emotional speech of the target text. The process of synthesizing emotional speech does not require any additional reference audio or emotional labels as an aid. Even in the absence of professional audio materials or emotional annotations, speech output with high emotional expressiveness can be obtained, which improves the generalization ability of the model, so that the trained emotional speech synthesis model can accurately identify emotions and generate natural and realistic emotional speech when faced with unseen emotional labels or emotional expression labels in natural conversations. Regardless of the scenario, speech synthesis can provide strong technical support. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0017] Figure 1 is a schematic diagram of an application environment of an emotional speech synthesis method in one embodiment of the present invention;

[0018] Figure 2 is a flow chart of an emotional speech synthesis method according to an embodiment of the present invention;

[0019] Figure 3 It is a flowchart of a specific implementation method of the emotional speech synthesis method;

[0020] Figure 4 It is a flowchart of a specific implementation method of the emotional speech synthesis method;

[0021] Figure 5 It is a flowchart of a specific implementation method of the emotional speech synthesis method;

[0022] Figure 6 It is a flowchart of a specific implementation method of the emotional speech synthesis method;

[0023] Figure 7 It is a flowchart of a specific implementation method of the emotional speech synthesis method;

[0024] Figure 8 It is a flowchart of a specific implementation method of the emotional speech synthesis method;

[0025] Fig. 9 It is a flowchart of a specific implementation method of the emotional speech synthesis method;

[0026] Fig.10 is a structural schematic diagram of an emotional speech synthesis device in one embodiment of the present invention;

[0027] Fig.11 is a schematic diagram of a structure of a computer device in one embodiment of the present invention;

[0028] Fig.12 It is another structural schematic diagram of a computer device in one embodiment of the present invention. DETAILED DESCRIPTION

[0029] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0030] The emotional speech synthesis method provided by the embodiment of the present invention can be applied in Figure 1 In the application environment, the client communicates with the server through the network. The server can obtain the target text to be synthesized, identify the emotional vector representation of the target text, predict the emotional style of the emotional vector representation, obtain the emotional style weight, determine the phoneme embedding sequence of the target text based on the pre-trained acoustic model, and control the acoustic model to adjust the phoneme embedding sequence using the emotional style weight, so that the acoustic model generates the emotional speech of the target text using the adjusted phoneme embedding sequence, and feeds the generated emotional speech back to the client. In the present invention, for medical scenarios, the process of synthesizing emotional speech does not require any additional reference audio or emotional tags as an aid, and even in the absence of professional audio materials or emotional annotations, a speech output with high emotional expression can be obtained, which not only reduces the threshold of emotional speech synthesis, but also greatly broadens its application scenarios. Whether it is a warm reminder of medical intelligent customer service, or the explanation of products by financial software, or even the personalized dubbing of entertainment products, it can provide strong technical support. Among them, the client can be but not limited to various personal computers, laptops, smart phones, tablet computers and portable wearable devices. The server can be implemented with an independent server or a server cluster composed of multiple servers. The present invention is described in detail below through specific embodiments.

[0031] In order to realize the synthesis of emotional speech, a style adaptation model and an acoustic model are constructed in the embodiment of the present application. The training process of these two models is described below. Figure 2 As shown, Figure 2 A schematic diagram of a model training process of the emotional speech synthesis method provided by an embodiment of the present invention includes the following steps:

[0032] S10: Obtain multiple sample texts.

[0033] In the embodiment of the present application, in order to train the model, the server collects multiple sample texts, wherein each sample text corresponds to a sample voice and a sample style weight, the sample voice is the recording corresponding to the sample text, and the sample style weight reflects the expression style strength required by the sample text, such as gentle, serious or encouraging.

[0034] In a medical scenario, the collected sample texts can be doctors' diagnosis instructions and rehabilitation suggestions for patients. For a sample text about postoperative rehabilitation suggestions, if you want the voice to sound more inspiring, you can assign it a higher encouraging style weight. This provides rich and clearly labeled data resources for subsequent model training, which helps the model learn speech features in different styles.

[0035] S20: extracting a target sample text from a plurality of sample texts.

[0036] In the embodiment of the present application, the server will use the fine-tuned DistilROBERTa language model to predict the probability of the sentiment category of the text, and select text-speech pairs with obvious sentiment tendencies as target sample texts, so that the model trained using the target sample texts will be more accurate. Figure 3 As shown, extracting a target sample text from a plurality of sample texts includes the following steps S21 to S22:

[0037] S21: Use the fine-tuned pre-trained language model to predict the probability of the sentiment category corresponding to each sample text.

[0038] In an embodiment of the present application, the server first uses a fine-tuned DistilBERTa language model to predict the probability of sentiment categories for a series of sample texts. DistilBERTa is a lightweight variant of the BERT model that reduces the size and computational complexity of the model through knowledge distillation technology while maintaining most of the performance of the original BERT model. Fine-tuning refers to training the pre-trained DistilBERTa model on a specific sentiment classification task dataset so that it can better identify and classify the sentiment tendency of a given text.

[0039] In specific implementation, the server inputs each sample text into the fine-tuned DistilBERTa model. For example, the sample text may include patient comments such as "I am very satisfied with the treatment effect this time" or "I feel very painful and uncomfortable after the operation". The fine-tuned DistilBERTa model analyzes the words, phrases, and contextual information in these texts, and then outputs the probability distribution of each text belonging to different sentiment categories (such as positive, negative, and neutral).

[0040] Through the fine-tuned DistilBERTa model, the server can more accurately capture the unique emotional expressions in medical texts, such as patients' satisfaction with treatment effects and their feelings about pain levels, and efficiently obtain the emotional category probability of each sample text, providing reliable data support for subsequent emotional screening and model training. At the same time, due to the lightweight nature of the DistilBERTa model, this process is also highly feasible and practical in resource-constrained environments.

[0041] S22: Obtain a preset probability threshold, and extract sample texts whose corresponding emotion category probabilities are greater than the probability threshold from multiple sample texts as target sample texts.

[0042] In the embodiment of the present application, a preset probability threshold is set in the server. After obtaining the emotion category probability of each sample text, the server will obtain the preset probability threshold, and extract the sample text whose corresponding emotion category probability is greater than the probability threshold from multiple sample texts as the target sample text, so as to screen out the text with clear emotion expression as the target sample text. Among them, the preset probability threshold is set according to the actual application needs and the requirements for emotion classification accuracy, and can be determined through experience or experiments.

[0043] Taking the medical scenario as an example, the server may set a high probability threshold (such as 0.8 or higher) to ensure that the screened text has a strong and clear emotional tendency. In this way, only those patient feedbacks that the model judges to be highly positive or highly negative will be selected as target sample texts. For example, if the model predicts that the sentence "I am very satisfied with the treatment effect this time" has a probability of 0.95 belonging to the positive emotion category, then this sentence will be selected as the target sample text because it exceeds the preset probability threshold. On the contrary, if the emotion category probability of a sentence is relatively scattered or low, it indicates that its emotional expression is not clear enough, so it will not be selected. In this way, by setting a reasonable probability threshold and screening out sample texts with emotion category probabilities higher than the threshold, the server can effectively reduce noise data and improve the accuracy and efficiency of subsequent model training. In the medical scenario, the server can focus more on those texts that have clear indications of the patient's emotional state, thereby training a model that is better able to understand and respond to the patient's emotional needs.

[0044] S30. Use the fine-tuned pre-trained language model to extract a sample sentiment vector of the target sample text, and use the sample sentiment vector and the target sample style weight corresponding to the target sample text to train a style adaptation model.

[0045] In the embodiment of the present application, the server uses the fine-tuned pre-trained language model to analyze the target sample text and extract the sentiment vector that can represent the sentiment expression of the target sample text. The purpose of fine-tuning the pre-trained language model is to adapt it to the sentiment analysis task of medical text. The fine-tuned pre-trained language model is consistent with that mentioned in the above step S21 and will not be described here.

[0046] Subsequently, the emotion vector and style weight are used as input through the style adaptation model framework, and the training model outputs stylized emotion features. Among them, the style adaptation model framework can be a GST network. In this way, a style adaptation model is trained in combination with the style weight corresponding to the text. The style adaptation model can learn how to combine the emotion vector with the style weight to generate voice instructions with a specific emotional style and intensity. For example, for a text encouraging children to take medicine on time, the style adaptation model will adjust the tone, rhythm and other parameters of the voice according to its emotion vector and the "encouragement" style weight to make it sound more positive.

[0047] In this way, by introducing emotion vectors and style weights, the trained style adaptation model can generate more delicate speech that meets the needs of medical scenarios, enhancing the emotional expression ability and personalized characteristics of speech synthesis. Figure 4 As shown, the process of training the style adaptation model includes the following steps S31 to S32:

[0048] S31. Use the fine-tuned pre-trained language model to extract the sample sentiment vector of the target sample text, take the sample sentiment vector as input and the target sample style weight as the target value, use a multi-category classification task to train the model, and obtain the initial adaptation model.

[0049] In the embodiment of the present application, the server first uses a fine-tuned pre-trained language model to extract the sentiment vector of the target sample text. The fine-tuned pre-trained language model can deeply understand the semantics and sentiment of the text, thereby accurately extracting the sample sentiment vector. Among them, the fine-tuned pre-trained language model is consistent with that mentioned in the above step S21, and will not be repeated here.

[0050] Next, the server takes the extracted sample sentiment vector as input and the target sample style weight as the target value. The target sample style weight (also known as GST weight) is defined according to actual application requirements, and it represents the weight distribution of different sentiment styles, such as optimism, pessimism, neutrality, etc. In medical scenarios, these style weights correspond to different expectations and attitudes of patients or doctors towards treatment effects.

[0051] To train the model, the server uses a multi-class classification task so that the model learns to classify the input sample sentiment vector into one of the predefined GST weight categories. Through training, the model gradually establishes a mapping relationship between sentiment vectors and GST weights, thereby obtaining an initial adaptation model.

[0052] Taking the medical scenario as an example, suppose the server has a set of texts about cancer treatment, which contain the emotional expressions of patients. The server uses a fine-tuned pre-trained language model to extract the sentiment vectors of these texts and matches them with the target sample style weights (such as optimism and pessimism). Through training, the model can learn to identify the style weights represented by different sentiment vectors, thus providing a basis for subsequent sentiment style adaptation.

[0053] S32. Calculate the loss value of the initial adaptation model using the classification cross entropy loss function, and adjust the model parameters of the initial adaptation model. Continue to train the adjusted initial adaptation model using the sample sentiment vector as input and the target sample style weight as the target value, and recalculate the loss value until the calculated loss value reaches the first preset condition. Use the initial adaptation model that has completed the model parameter adjustment as the style adaptation model.

[0054] After obtaining the initial adaptation model, the server needs to further optimize the performance of the model. To this end, the server uses the classification cross entropy loss function to calculate the loss value of the model. The classification cross entropy loss function is a commonly used loss function that can measure the difference between the probability distribution predicted by the model and the true label distribution. The specific function formula is as follows:

[0055] Formula 1: L = -∑i = 1Ny i log(p i )

[0056] Among them, L represents the calculated loss value; N represents the total number of sample sentiment vectors; yi represents the target sample style weight corresponding to the i-th target sample text; pi represents the predicted sample style weight predicted by the initial adaptation model for the sample sentiment vector of the i-th target sample text. In this way, through the above formula 1, the server calculates the predicted sample style weight p of each target sample text i and the target sample style weight y i The cross entropy loss between them is calculated, and then the loss values ​​of all samples are added together to get the total loss (that is, the loss value). The total loss value reflects the current mastery of the model for the emotional style adaptation task.

[0057] In order to reduce the loss value, the server will use the back-propagation algorithm to calculate the gradient of the loss value to the model parameters, and use optimization algorithms (such as Adam, SGD, etc.) to adjust the model parameters. The adjusted model will be trained again, using the same sample sentiment vector as input and the target sample style weight as the target value. During the training process, the server continuously calculates the loss value and adjusts the model parameters to minimize the loss value.

[0058] This process will continue until the calculated loss value reaches the first preset condition. The first preset condition can be a smaller threshold or an upper limit of the number of training rounds, which indicates that the model has been fully trained and has reached the expected performance level. When the first preset condition is met, the server uses the initial adaptation model that has completed the model parameter adjustment as the style adaptation model.

[0059] S40: Train an acoustic model using the multiple sample texts and the sample speech corresponding to the multiple sample texts.

[0060] In the embodiment of the present application, after obtaining rich sample text and its corresponding voice data, the server will use these data to train the acoustic model, which is responsible for converting the text into the corresponding voice waveform and generating high-quality voice output. Figure 5 As shown, it includes the following steps S41 to S43:

[0061] S41: using a preset reference encoder, a preset text encoder, a preset variational adapter and a decoder, processing each sample text and the sample speech corresponding to each sample text, and generating a predicted mel-spectrogram for the sample speech corresponding to each sample text.

[0062] In the embodiment of the present application, the server prepares a complete processing framework in advance, which includes a preset reference encoder, a preset text encoder, a preset variational adapter and a decoder. These components work together to convert the sample text and its corresponding sample speech into a mel-spectrogram, an acoustic feature representation. When generating a predicted mel-spectrogram for the sample speech corresponding to each sample text, Figure 6 As shown, it includes the following steps S411 to S414:

[0063] S411: For each sample text, perform phoneme conversion on the sample text to obtain a sample phoneme sequence of the sample text.

[0064] In the embodiment of the present application, the server first processes the sample text and converts it into a phoneme sequence. Phonemes are the smallest units that make up language sounds. Through phoneme conversion, the server can convert text information into a lower-level acoustic representation, providing a basis for subsequent steps.

[0065] Specifically, the server can use text analysis software or algorithms to preprocess the sample text, including word segmentation, part-of-speech tagging, etc., and then use a phoneme dictionary or phoneme recognition model to convert each word or syllable in the text into a corresponding phoneme, and finally obtain a sample phoneme sequence of the sample text.

[0066] Taking the medical scenario as an example, suppose the server has a set of medical records about heart disease diagnosis. Through the processing of this step, the server can convert these texts into phoneme sequences, providing basic data for the subsequent generation of acoustic features.

[0067] S412: Encode the sample speech corresponding to the sample text using a preset reference encoder to obtain a sample style vector of the sample speech, and adjust the preset text encoder using the sample style vector.

[0068] In an embodiment of the present application, the server uses a preset reference encoder to encode the sample speech corresponding to the sample text and extracts the style features therein. The style features reflect the personalized characteristics of the speech, such as speaking speed, intonation, pronunciation habits, etc.

[0069] Specifically, the preset reference encoder is a pre-trained neural network model that can extract and encode the input speech signal and output a vector representing the speech style. The server then uses this sample style vector to adjust the preset text encoder to better adapt to the style characteristics of the current speech sample.

[0070] For example, when processing a recording of a conversation between a doctor and a patient, the voice information in the recording can be used to generate a sample style vector and use it to adjust the text encoder to more accurately capture the voice characteristics of the doctor when expressing medical terms or diagnostic opinions.

[0071] S413: Using the adjusted preset text encoder, convert the sample phoneme sequence into a sample hidden sequence.

[0072] In the embodiment of the present application, the server uses the adjusted preset text encoder to convert the obtained sample phoneme sequence into a sample hidden sequence. The hidden sequence is the internal representation obtained after the text encoder encodes the input phoneme sequence. It is a high-dimensional, abstract feature representation that contains comprehensive information about the sample text and the sample voice style.

[0073] Specifically, the preset text encoder is a neural network model that can encode the input phoneme sequence and output a hidden sequence representing text information. When encoding, it is necessary to ensure the continuity and smoothness of the transformation to retain the key features of the original information. Through the adjusted preset text encoder, the server can better capture the deep features in the text information and provide stronger support for the subsequent generation of acoustic features.

[0074] S414: using a preset variational adapter to adjust the sample hidden sequence, and using a decoder to convert the adjusted sample hidden sequence into a Mel-spectrogram to obtain a predicted Mel-spectrogram corresponding to the sample text.

[0075] In the embodiment of the present application, the server further adjusts and optimizes the sample hidden sequence. Specifically, the server uses a preset variational adapter to transform the sample hidden sequence to generate a representation that is more consistent with the target acoustic features. The variational adapter has nonlinear transformation capabilities and can flexibly adjust the feature distribution and interrelationships in the hidden sequence.

[0076] Subsequently, the server uses the decoder to convert the adjusted sample hidden sequence into a Mel-spectrogram. Mel-spectrogram is a commonly used acoustic feature representation method that reflects the acoustic characteristics of an audio signal by converting it into an energy distribution over a series of frequency bands. Through the conversion process of the decoder, the server can obtain the predicted Mel-spectrogram corresponding to the sample text.

[0077] Taking the medical scenario as an example, assuming there is a set of sample texts about heart disease diagnosis and corresponding doctor voice samples, a predicted mel-spectrogram corresponding to its content can be generated for each voice sample, providing acoustic feature support for subsequent analysis and diagnosis.

[0078] S42: Obtain a sample Mel-spectrogram associated with the sample speech corresponding to each sample text, and calculate a mean square error loss value between the predicted Mel-spectrogram generated for each sample text and its associated sample Mel-spectrogram.

[0079] In order to evaluate the accuracy of the predicted mel-spectrogram, the server needs to obtain the real sample mel-spectrogram (also known as the sample mel-spectrogram) corresponding to each sample text. Subsequently, the server calculates the mean square error (MSE) loss value between the predicted mel-spectrogram and the real sample mel-spectrogram to quantify the difference between the two. The smaller the MSE loss value, the closer the predicted mel-spectrogram is to the real sample mel-spectrogram, that is, the better the prediction performance of the model. In this way, by comparing the MSE loss values ​​of the predicted mel-spectrogram and the real sample mel-spectrogram, the server can evaluate the model performance and optimize the model.

[0080] S43: According to the calculated mean square error loss value, the preset text encoder, the preset variational adapter and the decoder are adjusted, and the adjusted preset text encoder, the adjusted preset variational adapter and the adjusted decoder are used to regenerate a predicted Mel-spectrogram for each sample speech and calculate a new mean square error loss value, and the preset text encoder, the preset variational adapter and the decoder are continued to be adjusted according to the new mean square error loss value until the calculated mean square error loss value reaches the preset second condition, and the currently adjusted preset text encoder, the preset variational adapter and the decoder are integrated to obtain an acoustic training model.

[0081] After obtaining the mean square error loss value, the server uses the back propagation algorithm to adjust the parameters of the preset text encoder, preset variational adapter and decoder. Through iterative optimization, the server continuously reduces the difference between the predicted mel spectrogram and the real sample mel spectrogram until the MSE loss value reaches the second preset condition. The setting strategy of the second preset condition is consistent with the setting strategy of the first preset condition mentioned above, and will not be repeated here.

[0082] Finally, the server integrates the adjusted preset text encoder, preset variational adapter, and decoder to form a complete acoustic model, in which the preset text encoder converts the phoneme embedding sequence into a hidden sequence, the preset variational adapter adjusts the hidden sequence to control rhythmic features such as pitch, duration, and energy, and the decoder converts the adjusted hidden sequence into a mel-spectrogram. In actual application, FastSpeech2 can be used as the basic framework to train the acoustic model.

[0083] The above training model does not involve the annotation of emotional labels. The training process does not rely on any emotional labels, but uses the emotional information of the text content itself to learn emotional expression. Moreover, based on the text emotion perception of the language model, the pre-trained BERT language model is used to convert the text content into emotional embedding, thereby establishing a connection between text emotion and speech rhythm. After completing the training of the required model, the server can use the trained model to realize the synthesis of emotional speech. Figure 7 As shown, the following steps S50 to S70 are included:

[0084] S50: Acquire the target text to be subjected to speech synthesis, and identify the sentiment vector representation of the target text.

[0085] In an embodiment of the present application, the server obtains the target text to be speech synthesized, uses a fine-tuned pre-trained language model to perform sentiment analysis on the target text, and identifies the sentiment vector representation of the target text. The fine-tuned pre-trained language model is described in the above content and will not be repeated here. Among them, in medical scenarios, the target text can be doctor's orders, patient comfort words, etc.; the sentiment vector is a multidimensional vector that can capture the emotional features in the text, such as positive, negative, comforting, nervous, etc. For example, when identifying the doctor's order text "Please rest assured, your condition is getting better", the server can capture the positive and comforting emotions therein and generate the corresponding sentiment vector. In this way, through the recognition of the sentiment vector, a basis is provided for the subsequent prediction of the sentiment style, so that the generated speech can be closer to the real emotional expression of humans.

[0086] S60: Predict the emotional style of the emotional vector representation to obtain the emotional style weight.

[0087] In an embodiment of the present application, after obtaining the emotion vector, the server will use the style adaptation model trained in the above process to further analyze the emotion vector and predict its emotion style weight. The emotion style weight reflects the proportion of different emotions in the synthesized speech and determines the emotional tendency of the final speech. For example, when processing the soothing text "Please stay calm, we will arrange surgery for you as soon as possible", the emotion style prediction can ensure that the generated speech has enough soothing and calming emotions to reduce the patient's tension. In this way, through the introduction of emotion style weights, the server can more finely control the emotional expression of the synthesized speech, thereby improving the applicability of speech synthesis in various scenarios.

[0088] S70: Based on the pre-trained acoustic model, determine the phoneme embedding sequence of the target text, and control the acoustic model to adjust the phoneme embedding sequence using the emotional style weight, so that the acoustic model generates the emotional speech of the target text using the adjusted phoneme embedding sequence.

[0089] In the embodiment of the present application, after obtaining the emotional style weight, the server determines the phoneme embedding sequence of the target text based on the pre-trained acoustic model, and controls the acoustic model to adjust the phoneme embedding sequence using the emotional style weight, so that the acoustic model generates the emotional speech of the target text using the adjusted phoneme embedding sequence. Specifically, when generating the emotional speech of the target text, Figure 8 As shown, it includes the following steps S71 to S74:

[0090] S71: Perform phoneme conversion on the target text to obtain a target phoneme sequence of the target text.

[0091] First, the server converts the target text into a phoneme sequence. Phonemes are the smallest unit of speech. Through phoneme conversion, text content can be converted into a format that can be processed by the acoustic model. For example, when converting "Please take a rest after surgery" into a phoneme sequence, the server will decompose each Chinese character into the corresponding pinyin phoneme.

[0092] S72: Input the target phoneme sequence into the acoustic model so that a preset text encoder in the acoustic model converts the target phoneme sequence into a target hidden sequence.

[0093] Next, the server inputs the target phoneme sequence into the pre-trained acoustic model. The preset text encoder in the acoustic model can convert the target phoneme sequence into a target hidden sequence, which contains the acoustic features of the speech.

[0094] S73: Based on a preset variational adapter in the acoustic model, adjusting the target hidden sequence, and converting the adjusted target hidden sequence into a target mel-spectrogram based on a decoder in the acoustic model.

[0095] After obtaining the target hidden sequence, the server uses the preset variational adapter in the acoustic model to adjust the hidden sequence according to the emotional style weight, aiming to integrate emotional features into acoustic features, thereby generating emotional speech. After adjustment, the target hidden sequence is converted into a Mel spectrogram through the decoder of the acoustic model. The Mel spectrogram is a frequency domain representation that can reflect the spectral characteristics of speech.

[0096] S74: calling a vocoder to convert the target mel-spectrogram, and using the converted speech as the emotional speech of the target text.

[0097] Finally, the server calls a vocoder (such as the Griffin-Lim algorithm, WaveGlow, etc.) to convert the mel-spectrogram and generate the final speech output. The vocoder can restore the mel-spectrogram to a time domain signal, that is, speech that can be heard by humans. In this process, since emotional features have been incorporated, the generated speech will have corresponding emotional colors. For example, when processing the good news "The operation was successful, congratulations!", the server can generate a speech with joy to convey it to the patient and his family.

[0098] In summary, the logical process of the technical solution of this application is summarized as follows: Fig. 9, the processing process of this application is divided into stage 1 and stage 2. Among them, in stage 1, first, select suitable emotional data from the available emotional data set, and the emotional data contains text and corresponding speech under different emotional states. Then, perform MFA alignment operation on the text content in the emotional data to obtain the phoneme vector of the text, and embed it into the phoneme embedding space through the text encoder. The GST network uses the reference encoder to encode the speech corresponding to the text into a style embedding, and uses the style embedding to adjust the text encoder so that the model can generate different speech styles. The text encoder converts the phoneme embedding sequence into a hidden sequence, the variational adapter adjusts the hidden sequence to control the rhythmic features such as pitch, duration and energy, and the decoder converts the adjusted hidden sequence into a mel spectrogram. In stage two, select suitable emotional data from the available emotional data set, and the emotional data contains text and corresponding speech under different emotional states. On the one hand, the selected emotional speech is input into the reference encoder, and the speech encoding output by the reference encoder is transmitted to the GST network, so that the GST network uses the reference encoder to encode the speech corresponding to the text into a style embedding and perform its subsequent operations. On the other hand, the selected sentiment text is input into the language model to generate the corresponding sentiment text vector. The sentiment text vector is used as input, and the GST weight is used as the target value. The adaptation module is trained using a multi-category classification task to map the sentiment embedding into the sentiment style space, realizing the style space exploration guided by the language model.

[0099] It can be seen that in the above scheme, the target text to be speech synthesized is obtained, the emotion vector representation of the target text is identified, the emotion style prediction is performed on the emotion vector representation, and the emotion style weight is obtained. Based on the pre-trained acoustic model, the phoneme embedding sequence of the target text is determined, and the acoustic model is controlled to adjust the phoneme embedding sequence using the emotion style weight, so that the acoustic model uses the adjusted phoneme embedding sequence to generate the emotional speech of the target text. The process of synthesizing emotional speech does not require any additional reference audio or emotion labels as an aid. Even in the absence of professional audio materials or emotion annotations, speech output with high emotional expression can be obtained, which improves the generalization ability of the model, so that the trained emotional speech synthesis model can accurately identify emotions and generate natural and realistic emotional speech when faced with unseen emotion labels or emotion expression labels in natural conversations. Regardless of the scenario, speech synthesis can provide strong technical support.

[0100] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0101] In one embodiment, an emotional speech synthesis device is provided, and the emotional speech synthesis device corresponds to the emotional speech synthesis method in the above embodiment. Fig.10 As shown, the emotional speech synthesis device includes a recognition module 101, a prediction module 102 and a generation module 103. The functional modules are described in detail as follows:

[0102] The recognition module 101 is used to obtain a target text to be subjected to speech synthesis and recognize a sentiment vector representation of the target text;

[0103] The prediction module 102 is used to predict the emotional style of the emotional vector representation to obtain the emotional style weight;

[0104] The generation module 103 is used to determine the phoneme embedding sequence of the target text based on the pre-trained acoustic model, and control the acoustic model to adjust the phoneme embedding sequence using the emotional style weight so that the acoustic model generates the emotional speech of the target text using the adjusted phoneme embedding sequence.

[0105] In one embodiment, the emotional speech synthesis device further includes:

[0106] A training module is used to obtain a plurality of sample texts, wherein each of the sample texts corresponds to a sample speech and a sample style weight; extract a target sample text from the plurality of sample texts, extract a sample emotion vector of the target sample text using a fine-tuned pre-trained language model, and train a style adaptation model using the sample emotion vector and the target sample style weight corresponding to the target sample text; and train an acoustic model using the plurality of sample texts and the sample speech corresponding to the plurality of sample texts;

[0107] Accordingly, the recognition module 101 is used to recognize the sentiment vector representation of the target text using the fine-tuned pre-trained language model;

[0108] Accordingly, the prediction module 102 is used to predict the emotional style weight represented by the emotional vector by using the style adaptation model.

[0109] In one embodiment, the training module is used to use the fine-tuned pre-trained language model to predict the probability of the emotion category corresponding to each of the sample texts; obtain a preset probability threshold, and extract the sample text whose corresponding emotion category probability is greater than the probability threshold from the multiple sample texts as the target sample text.

[0110] In one embodiment, the training module is used to use the fine-tuned pre-trained language model to extract the sample sentiment vector of the target sample text, take the sample sentiment vector as input and the target sample style weight as the target value, use a multi-category classification task to perform model training, and obtain an initial adaptation model; use the classification cross entropy loss function to calculate the loss value of the initial adaptation model, and adjust the model parameters of the initial adaptation model, continue to use the sample sentiment vector as input and the target sample style weight as the target value to train the adjusted initial adaptation model, and recalculate the loss value until the calculated loss value reaches the first preset condition, and use the initial adaptation model that has completed the model parameter adjustment as the style adaptation model.

[0111] In one embodiment, the training module is used to use a preset reference encoder, a preset text encoder, a preset variational adapter and a decoder to process each of the sample texts and the sample speech corresponding to each of the sample texts, and generate a predicted Mel-spectrogram for the sample speech corresponding to each of the sample texts; obtain a sample Mel-spectrogram associated with the sample speech corresponding to each of the sample texts, and calculate a mean square error loss value between the predicted Mel-spectrogram generated for each of the sample texts and the sample Mel-spectrogram associated with it; adjust the preset text encoder, the preset variational adapter and the decoder according to the calculated mean square error loss value, and use the adjusted preset text encoder, the adjusted preset variational adapter and the adjusted decoder to regenerate a predicted Mel-spectrogram for each of the sample speech and calculate a new mean square error loss value, continue to adjust the preset text encoder, the preset variational adapter and the decoder according to the new mean square error loss value until the calculated mean square error loss value reaches a second preset condition, and integrate the currently adjusted preset text encoder, the preset variational adapter and the decoder to obtain the acoustic model.

[0112] In one embodiment, the training module is used to perform phoneme conversion on each of the sample texts to obtain a sample phoneme sequence of the sample text; use the preset reference encoder to encode the sample speech corresponding to the sample text to obtain a sample style vector of the sample speech, and use the sample style vector to adjust the preset text encoder; use the adjusted preset text encoder to convert the sample phoneme sequence into a sample hidden sequence; use a preset variational adapter to adjust the sample hidden sequence, and use the decoder to convert the adjusted sample hidden sequence into a Mel-spectrogram to obtain a predicted Mel-spectrogram corresponding to the sample text.

[0113] In one embodiment, the generation module 103 is used to perform phoneme conversion on the target text to obtain a target phoneme sequence of the target text; input the target phoneme sequence into the acoustic model so that the preset text encoder in the acoustic model converts the target phoneme sequence into a target hidden sequence; based on the preset variational adapter in the acoustic model, adjust the target hidden sequence, and based on the decoder in the acoustic model, convert the adjusted target hidden sequence into a target Mel-spectrogram; call the vocoder to convert the target Mel-spectrogram, and use the converted speech as the emotional speech of the target text.

[0114] The present invention provides an emotional speech synthesis device, which obtains a target text to be speech synthesized, identifies an emotional vector representation of the target text, performs emotional style prediction on the emotional vector representation, obtains an emotional style weight, determines a phoneme embedding sequence of the target text based on a pre-trained acoustic model, and controls the acoustic model to adjust the phoneme embedding sequence using the emotional style weight, so that the acoustic model generates the emotional speech of the target text using the adjusted phoneme embedding sequence. The process of synthesizing the emotional speech does not require any additional reference audio or emotional label as an aid, and even in the absence of professional audio materials or emotional annotations, speech output with high emotional expression can be obtained, thereby improving the generalization ability of the model, so that the trained emotional speech synthesis model can accurately identify emotions and generate natural and realistic emotional speech when faced with unseen emotional labels or emotional expression labels in natural conversations, thereby providing strong technical support for speech synthesis in any scenario.

[0115] For the specific definition of the emotional speech synthesis device, please refer to the definition of the emotional speech synthesis method above, which will not be repeated here. The various modules in the above-mentioned emotional speech synthesis device can be implemented in whole or in part by software, hardware and their combination. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0116] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Fig.11As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external client through a network connection. When the computer program is executed by the processor, it implements the functions or steps on the server side of an emotional speech synthesis method.

[0117] In one embodiment, a computer device is provided. The computer device may be a client, and its internal structure diagram may be as follows: Fig.12 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it implements the functions or steps of the client side of an emotional speech synthesis method

[0118] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:

[0119] Acquire a target text to be subjected to speech synthesis, and identify a sentiment vector representation of the target text;

[0120] Performing emotional style prediction on the emotional vector representation to obtain an emotional style weight;

[0121] Based on the pre-trained acoustic model, the phoneme embedding sequence of the target text is determined, and the acoustic model is controlled to adjust the phoneme embedding sequence using the emotional style weight, so that the acoustic model generates the emotional speech of the target text using the adjusted phoneme embedding sequence.

[0122] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0123] Acquire a target text to be subjected to speech synthesis, and identify a sentiment vector representation of the target text;

[0124] Performing emotional style prediction on the emotional vector representation to obtain an emotional style weight;

[0125] Based on the pre-trained acoustic model, the phoneme embedding sequence of the target text is determined, and the acoustic model is controlled to adjust the phoneme embedding sequence using the emotional style weight, so that the acoustic model generates the emotional speech of the target text using the adjusted phoneme embedding sequence.

[0126] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can refer to the relevant descriptions on the server side and the client side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0127] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0128] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0129] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. An emotional speech synthesis method, characterized in that: include: Acquire a target text to be subjected to speech synthesis, and identify a sentiment vector representation of the target text; Performing emotional style prediction on the emotional vector representation to obtain an emotional style weight; Based on the pre-trained acoustic model, the phoneme embedding sequence of the target text is determined, and the acoustic model is controlled to adjust the phoneme embedding sequence using the emotional style weight, so that the acoustic model generates the emotional speech of the target text using the adjusted phoneme embedding sequence.

2. The method according to claim 1, characterized in that: Before acquiring the target text to be subjected to speech synthesis and identifying the emotion vector representation of the target text, the method further includes: Acquire multiple sample texts, wherein each of the sample texts corresponds to a sample voice and a sample style weight; Extracting a target sample text from the multiple sample texts, extracting a sample sentiment vector of the target sample text using a fine-tuned pre-trained language model, and training a style adaptation model using the sample sentiment vector and a target sample style weight corresponding to the target sample text; Training an acoustic model using the multiple sample texts and sample speech corresponding to the multiple sample texts; Accordingly, the identifying the sentiment vector representation of the target text includes: using the fine-tuned pre-trained language model to identify the sentiment vector representation of the target text; Correspondingly, the performing emotional style prediction on the emotional vector representation to obtain the emotional style weight includes: using the style adaptation model to predict the emotional style weight of the emotional vector representation.

3. The method according to claim 2, characterized in that The step of extracting a target sample text from the plurality of sample texts comprises: Using the fine-tuned pre-trained language model, predicting the probability of the sentiment category corresponding to each of the sample texts; A preset probability threshold is obtained, and sample texts whose corresponding emotion category probabilities are greater than the probability threshold are extracted from the multiple sample texts as the target sample texts.

4. The method according to claim 2, characterized in that: The method of extracting a sample sentiment vector of the target sample text using a fine-tuned pre-trained language model and training a style adaptation model using the sample sentiment vector and a target sample style weight corresponding to the target sample text includes: Extracting a sample sentiment vector of the target sample text using the fine-tuned pre-trained language model, taking the sample sentiment vector as input and the target sample style weight as a target value, and performing model training using a multi-category classification task to obtain an initial adaptation model; The classification cross entropy loss function is used to calculate the loss value of the initial adaptation model, and the model parameters of the initial adaptation model are adjusted. The sample emotion vector is continued to be used as input and the target sample style weight is used as the target value to train the adjusted initial adaptation model, and the loss value is recalculated until the calculated loss value reaches the first preset condition. The initial adaptation model that has completed the model parameter adjustment is used as the style adaptation model.

5. The method according to claim 2, characterized in that: The method of training an acoustic model by using the plurality of sample texts and sample speech corresponding to the plurality of sample texts includes: Using a preset reference encoder, a preset text encoder, a preset variational adapter and a decoder, each of the sample texts and the sample speech corresponding to each of the sample texts are processed to generate a predicted mel-spectrogram for the sample speech corresponding to each of the sample texts; Obtain a sample Mel spectrogram associated with the sample speech corresponding to each of the sample texts, and calculate a mean square error loss value between the predicted Mel spectrogram generated for each of the sample texts and its associated sample Mel spectrogram; The preset text encoder, the preset variational adapter and the decoder are adjusted according to the calculated mean square error loss value, and the adjusted preset text encoder, the adjusted preset variational adapter and the adjusted decoder are used to regenerate a predicted Mel-spectrogram for each sample speech and calculate a new mean square error loss value, and the preset text encoder, the preset variational adapter and the decoder are continued to be adjusted according to the new mean square error loss value until the calculated mean square error loss value reaches a second preset condition, and the currently adjusted preset text encoder, the preset variational adapter and the decoder are integrated to obtain the acoustic model.

6. The method according to claim 5, characterized in that The method of using a preset reference encoder, a preset text encoder, and a preset variational adapter to process each sample text and a sample speech corresponding to each sample text, and generating a predicted mel-spectrogram for the sample speech corresponding to each sample text, includes: For each of the sample texts, performing phoneme conversion on the sample text to obtain a sample phoneme sequence of the sample text; Using the preset reference encoder to encode the sample speech corresponding to the sample text to obtain a sample style vector of the sample speech, and using the sample style vector to adjust the preset text encoder; Using the adjusted preset text encoder, converting the sample phoneme sequence into a sample hidden sequence; The sample hidden sequence is adjusted by using a preset variational adapter, and the adjusted sample hidden sequence is converted into a mel-spectrogram by using the decoder to obtain a predicted mel-spectrogram corresponding to the sample text.

7. The method according to claim 1, characterized in that The method of determining the phoneme embedding sequence of the target text based on the pre-trained acoustic model, and controlling the acoustic model to adjust the phoneme embedding sequence using the emotional style weight so that the acoustic model generates the emotional speech of the target text using the adjusted phoneme embedding sequence comprises: Performing phoneme conversion on the target text to obtain a target phoneme sequence of the target text; Inputting the target phoneme sequence into the acoustic model so that a preset text encoder in the acoustic model converts the target phoneme sequence into a target hidden sequence; Based on a preset variational adapter in the acoustic model, adjusting the target hidden sequence, and converting the adjusted target hidden sequence into a target mel-spectrogram based on a decoder in the acoustic model; A vocoder is called to convert the target mel-spectrogram, and the converted speech is used as the emotional speech of the target text.

8. An emotional speech synthesis device, characterized in that: include: A recognition module is used to obtain a target text to be subjected to speech synthesis and recognize a sentiment vector representation of the target text; A prediction module, used for predicting the emotional style of the emotional vector representation to obtain an emotional style weight; A generation module is used to determine the phoneme embedding sequence of the target text based on a pre-trained acoustic model, and control the acoustic model to adjust the phoneme embedding sequence using the emotional style weight so that the acoustic model generates the emotional speech of the target text using the adjusted phoneme embedding sequence.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the emotional speech synthesis method according to any one of claims 1 to 7 are implemented.

10. A readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the emotional speech synthesis method as claimed in any one of claims 1 to 7 are implemented.

Images