Emotional speech synthesis method and device, computer device and storage medium

Abstract

This application discloses an emotional speech synthesis method, apparatus, computer device, and storage medium. The method involves acquiring speech and text information to be synthesized; inputting the speech information into a pre-trained emotion extraction model, which parses the speech information and outputs the corresponding emotional representation; inputting the text information into a pre-set text encoder, which parses the text information and outputs the corresponding text features; inputting the text features and emotional representation into a pre-trained duration predictor, which parses the text features and emotional representation and outputs alignment information corresponding to the text features and emotional representation; generating a target Mel spectrum based on the alignment information, emotional representation, and text features; and generating emotional speech based on the target Mel spectrum, thus completing the synthesis of emotional speech. Furthermore, by achieving unlabeled emotional speech synthesis, it solves the problem of difficulty in obtaining labeled emotion data and can achieve speech synthesis with arbitrarily many emotional styles.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of speech processing, in particular to an emotional speech synthesis method and device, computer equipment and a storage medium. BACKGROUND

[0002] With the development of artificial intelligence technology, speech synthesis plays an increasingly important role in human-computer dialogue. Emotional speech synthesis, as an important supplement to speech synthesis, greatly improves the application scenarios of speech synthesis. According to different scenarios and environments, switching to corresponding emotional expressions can greatly improve the user experience, such as arranging intelligent customer service in the financial system by synthesizing emotional speech for intelligent customer service.

[0003] Traditional emotional speech synthesis needs to record a large amount of data related to emotion and style on the basis of labeled synthesis corpus. This leads to the following problems: first, the recording of emotional data is very difficult, and the emotional audio library of a single speaker cannot be migrated to other audio libraries; second, the difficulty of labeling emotion increases, and emotion is a relatively subjective expression characteristic. Different personnel have different understanding and cognition of different emotional expressions. Based on the above two points, the traditional speech emotion synthesis system is greatly limited by data labeling conditions, resulting in poor synthesized speech effect. SUMMARY

[0004] The present application provides an emotional speech synthesis method, device, computer equipment and storage medium, aiming to solve the problem that in the traditional mode, when the role given to the user contains many permissions that the user does not need to use, there is a risk that the user will use these permissions to obtain relevant private information, but if the user is not granted the corresponding role permission, it will bring a huge workload to the corresponding superior managers, resulting in the current emotional speech synthesis method having many problems such as security risks of permission data, data leakage risks, and non-uniformity of permission system standards. The provided method can form a complete, standard and safe automatic emotional speech synthesis scheme, ensuring that the preset system can be safely operated while responding quickly to business.

[0005] In a first aspect, the present application provides an emotional speech synthesis method, which includes:

[0006] obtaining speech information and text information to be synthesized;

[0007] inputting the speech information into a pre-trained emotion extraction model, the emotion extraction model analyzing the speech information and outputting an emotional representation corresponding to the speech information;

[0008] inputting the text information into a pre-set text encoder, the text encoder analyzing the text information and outputting a text feature corresponding to the text information;

[0009] inputting the text feature and the sentiment representation into a pre-trained duration predictor, the duration predictor analyzing the text feature and the sentiment representation, and outputting alignment information corresponding to the text feature and the sentiment representation;

[0010] generating a target mel-spectrum according to the alignment information, the sentiment representation and the text feature, generating the sentiment speech according to the target mel-spectrum, and completing the synthesis of the sentiment speech.

[0011] In some embodiments, before inputting the speech information into the pre-trained sentiment extraction model, the method further comprises: obtaining a synthesized historical sentiment speech, historical speech information and historical text information corresponding to the historical sentiment speech; inputting the historical sentiment speech into the to-be-trained sentiment extraction model to output historical sentiment representation corresponding to the historical sentiment speech; inputting the historical text information into the pre-trained text encoder to output historical text feature corresponding to the historical text information; inputting the historical sentiment representation and the historical text feature into the pre-trained duration predictor, the duration predictor outputting historical alignment information corresponding to the historical text feature and the historical sentiment representation; generating a predicted mel-spectrum according to the historical alignment information, the historical sentiment representation and the historical text feature, generating a predicted sentiment speech according to the predicted mel-spectrum; and completing the training of the sentiment extraction model according to the historical sentiment speech and the predicted sentiment speech.

[0012] For example, before inputting the historical text information and the historical text feature into the pre-trained duration predictor, the method further comprises: obtaining linear spectrum information corresponding to the historical sentiment speech, inputting the linear spectrum information into a pre-set posterior encoder, the posterior encoder analyzing the linear spectrum information to output hidden variable information and acoustic feature corresponding to the linear spectrum information; inputting the hidden variable information and the historical text information into a pre-set forced alignment module, the forced alignment module outputting forced alignment information corresponding to the hidden variable information and the historical text information; inputting the historical sentiment representation and the historical text feature into the to-be-trained duration predictor, the duration predictor outputting predicted alignment information corresponding to the historical text feature and the historical sentiment representation; and completing the training of the duration predictor according to the predicted alignment information and the forced alignment information.

[0013] It should be noted that, in some embodiments, before generating the sentiment speech according to the target mel-spectrum, the method further comprises: inputting the acoustic feature into a to-be-trained speech decoder, the speech decoder analyzing the acoustic feature to output predicted speech information corresponding to the acoustic feature; completing the training of the speech decoder according to the historical speech information and the predicted speech information, inputting the target mel-spectrum into the trained speech decoder, and outputting the sentiment speech.

[0014] It should be noted that in some embodiments, generating the target mel spectrum according to the alignment information, the emotion representation and the text feature, and generating the emotional speech according to the target mel spectrum, comprises: inputting the alignment information, the emotion representation and the text feature into a preset flow model, and the flow model outputting the target mel spectrum; wherein the posterior encoder analyzes the linear spectrum information, and outputs the hidden variable information corresponding to the linear spectrum information as simple hidden variable information; before inputting the hidden variable information and the historical text information into the preset forced alignment module, further comprising: inputting the simple hidden variable information into the flow model in reverse, and the flow model outputting complex hidden variable information corresponding to the simple hidden variable information, so as to input the complex hidden variable information into the forced alignment module.

[0015] It should be noted that in some embodiments, before inputting the historical text information into the pre-trained text encoder, further comprising: inputting the historical text information into the text encoder to be trained, outputting the relative entropy information of the predicted text feature and the acoustic feature corresponding to the predicted text feature; and completing the training of the text encoder according to the relative entropy information.

[0016] For example, completing the training of the emotion extraction model according to the historical emotional speech and the predicted emotional speech comprises: obtaining historical mel spectrum corresponding to the historical emotional speech; obtaining cross-loss information of the historical mel spectrum and the predicted mel spectrum; and completing the training of the emotion extraction model according to the cross-loss information, the historical emotional speech and the predicted emotional speech.

[0017] In a second aspect, the present application provides an emotional speech synthesis device, comprising:

[0018] An information acquisition unit acquires voice information and text information to be synthesized;

[0019] A representation extraction unit inputs the voice information into a pre-trained emotion extraction model, and the emotion extraction model analyzes the voice information and outputs emotion representation corresponding to the voice information;

[0020] A feature output unit inputs the text information into a preset text encoder, and the text encoder analyzes the text information and outputs text feature corresponding to the text information;

[0021] A duration prediction unit inputs the text feature and the emotion representation into a pre-trained duration predictor, and the duration predictor analyzes the text feature and the emotion representation and outputs alignment information corresponding to the text feature and the emotion representation;

[0022] A speech synthesis unit generates a target mel spectrum according to the alignment information, the emotion representation and the text feature, generates an emotional speech according to the target mel spectrum, and completes the synthesis of the emotional speech. In a third aspect, the present application further provides a computer device, comprising:

[0023] The memory and the processor;

[0024] The memory is used for storing a computer program;

[0025] The processor is used for executing the computer program and realizing the steps of the emotional speech synthesis method according to the first aspect.

[0026] In a fourth aspect, the present application further provides a computer readable storage medium, which stores a computer program, and the computer program, when executed by a processor, causes the processor to realize the steps of the emotional speech synthesis method according to the first aspect.

[0027] The present application discloses an emotional speech synthesis method, device, computer equipment and storage medium. The method comprises the following steps: obtaining speech information and text information to be synthesized; inputting the speech information into a pre-trained emotion extraction model, the emotion extraction model analyzing the speech information, and outputting an emotional representation corresponding to the speech information; inputting the text information into a pre-set text encoder, the text encoder analyzing the text information, and outputting a text feature corresponding to the text information; inputting the text feature and the emotional representation into a pre-trained duration predictor, the duration predictor analyzing the text feature and the emotional representation, and outputting alignment information corresponding to the text feature and the emotional representation; generating a target mel spectrum according to the alignment information, the emotional representation and the text feature; and generating an emotional speech according to the target mel spectrum, thereby completing the synthesis of the emotional speech.

[0028] By extracting the speech information and the text information, the method can align and fuse the emotional representation corresponding to the language information and the text feature corresponding to the text information, and then train the related model through an end-to-end method without labeling the training samples to realize accurate synthesis of the emotional speech.

[0029] Furthermore, the method can directly learn the role of the emotional representation in the speech synthesis model by realizing the un-labeled emotional speech synthesis, thereby solving the problem of difficulty in emotional labeling data.

[0030] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present application. BRIEF DESCRIPTION OF DRAWINGS

[0031] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are some embodiments of the present application, and those skilled in the art can also obtain other drawings according to these drawings without creative labor.

[0032] Figure 1 This is a schematic flowchart illustrating the steps of an embodiment of the emotional speech synthesis method provided in this application;

[0033] Figure 2 This is a schematic flowchart illustrating the steps of a model training method provided in an embodiment of this application;

[0034] Figure 3 This is a schematic flowchart illustrating the steps of a cross-entropy combined training method provided in an embodiment of this application;

[0035] Figure 4 This is a schematic diagram of the structure of an emotional speech synthesis device provided in an embodiment of this application;

[0036] Figure 5 This is a schematic block diagram of the structure of a computer device provided in an embodiment of this application.

[0037] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Detailed Implementation

[0038] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0039] The flowchart shown in the attached diagram is for illustrative purposes only and does not necessarily include all content and operations / steps, nor does it necessarily have to be performed in the order described. For example, some operations / steps can be broken down, combined, or partially merged, so the actual execution order may change depending on the actual situation.

[0040] It should be understood that, in order to clearly describe the technical solutions of the embodiments of this utility model, the terms "first" and "second" are used in the embodiments of this utility model to distinguish identical or similar items with essentially the same function and effect. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order, and the terms "first" and "second" are not necessarily different.

[0041] It should be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of the application. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0042] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0043] To facilitate understanding of the contents of this application specification, some terms appearing in this application are explained below.

[0044] 1. Flow-based model: A flow model describes a distribution that can flow from one distribution A to another, much like water flowing. The distribution transformation in a flow model is phased, meaning the model transforms from one distribution to another specified distribution step by step.

[0045] At the same time, the flow model is also reversible. Specifically, it can not only transform from distribution A to distribution B, but also from distribution B to distribution A.

[0046] 2. Linear Spectrum: A sliding-window Fourier transform is performed on the speech signal in the time domain. Before the Fourier transform, a window function, typically the Hanning window, is used to prevent energy leakage. After the short-time Fourier transform, the linear spectrum of the speech signal is obtained. The linear spectrum has a high dimension, typically set to 1024.

[0047] 3. Latent variables: Comprehensive variables with characteristics that cannot be directly observed. They are unobservable, or in other words, "latent" is their appearance; comprehensiveness is their essence.

[0048] In contrast to latent variables, we call ordinary variables manifest variables or observed variables.

[0049] 4. Forced Alignment Module: The forced alignment module is a commonly used module in speech tasks, widely applied in frame-level speech recognition and speech synthesis tasks. It is also a core algorithm in tasks such as automatic captioning and spoken language assessment. Forced alignment is similar to speech recognition and decoding. Speech recognition and decoding refers to taking a speech sequence as input and outputting the corresponding recognized text, given a speech recognition model. Forced alignment, on the other hand, refers to taking a speech sequence as input and a correct labeled text (which can be a text sequence or a phoneme sequence, etc.) and outputting the correspondence between speech frames and text, such as frames 20 to 25 corresponding to the phoneme 'a'.

[0050] 5. End-to-end learning: In traditional machine learning solutions, there are often multiple independent modules. For example, in a typical natural language processing problem, there are multiple steps such as word segmentation, part-of-speech tagging, syntactic analysis, and semantic analysis. Each step is an independent task module that needs to be labeled and trained separately. At the same time, since the result of the previous step will serve as the condition for the next step, the effect of the previous step will affect the next step, thus affecting the entire solution.

[0051] During training, deep learning models generate a prediction from input to output. This prediction is compared to the actual observation, resulting in an error. This error propagates through each layer of the model (backpropagation), and each layer's representation is adjusted based on this error until the model converges or achieves the desired effect. This is end-to-end learning. Compared to traditional machine learning, end-to-end learning eliminates the need for separate labeling and training of multiple independent task modules, and also reduces the possibility of errors arising from too many intermediate steps.

[0052] 6. Text Encoder: A text encoder is used to extract text features.

[0053] 7. Duration Predictor: Typically, the relationship between text and pre-speech is not equal in length. A single text feature occupies one or more speech frames. The role of the duration predictor is to establish the alignment relationship between the input and output text.

[0054] The following detailed description of some embodiments of this application is provided in conjunction with the accompanying drawings. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0055] With the development of artificial intelligence technology, speech synthesis is playing an increasingly important role in the field of human-computer dialogue. Emotional speech synthesis, as an important supplement to speech synthesis, greatly expands the application scenarios of speech synthesis. Switching to corresponding emotional expressions according to different scenarios and environments can significantly improve the user experience, such as when deploying intelligent customer service in financial systems, by synthesizing engaging voices for intelligent customer service representatives.

[0056] Traditional emotional speech synthesis requires recording a large amount of emotion and style-related data based on annotated corpora. This leads to the following problems: First, recording emotional data is very difficult, as a single speaker's emotional voice database cannot be transferred to other voice databases; second, annotating emotions is more difficult, as emotions are relatively subjective expressive features, and different people have different perceptions and understandings of different emotional expressions. Based on these two points, traditional speech emotion synthesis systems are greatly limited by data annotation conditions, resulting in poor synthesized speech quality.

[0057] To resolve the above issues, please refer to [link / reference]. Figure 1 , Figure 1 This is a schematic flowchart of an embodiment of the emotional speech synthesis method provided in this application. This emotional speech synthesis method can be implemented by a computer device, which can be deployed on a single server or a server cluster. It can also be deployed on handheld terminals, laptops, wearable devices, or robots, etc.

[0058] The provided method can be used in the financial sector to synthesize emotional voice for intelligent financial customer service, making the AI ​​customer service more human-like and providing users with a better service experience. It can also be applied to any field requiring the deployment of AI customer service, such as healthcare and education; this application does not impose any limitations on this application.

[0059] To solve the above problem, please refer to Figure 1 Specifically, such as Figure 1 As shown, the provided method includes steps S101 to S105. Details are as follows:

[0060] S101. Obtain the speech and text information to be synthesized.

[0061] Specifically, before speech synthesis, the server obtains unlabeled speech and text information. The speech information contains emotional representations that are intended to be added through emotional speech synthesis, such as the speech of a customer service representative used as a template in a pre-set financial system. This will enable the final synthesized emotional speech to possess the emotions of that customer service representative, thereby improving the user experience.

[0062] S102. Input the speech information into the pre-trained sentiment extraction model. The sentiment extraction model parses the speech information and outputs the sentiment representation corresponding to the speech information.

[0063] Specifically, by inputting unlabeled speech information into a pre-trained emotion extraction model via a server, the model can extract the emotion representation from the speech information. This representation serves as the corresponding emotion label for that speech information. The emotion representation of a speech message, previously represented by discrete labels (happiness, fear, calmness, etc.), is transformed into a fixed-dimensional vector representing the emotion of that speech message. This achieves the continuation of discrete data, facilitating the subsequent accurate synthesis of emotion representations.

[0064] S103. Input the text information into the preset text encoder. The text encoder parses the text information and outputs the text features corresponding to the text information.

[0065] Specifically, the server extracts text features, such as feature vectors, from the text information to be synthesized by inputting it into a pre-defined text encoder. Text features include the relevance of each word in the text, i.e., Tf-i df (term frequency-inverse document frequency). This is achieved by dividing the number of times a word appears by the number of documents containing that word. When a word appears frequently in one text but not in others, it is considered important to that particular document. A vector using one-hot encoding contains zeros everywhere except for the first index, which uniquely identifies each word. Text information can also be converted into corresponding text features using word2vec (word-to-vector) technology.

[0066] S104. Input the text features and sentiment representations into the pre-trained duration predictor. The duration predictor parses the text features and sentiment representations and outputs the alignment information corresponding to the text features and sentiment representations.

[0067] Specifically, in the process of emotional speech synthesis, the relationship between text and speech is not equal in length. A single text feature occupies one or more speech frames. This application establishes an alignment relationship between the input and output text by setting a duration predictor, thereby ensuring that the synthesized emotional speech has better naturalness.

[0068] S105. Generate the target Mel spectrum based on alignment information, sentiment representation, and text features, and generate the emotional speech based on the target Mel spectrum to complete the synthesis of the emotional speech.

[0069] Specifically, after acquiring alignment information, sentiment representation, and text features, the server can generate target Mel spectra corresponding to these elements. This can be used to generate emotionally rich and highly natural-sounding speech, even when input into a pre-defined speech generator or streaming model. Furthermore, the provided method can synthesize emotional speech of any single or multiple styles without labeling.

[0070] In some embodiments, such as Figure 2 As shown, steps S201 to S206 are included before inputting the speech information into the pre-trained emotion extraction model.

[0071] Step S201. Obtain the synthesized historical emotional speech, the historical speech information corresponding to the historical emotional speech, and the historical text information;

[0072] Step S202. Input the historical emotional speech into the emotion extraction model to be trained, and output the historical emotion representation corresponding to the historical emotional speech;

[0073] Step S203. Input the historical text information into the pre-trained text encoder and output the historical text features corresponding to the historical text information;

[0074] Step S204. Input the historical sentiment representation and historical text features into the pre-trained duration predictor, and the duration predictor outputs the historical alignment information corresponding to the historical text features and historical sentiment representation;

[0075] Step S205. Generate a predicted Mel spectrum based on historical alignment information, historical sentiment representation and historical text features, and generate predicted sentiment speech based on the predicted Mel spectrum;

[0076] Step S206. Train the emotion extraction model based on historical and predicted emotional speech.

[0077] The proposed method acquires synthesized historical emotional speech and its corresponding pre-synthesis historical speech and historical text information, enabling the training of the emotion extraction model based on unlabeled samples. In other words, the training of the emotion extraction model is completed through the reconstruction loss between speech and real speech.

[0078] For example, before inputting historical text information and historical text features into the pre-trained duration predictor, the method further includes: obtaining linear spectrum information corresponding to historical emotional speech; inputting the linear spectrum information into a preset posterior encoder; the posterior encoder parses the linear spectrum information and outputs latent variable information and acoustic features corresponding to the linear spectrum information; inputting the latent variable information and historical text information into a preset forced alignment module; the forced alignment module outputs forced alignment information corresponding to the latent variable information and historical text information; inputting historical emotional representations and historical text features into the duration predictor to be trained; the duration predictor outputs predicted alignment information corresponding to historical text features and historical emotional representations; and completing the training of the duration predictor based on the predicted alignment information and forced alignment information.

[0079] By extracting latent variable information corresponding to the linear spectrum information of historical emotional speech, the proposed method can simultaneously combine a forced alignment module to determine the forced alignment information between the latent variable information and the historical text information. This enables the trained duration predictor to learn to synthesize text features and sentiment representations based on the latent variable information. Furthermore, the proposed method allows for end-to-end training of both the sentiment extraction model and the duration predictor.

[0080] It should be noted that, in some embodiments, before generating emotional speech based on the target Mel spectrum, the method further includes: inputting acoustic features into a speech decoder to be trained; the speech decoder parses the acoustic features and outputs the predicted speech information corresponding to the acoustic features; and training the speech decoder based on historical speech information and predicted speech information, so as to input the target Mel spectrum into the trained speech decoder and output emotional speech.

[0081] By extracting acoustic features of speech information (such as Mel spectrum information), end-to-end training of the speech decoder can be achieved simultaneously, thereby ensuring that the methods provided in emotion extraction, speech alignment, and speech synthesis can all achieve ideal results.

[0082] It should be noted that, in some embodiments, generating a target Mel spectrum based on alignment information, sentiment representation, and text features, and generating emotional speech based on the target Mel spectrum, includes: inputting alignment information, sentiment representation, and text features into a preset streaming model, and the streaming model outputting the target Mel spectrum; wherein, the posterior encoder parses the linear spectrum information and outputs the latent variable information corresponding to the linear spectrum information as simple latent variable information; before inputting the latent variable information and historical text information into a preset forced alignment module, the method further includes: inversely inputting the simple latent variable information into the streaming model, and the streaming model outputting the complex latent variable information corresponding to the simple latent variable information, so as to input the complex latent variable information into the forced alignment module.

[0083] Due to the complexity of latent variables' distributions, direct sampling during the generation process is difficult. The proposed method transforms the complex distribution of latent variables into a simpler distribution (such as a Gaussian distribution). Specifically, a posterior encoder first outputs simple latent variable information, and then, combined with the inverse generation characteristics of a flow model, the generation process only requires sampling from the simple distribution and then performing the inverse process of the flow model to obtain the complex latent variable information. This improves the end-to-end training efficiency of modules such as the sentiment extraction model and duration predictor while simplifying the training steps.

[0084] It should be noted that, in some embodiments, before inputting historical text information into the pre-trained text encoder, the method further includes: inputting historical text information into the text encoder to be trained, outputting the predicted text features corresponding to the historical text information to obtain the relative entropy information between the acoustic features and the predicted text features; and completing the training of the text encoder based on the relative entropy information.

[0085] By combining the acoustic features output from the posterior encoder with the text features of the text encoder using the spectral density (also known as the KL distance), the text encoder can be precisely trained end-to-end, ensuring the accuracy of text feature extraction.

[0086] For example, such as Figure 3 As shown, the training of the emotion extraction model is completed based on historical emotional speech and predicted emotional speech, including steps S206a to S206c.

[0087] Step S206a. Obtain the historical Mel spectrum corresponding to the historical emotional speech;

[0088] Step S206b. Obtain the cross-loss information of historical Mel spectrum and predicted Mel spectrum;

[0089] Step S206c. Train the emotion extraction model based on the cross-loss information, historical emotional speech, and predicted emotional speech.

[0090] The proposed method improves the accuracy of model training by combining the reconstruction loss between generated speech and real speech with the cross loss between the mel spectrum features of generated speech and real mel spectrum, ensuring that accurate emotional representations can be extracted from speech information and improving the accuracy of the final synthesized speech information.

[0091] This application discloses a method for synthesizing emotional speech. The method involves: acquiring speech and text information to be synthesized; inputting the speech information into a pre-trained emotion extraction model, which parses the speech information and outputs the corresponding emotional representation; inputting the text information into a pre-set text encoder, which parses the text information and outputs the corresponding text features; inputting the text features and emotional representation into a pre-trained duration predictor, which parses the text features and emotional representation and outputs alignment information corresponding to the text features and emotional representation; generating a target Mel spectrum based on the alignment information, emotional representation, and text features; and generating emotional speech based on the target Mel spectrum, thus completing the synthesis of emotional speech.

[0092] By extracting speech and text information, the proposed method can perform time-time alignment and fusion of the emotional representations corresponding to the speech information and the text features corresponding to the text information, thereby achieving accurate synthesis of emotional speech without the need to label the training samples.

[0093] Furthermore, the proposed method addresses the challenge of obtaining labeled emotional data by achieving unlabeled emotional speech synthesis, enabling direct learning of the role of emotional representations in the speech synthesis model. Moreover, the method can synthesize speech with any number of emotional styles as needed.

[0094] Please see Figure 4 As shown, Figure 4 This is a schematic diagram of the structure of the emotional speech synthesis device 300 provided in the embodiments of this application. The emotional speech synthesis device 300 is used to perform the steps of the emotional speech synthesis method shown in the above embodiments. The emotional speech synthesis device 300 can be a single server or a server cluster, or the emotional speech synthesis device 300 can be a terminal, such as a handheld terminal, a laptop computer, a wearable device, or a robot.

[0095] like Figure 4 As shown, the emotional speech synthesis device 300 includes:

[0096] Information acquisition unit 301 acquires speech information and text information to be synthesized;

[0097] The representation extraction unit 302 is used to input speech information into a pre-trained emotion extraction model. The emotion extraction model parses the speech information and outputs the emotion representation corresponding to the speech information.

[0098] The feature output unit 303 is used to input text information into a preset text encoder, which parses the text information and outputs the text features corresponding to the text information.

[0099] The duration prediction unit 304 is used to input text features and sentiment representations into a pre-trained duration predictor. The duration predictor parses the text features and sentiment representations and outputs the alignment information corresponding to the text features and sentiment representations.

[0100] The speech synthesis unit 305 is used to generate a target Mel spectrum based on alignment information, emotional representation, and text features, and to generate emotional speech based on the target Mel spectrum, thus completing the synthesis of emotional speech. It should be noted that those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the emotional speech synthesis device and its modules described above can be referred to the corresponding processes in the embodiments of the emotional speech synthesis method described above, and will not be repeated here.

[0101] The aforementioned emotional speech synthesis method can be implemented as a computer program, which can be used in, for example... Figure 4 It runs on the device shown.

[0102] Please see Figure 5 , Figure 5 This is a schematic block diagram of the structure of a computer device provided in an embodiment of this application. The computer device includes a processor, a memory, and a network interface connected via a device bus, wherein the memory may include a storage medium and internal memory.

[0103] The storage medium may store operating devices and computer programs. The computer program includes program instructions that, when executed, cause the processor to perform any emotional speech synthesis method.

[0104] The processor provides computing and control capabilities, supporting the operation of the entire computer device.

[0105] Internal memory provides an environment for the execution of computer programs in non-volatile storage media, which, when executed by a processor, enable the processor to perform any emotional speech synthesis method.

[0106] This network interface is used for network communication, such as sending assigned tasks. Those skilled in the art will understand that... Figure 5 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the terminal to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0107] It should be understood that the processor can be a Central Processing Unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among these, a general-purpose processor can be a microprocessor or any conventional processor.

[0108] In one embodiment, the processor is configured to run a computer program stored in memory to perform the following steps:

[0109] Obtain the speech and text information to be synthesized;

[0110] The speech information is input into a pre-trained emotion extraction model, which parses the speech information and outputs the emotion representation corresponding to the speech information.

[0111] The text information is input into a preset text encoder, which parses the text information and outputs the text features corresponding to the text information.

[0112] Text features and sentiment representations are input into a pre-trained duration predictor. The duration predictor parses the text features and sentiment representations and outputs the alignment information corresponding to the text features and sentiment representations.

[0113] The target Mel spectrum is generated based on alignment information, sentiment representation, and text features. Emotional speech is then generated based on the target Mel spectrum, thus completing the synthesis of emotional speech.

[0114] In some embodiments, before inputting the speech information into the pre-trained sentiment extraction model, the method further includes: acquiring synthesized historical sentiment speech, historical speech information corresponding to the historical sentiment speech, and historical text information; inputting the historical sentiment speech into the sentiment extraction model to be trained, and outputting the historical sentiment representation corresponding to the historical sentiment speech; inputting the historical text information into a pre-trained text encoder, and outputting the historical text features corresponding to the historical text information; inputting the historical sentiment representation and the historical text features into a pre-trained duration predictor, and the duration predictor outputting the historical alignment information corresponding to the historical text features and the historical sentiment representation; generating a predicted Mel spectrum based on the historical alignment information, the historical sentiment representation, and the historical text features, and generating a predicted sentiment speech based on the predicted Mel spectrum; and completing the training of the sentiment extraction model based on the historical sentiment speech and the predicted sentiment speech.

[0115] For example, before inputting historical text information and historical text features into the pre-trained duration predictor, the method further includes: obtaining linear spectrum information corresponding to historical emotional speech; inputting the linear spectrum information into a preset posterior encoder; the posterior encoder parses the linear spectrum information and outputs latent variable information and acoustic features corresponding to the linear spectrum information; inputting the latent variable information and historical text information into a preset forced alignment module; the forced alignment module outputs forced alignment information corresponding to the latent variable information and historical text information; inputting historical emotional representations and historical text features into the duration predictor to be trained; the duration predictor outputs predicted alignment information corresponding to historical text features and historical emotional representations; and completing the training of the duration predictor based on the predicted alignment information and forced alignment information.

[0116] It should be noted that, in some embodiments, before generating emotional speech based on the target Mel spectrum, the method further includes: inputting acoustic features into a speech decoder to be trained; the speech decoder parses the acoustic features and outputs the predicted speech information corresponding to the acoustic features; and training the speech decoder based on historical speech information and predicted speech information, so as to input the target Mel spectrum into the trained speech decoder and output emotional speech.

[0117] It should be noted that, in some embodiments, generating a target Mel spectrum based on alignment information, sentiment representation, and text features, and generating emotional speech based on the target Mel spectrum, includes: inputting alignment information, sentiment representation, and text features into a preset streaming model, and the streaming model outputting the target Mel spectrum; wherein, the posterior encoder parses the linear spectrum information and outputs the latent variable information corresponding to the linear spectrum information as simple latent variable information; before inputting the latent variable information and historical text information into a preset forced alignment module, the method further includes: inversely inputting the simple latent variable information into the streaming model, and the streaming model outputting the complex latent variable information corresponding to the simple latent variable information, so as to input the complex latent variable information into the forced alignment module.

[0118] It should be noted that, in some embodiments, before inputting historical text information into the pre-trained text encoder, the method further includes: inputting historical text information into the text encoder to be trained, outputting the predicted text features corresponding to the historical text information to obtain the relative entropy information between the acoustic features and the predicted text features; and completing the training of the text encoder based on the relative entropy information.

[0119] For example, training the emotion extraction model based on historical emotional speech and predicted emotional speech includes: obtaining the historical Mel spectrum corresponding to the historical emotional speech; obtaining the cross-loss information of the historical Mel spectrum and the predicted Mel spectrum; and training the emotion extraction model based on the cross-loss information, the historical emotional speech, and the predicted emotional speech.

[0120] The embodiments of this application also provide a computer-readable storage medium storing a computer program, the computer program including program instructions, and the processor executing the program instructions to implement the steps of the emotional speech synthesis method provided in the above embodiments of this application.

[0121] The computer-readable storage medium may be an internal storage unit of the computer device described in the foregoing embodiments, such as the hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, SmartMedia Card (SMC), Secure Digital (SD) card, or Flash Card equipped on the computer device.

[0122] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for synthesizing emotional speech, characterized in that, include: Acquire the speech information and text information to be synthesized; acquire the synthesized historical emotional speech, the historical speech information and historical text information corresponding to the historical emotional speech; input the historical emotional speech into the emotion extraction model to be trained, and output the historical emotional representation corresponding to the historical emotional speech; input the historical text information into the pre-trained text encoder, and output the historical text features corresponding to the historical text information. The historical sentiment representation and historical text features are input into a pre-trained duration predictor, which outputs historical alignment information corresponding to the historical text features and the historical sentiment representation; a predicted Mel spectrum is generated based on the historical alignment information, the historical sentiment representation, and the historical text features; a predicted sentiment speech is generated based on the predicted Mel spectrum; and the sentiment extraction model is trained based on the historical sentiment speech and the predicted sentiment speech. The speech information is input into a pre-trained emotion extraction model, which parses the speech information and outputs the emotion representation corresponding to the speech information. The text information is input into a preset text encoder, which parses the text information and outputs the text features corresponding to the text information; the text features include the term frequency-reverse document frequency information of each word in the text; The text features and sentiment representations are input into a pre-trained duration predictor, which parses the text features and sentiment representations and outputs alignment information corresponding to the text features and sentiment representations. A target Mel spectrum is generated based on the alignment information, sentiment representation, and text features. Emotional speech is then generated based on the target Mel spectrum, thus completing the synthesis of the emotional speech.

2. The method according to claim 1, characterized in that, Before inputting the historical text information and historical text features into the pre-trained duration predictor, the method further includes: Obtain the linear spectrum information corresponding to the historical emotional speech, input the linear spectrum information into a preset posterior encoder, the posterior encoder parses the linear spectrum information, and outputs the latent variable information and acoustic features corresponding to the linear spectrum information; The latent variable information and the historical text information are input into a preset forced alignment module, and the forced alignment module outputs forced alignment information corresponding to the latent variable information and the historical text information. The historical sentiment representation and historical text features are input into the duration predictor to be trained, and the duration predictor outputs the prediction alignment information corresponding to the historical text features and the historical sentiment representation. The duration predictor is trained based on the predicted alignment information and the forced alignment information.

3. The method according to claim 2, characterized in that, Prior to generating the emotional speech based on the target Mel spectrum, the method further includes: The acoustic features are input into the speech decoder to be trained. The speech decoder parses the acoustic features and outputs the predicted speech information corresponding to the acoustic features. The speech decoder is trained based on the historical speech information and the predicted speech information, and the target Mel spectrum is input into the trained speech decoder to output the emotional speech.

4. The method according to claim 2, characterized in that, The step of generating a target Mel spectrum based on the alignment information, emotional representation, and text features, and generating emotional speech based on the target Mel spectrum, includes: The alignment information, sentiment representation, and text features are input into a preset flow model, which outputs the target Mel spectrum. The posterior encoder parses the linear spectrum information and outputs the latent variable information corresponding to the linear spectrum information as simple latent variable information; before inputting the latent variable information and the historical text information into the preset forced alignment module, the method further includes: The simple latent variable information is inversely input into the flow model, and the flow model outputs the complex latent variable information corresponding to the simple latent variable information, so as to input the complex latent variable information into the forced alignment module.

5. The method according to claim 2, characterized in that, Before inputting the historical text information into the pre-trained text encoder, the method further includes: The historical text information is input into the text encoder to be trained, and the predicted text features corresponding to the historical text information are output. Obtain the relative entropy information between the acoustic features and the predicted text features; The text encoder is trained based on relative entropy information.

6. The method according to claim 1, characterized in that, The step of training the emotion extraction model based on the historical emotional speech and the predicted emotional speech includes: Obtain the historical Mel spectrum corresponding to the historical emotional speech; Obtain the cross-loss information between the historical Mel spectrum and the predicted Mel spectrum; The emotion extraction model is trained based on the cross-loss information, the historical emotional speech, and the predicted emotional speech.

7. An emotional speech synthesis device, characterized in that, include: The information acquisition unit acquires speech information and text information to be synthesized; acquires synthesized historical emotional speech, historical speech information and historical text information corresponding to the historical emotional speech; inputs the historical emotional speech into the emotion extraction model to be trained, and outputs the historical emotional representation corresponding to the historical emotional speech; inputs the historical text information into the pre-trained text encoder, and outputs the historical text features corresponding to the historical text information. The historical sentiment representation and historical text features are input into a pre-trained duration predictor, which outputs historical alignment information corresponding to the historical text features and the historical sentiment representation; a predicted Mel spectrum is generated based on the historical alignment information, the historical sentiment representation, and the historical text features; a predicted sentiment speech is generated based on the predicted Mel spectrum; and the sentiment extraction model is trained based on the historical sentiment speech and the predicted sentiment speech. The representation extraction unit is used to input the speech information into a pre-trained emotion extraction model, the emotion extraction model parses the speech information and outputs the emotion representation corresponding to the speech information; The feature output unit is used to input the text information to a preset text encoder, the text encoder parses the text information and outputs the text features corresponding to the text information; the text features include the term frequency-reverse document frequency information of each word in the text; The duration prediction unit is used to input the text features and sentiment representations into a pre-trained duration predictor, which parses the text features and sentiment representations and outputs alignment information corresponding to the text features and sentiment representations. The speech synthesis unit is used to generate a target Mel spectrum based on the alignment information, emotional representation, and text features, generate emotional speech based on the target Mel spectrum, and complete the synthesis of the emotional speech.

8. A computer device, characterized in that, The computer device includes a memory and a processor; The memory is used to store computer programs; The processor is configured to execute the computer program and, in executing the computer program, implement the method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, causes the processor to implement the method as described in any one of claims 1 to 6.

Images