Emotional speech synthesis method and device, computer equipment and storage medium

Abstract

The invention discloses an emotional speech synthesis method and device, computer equipment and a storage medium. The method comprises the following steps: acquiring to-be-synthesized voice information and text information; inputting the voice information into a pre-trained emotion extraction model, analyzing the voice information by the emotion extraction model, and outputting emotion representation corresponding to the voice information; inputting the text information into a preset text encoder, analyzing the text information by the text encoder, and outputting text features corresponding to the text information; inputting the text features and the emotion features into a pre-trained duration predictor, analyzing the text features and the emotion features by the duration predictor, and outputting alignment information corresponding to the text features and the emotion features; and generating a target Mel spectrum according to the alignment information, the emotional representation and the text features, generating emotional speech according to the target Mel spectrum, and completing synthesis of the emotional speech. Therefore, the problem of difficulty in emotion data labeling is solved by realizing unlabeled emotion speech synthesis, and speech synthesis of any multi-emotion style can be realized.

Description

Technical Field

[0001] The present application relates to the field of speech processing technology, and in particular to an emotional speech synthesis method, apparatus, computer equipment and storage medium. Background Art

[0002] With the development of artificial intelligence technology, speech synthesis plays an increasingly important role in the field of human-computer dialogue. As an important supplement to speech synthesis, emotional speech synthesis has greatly improved the application scenarios of speech synthesis. According to different scenarios and environments, switching corresponding emotional expressions can greatly improve the user experience when deploying intelligent customer service in financial systems by synthesizing infectious speech for intelligent customer service.

[0003] Traditional emotional speech synthesis requires recording a large amount of emotion and style-related data based on annotated synthetic corpus. This leads to the following problems: First, it is very difficult to record emotional data, and the emotional sound library of a single speaker cannot be transferred to other sound libraries; second, it is more difficult to annotate emotions. Emotions are relatively subjective expression characteristics, and different people have different cognition and understanding of the expression of different emotions. Based on the above two points, the traditional speech emotion synthesis system is greatly restricted by data annotation conditions, resulting in poor synthesized speech effects. Summary of the invention

[0004] This application provides an emotional speech synthesis method, device, computer equipment and storage medium, which aims to solve the problem that in the traditional mode, when the role given to the user contains many permissions that the user does not need to use, there is a risk that the user uses these permissions to obtain relevant privacy information. However, if the user is not granted the corresponding role permissions, it will bring a huge workload to the corresponding senior management personnel, resulting in many problems in the current emotional speech synthesis method, such as hidden dangers of permission data security, data leakage risks, and inconsistent permission system standards. The method provided can form a complete, standard, and safe automated emotional speech synthesis solution to ensure that the preset system can operate safely while responding to business quickly.

[0005] In a first aspect, the present application provides an emotional speech synthesis method, comprising:

[0006] Obtaining voice information and text information to be synthesized;

[0007] The speech information is input into the pre-trained emotion extraction model, and the emotion extraction model analyzes the speech information and outputs the emotion representation corresponding to the speech information;

[0008] Input the text information into a preset text encoder, the text encoder parses the text information, and outputs text features corresponding to the text information;

[0009] Input text features and sentiment representations into the pre-trained duration predictor, which parses the text features and sentiment representations and outputs alignment information corresponding to the text features and sentiment representations;

[0010] A target Mel spectrum is generated according to the alignment information, sentiment representation and text features, and emotional speech is generated according to the target Mel spectrum to complete the synthesis of emotional speech.

[0011] In some embodiments, before inputting speech information into a pre-trained emotion extraction model, it also includes: obtaining synthesized historical emotion speech, historical speech information corresponding to the historical emotion speech, and historical text information; inputting the historical emotion speech into the emotion extraction model to be trained, and outputting the historical emotion representation corresponding to the historical emotion speech; inputting the historical text information into a pre-trained text encoder, and outputting the historical text features corresponding to the historical text information; inputting the historical emotion representation and the historical text features into a pre-trained duration predictor, and the duration predictor outputs the historical alignment information corresponding to the historical text features and the historical emotion representation; generating a predicted Mel spectrum based on the historical alignment information, the historical emotion representation, and the historical text features, and generating a predicted emotion speech based on the predicted Mel spectrum; and completing the training of the emotion extraction model based on the historical emotion speech and the predicted emotion speech.

[0012] Exemplarily, before inputting historical text information and historical text features into a pre-trained duration predictor, it also includes: obtaining linear spectrum information corresponding to historical emotional speech, inputting the linear spectrum information into a preset posterior encoder, the posterior encoder parsing the linear spectrum information, and outputting latent variable information and acoustic features corresponding to the linear spectrum information; inputting the latent variable information and historical text information into a preset forced alignment module, the forced alignment module outputs forced alignment information corresponding to the latent variable information and the historical text information; inputting historical emotional representation and historical text features into the duration predictor to be trained, the duration predictor outputs predicted alignment information corresponding to the historical text features and the historical emotional representation, and completing the training of the duration predictor according to the predicted alignment information and the forced alignment information.

[0013] It should be noted that, in some embodiments, before generating emotional speech according to the target Mel spectrum, it also includes: inputting the acoustic features into the speech decoder to be trained, the speech decoder parses the acoustic features, and outputs the predicted speech information corresponding to the acoustic features; completing the training of the speech decoder according to the historical speech information and the predicted speech information, so as to input the target Mel spectrum into the trained speech decoder and output the emotional speech.

[0014] It should be noted that, in some embodiments, a target Mel spectrum is generated according to alignment information, emotion representation and text features, and emotional speech is generated according to the target Mel spectrum, including: inputting the alignment information, emotion representation and text features into a preset stream model, and the stream model outputs the target Mel spectrum; wherein the posterior encoder parses the linear spectrum information, and outputs the latent variable information corresponding to the linear spectrum information as simple latent variable information; before inputting the latent variable information and historical text information into a preset forced alignment module, it also includes: inversely inputting the simple latent variable information into the stream model, and the stream model outputs the complex latent variable information corresponding to the simple latent variable information, so as to input the complex latent variable information into the forced alignment module.

[0015] It should be noted that, in some embodiments, before inputting the historical text information into the pre-trained text encoder, it also includes: inputting the historical text information into the text encoder to be trained, outputting the predicted text features corresponding to the historical text information to obtain the relative entropy information of the acoustic features and the predicted text features; and completing the training of the text encoder according to the relative entropy information.

[0016] Exemplarily, the training of the emotion extraction model is completed based on historical emotional speech and predicted emotional speech, including: obtaining historical Mel spectrum corresponding to the historical emotional speech; obtaining cross-loss information of the historical Mel spectrum and the predicted Mel spectrum; and completing the training of the emotion extraction model based on the cross-loss information, historical emotional speech and predicted emotional speech.

[0017] In a second aspect, the present application provides an emotional speech synthesis device, comprising:

[0018] An information acquisition unit, which acquires voice information and text information to be synthesized;

[0019] A representation extraction unit, used to input the speech information into a pre-trained emotion extraction model, the emotion extraction model analyzes the speech information, and outputs the emotion representation corresponding to the speech information;

[0020] A feature output unit, used to input text information into a preset text encoder, the text encoder parses the text information, and outputs text features corresponding to the text information;

[0021] A duration prediction unit, used for inputting text features and sentiment representation into a pre-trained duration predictor, the duration predictor parses the text features and sentiment representation, and outputs alignment information corresponding to the text features and sentiment representation;

[0022] The speech synthesis unit is used to generate a target Mel spectrum according to the alignment information, the emotion representation and the text feature, generate the emotional speech according to the target Mel spectrum, and complete the synthesis of the emotional speech. In a third aspect, the present application also provides a computer device, including:

[0023] Memory and processor;

[0024] The memory is used to store computer programs;

[0025] The processor is used to execute the computer program and implement the steps of the emotional speech synthesis method described in the first aspect when executing the computer program.

[0026] In a fourth aspect, the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor implements the steps of the emotional speech synthesis method as described in the first aspect above.

[0027] The present application discloses an emotional speech synthesis method, device, computer equipment and storage medium. The method obtains speech information and text information to be synthesized; inputs the speech information into a pre-trained emotion extraction model, the emotion extraction model parses the speech information, and outputs the emotion representation corresponding to the speech information; inputs the text information into a preset text encoder, the text encoder parses the text information, and outputs the text features corresponding to the text information; inputs the text features and the emotion representation into a pre-trained duration predictor, the duration predictor parses the text features and the emotion representation, and outputs the alignment information corresponding to the text features and the emotion representation; generates a target Mel spectrum according to the alignment information, the emotion representation and the text features, and generates emotional speech according to the target Mel spectrum, thereby completing the synthesis of emotional speech.

[0028] By extracting speech information and text information, the provided method can align and fuse the emotion representation corresponding to the language information and the text features corresponding to the text information in time, and then train the relevant model through an end-to-end method without labeling the training samples to achieve accurate synthesis of emotional speech.

[0029] Furthermore, the method provided solves the problem of difficulty in obtaining emotional annotation data by realizing unlabeled emotional speech synthesis, and can directly learn the role of emotional representation in speech synthesis models. Not only that, the entire method can also realize speech synthesis of any number of emotional styles according to needs.

[0030] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0032] Figure 1 is a schematic flow chart of the steps of an emotional speech synthesis method provided by an embodiment of the present application;

[0033] Figure 2 is a schematic flow chart of the steps of a model training method provided in one embodiment of the present application;

[0034] Figure 3 is a schematic flow chart of the steps of a cross entropy combined training method provided in one embodiment of the present application;

[0035] Figure 4 is a structural schematic diagram of an emotional speech synthesis device provided in an embodiment of the present application;

[0036] Figure 5 It is a schematic block diagram of the structure of a computer device provided in one embodiment of the present application.

[0037] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. DETAILED DESCRIPTION

[0038] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0039] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

[0040] It should be understood that, in order to clearly describe the technical solutions of the embodiments of the present utility model, in the embodiments of the present utility model, words such as "first" and "second" are used to distinguish the same or similar items with substantially the same functions and effects. Those skilled in the art can understand that words such as "first" and "second" do not limit the quantity and execution order, and words such as "first" and "second" do not necessarily limit the difference.

[0041] It should be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in this application specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0042] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0043] To facilitate understanding of the contents of this application specification, some words that appear in this application are now explained.

[0044] 1. Flow-based model: The flow model can flow from distribution A to distribution B like a stream. The distribution transformation of the flow model is staged, that is, the model transforms from one distribution to another specified distribution step by step.

[0045] At the same time, the flow model is also reversible. Specifically, it can not only transform from distribution A to distribution B, but also from distribution B to distribution A.

[0046] 2. Linear spectrum: Perform sliding window Fourier transform on the speech signal in the time domain. Before Fourier transform, a window function is used to prevent energy leakage, usually the Hanning window function. After short-time Fourier transform processing, the linear spectrum of the speech signal is obtained. The linear spectrum dimension is relatively high and is generally set to 1024.

[0047] 3. Latent variable: a comprehensive variable with characteristics that cannot be directly observed. Unobservable, or "hidden" is its appearance; comprehensiveness is its essence.

[0048] Corresponding to latent variables, we call ordinary variables manifest variables or observed variables.

[0049] 4. Forced alignment module: The forced alignment module is a commonly used module in speech tasks. It is widely used in frame-level speech recognition and speech synthesis tasks. It is also the core algorithm in tasks such as automatic subtitle reeling and oral evaluation. Forced alignment and recognition decoding have certain similarities. Recognition decoding means that given a speech recognition model, a speech sequence is input, and the corresponding recognition text is output. Forced alignment means that given a speech recognition model, a speech sequence and a correct annotated text (which can be a text sequence or a phoneme sequence, etc.) are input, and the correspondence between the speech frame and the text is given, such as the 20th to 25th frames correspond to the phoneme 'a'.

[0050] 5. End-to-end learning: In traditional machine learning solutions, it is often composed of multiple independent modules. For example, in a typical natural language processing problem, it includes multiple steps such as word segmentation, part-of-speech tagging, syntactic analysis, and semantic analysis. Each step is an independent task module that needs to be labeled and trained separately. At the same time, since the result of the previous step will serve as the condition for the next step, the effect of the previous step will affect the next step, thereby affecting the entire solution.

[0051] During the training process of a deep learning model, a prediction result will be obtained from the input to the output. By comparing it with the actual observation result, an error will be obtained. This error will be transmitted (back propagated) in each layer of the model, and the representation of each layer will be adjusted according to this error until the model converges or achieves the expected effect. This is end-to-end learning. Compared with traditional machine learning, end-to-end learning eliminates the process of labeling and training multiple independent task modules separately, and also reduces the possibility of errors caused by too many intermediate links.

[0052] 6. Text encoder: The text encoder is used to extract text features.

[0053] 7. Duration predictor: Generally speaking, the relationship between text and speech is of unequal length. A single text feature takes up one or more speech frames. The role of the duration predictor is to establish an alignment relationship between the input text and the output text.

[0054] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0055] With the development of artificial intelligence technology, speech synthesis plays an increasingly important role in the field of human-computer dialogue. As an important supplement to speech synthesis, emotional speech synthesis has greatly improved the application scenarios of speech synthesis. According to different scenarios and environments, switching corresponding emotional expressions can greatly improve the user experience when deploying intelligent customer service in financial systems by synthesizing infectious speech for intelligent customer service.

[0056] Traditional emotional speech synthesis requires recording a large amount of emotion and style-related data based on annotated synthetic corpus. This leads to the following problems: First, it is very difficult to record emotional data, and the emotional sound library of a single speaker cannot be transferred to other sound libraries; second, it is more difficult to annotate emotions. Emotions are relatively subjective expression characteristics, and different people have different cognition and understanding of the expression of different emotions. Based on the above two points, the traditional speech emotion synthesis system is greatly restricted by data annotation conditions, resulting in poor synthesized speech effects.

[0057] To solve the above problems, please refer to Figure 1 , Figure 1 1 is a schematic flow chart of an emotional speech synthesis method provided by an embodiment of the present application. The emotional speech synthesis method can be implemented by a computer device, and the computer device can be deployed on a single server or a server cluster. It can also be deployed on a handheld terminal, a laptop computer, a wearable device or a robot, etc.

[0058] The provided method can be used in the financial field to synthesize emotional speech of financial intelligent customer service, making the artificial intelligence customer service more humanized and giving users a better service experience. It can also be applied to any field that needs to deploy artificial intelligence customer service, such as medical care and education, and this application does not limit this.

[0059] To solve the above problems, please refer to Figure 1 Specifically, Figure 1 As shown, the provided method includes steps S101 to S105. The details are as follows:

[0060] S101. Acquire voice information and text information to be synthesized.

[0061] Specifically, before the server synthesizes the speech, this application obtains unlabeled speech information and text information. The speech information contains emotional representations that are hoped to be added through emotional speech synthesis, such as the voice of a customer service staff member who serves as a template in a preset financial system, so that the final synthesized emotional speech can have the emotions of the customer service staff member, thereby improving the user experience.

[0062] S102. Input the speech information into a pre-trained emotion extraction model. The emotion extraction model analyzes the speech information and outputs the emotion representation corresponding to the speech information.

[0063] Specifically, the server inputs the unlabeled voice information into the pre-trained emotion extraction model, which can represent the emotion in the voice information, and this representation is used as the emotion label corresponding to this voice information. The emotion representation of a voice message is transformed from a discrete label information (happy, afraid, calm, etc.) to a fixed-dimensional vector to represent the emotion representation of this voice. In this way, the discrete data can be continuous, which is convenient for the subsequent accurate synthesis of the emotion representation.

[0064] S103. Input the text information into a preset text encoder, and the text encoder parses the text information and outputs text features corresponding to the text information.

[0065] Specifically, the server can extract text features corresponding to the text information, such as corresponding feature vectors, by inputting the text information to be synthesized into a preset text encoder. Text features include information such as the relevance of each word in the text, namely Tf-idf (term frequency-inverse document frequency). By dividing the number of occurrences of a word by the number of documents containing the word. When a word appears frequently in a text but not in other texts, it must be important to that particular file. A vector using single-hot encoding contains zeros except for the first index, and the first index uniquely identifies each word. At the same time, text information can also be converted into corresponding text features in the form of word2vec (word to vector).

[0066] S104. Input the text features and sentiment representation into the pre-trained duration predictor, the duration predictor parses the text features and sentiment representation, and outputs alignment information corresponding to the text features and sentiment representation.

[0067] Specifically, in the process of emotional speech synthesis, the relationship between text and speech is unequal in length. A single text feature takes up one or more speech frames. This application can establish an alignment relationship between the input text and the output text by setting a duration predictor, thereby ensuring that the synthesized emotional speech has better naturalness.

[0068] S105. Generate a target Mel spectrum according to the alignment information, the emotion representation and the text feature, generate emotional speech according to the target Mel spectrum, and complete the synthesis of emotional speech.

[0069] Specifically, after obtaining the alignment information, emotion representation and text features, the server can generate target mel-spectrum corresponding to the alignment information, emotion representation and text features, and input the preset speech generator or stream model to generate emotional speech that is full of emotion and highly natural. The provided method can complete the synthesis of any single or multiple styles of emotional speech without labels.

[0070] In some embodiments, Figure 2 As shown, before the voice information is input into the pre-trained emotion extraction model, steps S201 to S206 are also included.

[0071] Step S201. Acquire the synthesized historical emotional speech, historical speech information and historical text information corresponding to the historical emotional speech;

[0072] Step S202: inputting the historical emotional speech into the emotion extraction model to be trained, and outputting the historical emotional representation corresponding to the historical emotional speech;

[0073] Step S203. Input the historical text information into the pre-trained text encoder, and output the historical text features corresponding to the historical text information;

[0074] Step S204: inputting the historical sentiment representation and the historical text features into a pre-trained duration predictor, and the duration predictor outputs historical alignment information corresponding to the historical text features and the historical sentiment representation;

[0075] Step S205. Generate a predicted Mel spectrum according to the historical alignment information, the historical emotion representation and the historical text features, and generate a predicted emotional speech according to the predicted Mel spectrum;

[0076] Step S206: Complete the training of the emotion extraction model based on the historical emotional speech and the predicted emotional speech.

[0077] The provided method can train the emotion extraction model based on unlabeled samples by obtaining the synthesized historical emotional speech and its corresponding historical speech information and historical text information before synthesis, that is, the training of the emotion extraction model is completed through the reconstruction loss between the speech and the real speech.

[0078] Exemplarily, before inputting historical text information and historical text features into a pre-trained duration predictor, it also includes: obtaining linear spectrum information corresponding to historical emotional speech, inputting the linear spectrum information into a preset posterior encoder, the posterior encoder parsing the linear spectrum information, and outputting latent variable information and acoustic features corresponding to the linear spectrum information; inputting the latent variable information and historical text information into a preset forced alignment module, the forced alignment module outputs forced alignment information corresponding to the latent variable information and the historical text information; inputting historical emotional representation and historical text features into the duration predictor to be trained, the duration predictor outputs predicted alignment information corresponding to the historical text features and the historical emotional representation, and completing the training of the duration predictor according to the predicted alignment information and the forced alignment information.

[0079] By extracting latent variable information corresponding to the linear spectrum information of historical emotional speech, the provided method can simultaneously combine the forced alignment module to determine the forced alignment information corresponding to the latent variable information and the historical text information, so that the trained duration predictor can also learn to synthesize text features and emotional representations based on the latent variable information. At the same time, the provided method can realize end-to-end training of the emotion extraction model and the duration predictor.

[0080] It should be noted that, in some embodiments, before generating emotional speech according to the target Mel spectrum, it also includes: inputting the acoustic features into the speech decoder to be trained, the speech decoder parses the acoustic features, and outputs the predicted speech information corresponding to the acoustic features; completing the training of the speech decoder according to the historical speech information and the predicted speech information, so as to input the target Mel spectrum into the trained speech decoder and output the emotional speech.

[0081] By extracting the acoustic features of speech information (such as Mel-spectrogram information), end-to-end training of the speech decoder can also be achieved, thereby ensuring that the methods provided in emotion extraction, speech alignment and speech synthesis can achieve ideal results.

[0082] It should be noted that, in some embodiments, a target Mel spectrum is generated according to alignment information, emotion representation and text features, and emotional speech is generated according to the target Mel spectrum, including: inputting the alignment information, emotion representation and text features into a preset stream model, and the stream model outputs the target Mel spectrum; wherein the posterior encoder parses the linear spectrum information, and outputs the latent variable information corresponding to the linear spectrum information as simple latent variable information; before inputting the latent variable information and historical text information into a preset forced alignment module, it also includes: inversely inputting the simple latent variable information into the stream model, and the stream model outputs the complex latent variable information corresponding to the simple latent variable information, so as to input the complex latent variable information into the forced alignment module.

[0083] As the distribution of latent variables is relatively complex, it is difficult to directly sample them in the generation process. The method provided converts the complex distribution of latent variables into a simple distribution (such as Gaussian distribution), that is, the simple latent variable information is first output through the posterior encoder, and then combined with the inverse generation characteristics of the flow model, the generation process only needs to sample in the simple distribution and then obtain the complex latent variable information through the inverse process of the flow model. This improves the end-to-end training efficiency of modules such as the emotion extraction model and the duration predictor while also simplifying the training steps.

[0084] It should be noted that, in some embodiments, before inputting the historical text information into the pre-trained text encoder, it also includes: inputting the historical text information into the text encoder to be trained, outputting the predicted text features corresponding to the historical text information to obtain the relative entropy information of the acoustic features and the predicted text features; and completing the training of the text encoder according to the relative entropy information.

[0085] By combining the relative entropy (also called KL distance (Kull back-Leibl er Di vergence)) information between the acoustic features output by the posterior encoder and the text features of the text encoder, the text encoder can also be accurately trained end-to-end to ensure the accuracy of text feature extraction.

[0086] For example, Figure 3 As shown, the emotion extraction model is trained based on the historical emotion speech and the predicted emotion speech, including steps S206a to S206c.

[0087] Step S206a. Obtain the historical Mel-spectrogram corresponding to the historical emotional speech;

[0088] Step S206b. Obtain cross loss information between the historical Mel spectrum and the predicted Mel spectrum;

[0089] Step S206c. Complete the training of the emotion extraction model according to the cross loss information, the historical emotional speech and the predicted emotional speech.

[0090] The provided method can improve the accuracy of model training by combining the reconstruction loss between the generated speech and the real speech and the cross loss between the Mel-spectrogram features of the generated speech and the real Mel-spectrogram, ensuring that accurate emotion representation can be extracted from the speech information and improving the accuracy of the final synthesized speech information.

[0091] The present application discloses a method for synthesizing emotional speech. The method obtains speech information and text information to be synthesized; inputs the speech information into a pre-trained emotion extraction model, the emotion extraction model parses the speech information, and outputs the emotion representation corresponding to the speech information; inputs the text information into a preset text encoder, the text encoder parses the text information, and outputs the text features corresponding to the text information; inputs the text features and the emotion representation into a pre-trained duration predictor, the duration predictor parses the text features and the emotion representation, and outputs the alignment information corresponding to the text features and the emotion representation; generates a target Mel spectrum according to the alignment information, the emotion representation and the text features, generates emotional speech according to the target Mel spectrum, and completes the synthesis of emotional speech.

[0092] By extracting speech information and text information, the provided method can align and fuse the emotion representation corresponding to the language information and the text features corresponding to the text information in time, thereby achieving accurate synthesis of emotional speech without labeling the training samples.

[0093] Furthermore, the method provided solves the problem of difficulty in obtaining emotional annotation data by realizing unlabeled emotional speech synthesis, and can directly learn the role of emotional representation in speech synthesis models. Not only that, the entire method can also realize speech synthesis of any number of emotional styles according to needs.

[0094] See also Figure 4 As shown, Figure 4 : is a schematic diagram of the structure of the emotional speech synthesis device 300 provided in an embodiment of the present application. The emotional speech synthesis device 300 is used to execute the steps of the emotional speech synthesis method shown in the above embodiments. The emotional speech synthesis device 300 can be a single server or a server cluster, or the emotional speech synthesis device 300 can be a terminal, which can be a handheld terminal, a laptop computer, a wearable device or a robot, etc.

[0095] like Figure 4 As shown, the emotional speech synthesis device 300 includes:

[0096] Information acquisition unit 301, voice acquisition of voice information and text information to be synthesized;

[0097] The representation extraction unit 302 is used to input the speech information into the pre-trained emotion extraction model, and the emotion extraction model analyzes the speech information and outputs the emotion representation corresponding to the speech information;

[0098] The feature output unit 303 is used to input the text information into a preset text encoder, and the text encoder parses the text information and outputs text features corresponding to the text information;

[0099] The duration prediction unit 304 is used to input the text features and the sentiment representation into a pre-trained duration predictor, and the duration predictor analyzes the text features and the sentiment representation and outputs alignment information corresponding to the text features and the sentiment representation;

[0100] The speech synthesis unit 305 is used to generate a target Mel spectrum according to the alignment information, the emotion representation and the text feature, generate the emotional speech according to the target Mel spectrum, and complete the synthesis of the emotional speech. It should be noted that those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process of the emotional speech synthesis device and each module described above can refer to the corresponding process in the emotional speech synthesis method embodiment described in the above embodiments, and will not be repeated here.

[0101] The above-mentioned emotional speech synthesis method can be implemented in the form of a computer program. Figure 4 Run on the device shown.

[0102] See also Figure 5 , Figure 5 1 is a schematic block diagram of the structure of a computer device provided in an embodiment of the present application. The computer device includes a processor, a memory and a network interface connected via a device bus, wherein the memory may include a storage medium and an internal memory.

[0103] The storage medium can store an operating device and a computer program. The computer program includes program instructions, and when the program instructions are executed, the processor can execute any emotional speech synthesis method.

[0104] The processor is used to provide computing and control capabilities and support the operation of the entire computer equipment.

[0105] The internal memory provides an environment for the operation of the computer program in the non-volatile storage medium. When the computer program is executed by the processor, the processor can execute any emotional speech synthesis method.

[0106] The network interface is used for network communication, such as sending assigned tasks, etc. Those skilled in the art will appreciate that Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the scheme of the present application, and does not constitute a limitation on the terminal to which the scheme of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0107] It should be understood that the processor may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0108] In one embodiment, the processor is used to run a computer program stored in the memory to implement the following steps:

[0109] Obtaining voice information and text information to be synthesized;

[0110] The speech information is input into the pre-trained emotion extraction model, and the emotion extraction model analyzes the speech information and outputs the emotion representation corresponding to the speech information;

[0111] Input the text information into a preset text encoder, the text encoder parses the text information, and outputs text features corresponding to the text information;

[0112] Input text features and sentiment representations into the pre-trained duration predictor, which parses the text features and sentiment representations and outputs alignment information corresponding to the text features and sentiment representations;

[0113] A target Mel spectrum is generated according to the alignment information, sentiment representation and text features, and emotional speech is generated according to the target Mel spectrum to complete the synthesis of emotional speech.

[0114] In some embodiments, before inputting speech information into a pre-trained emotion extraction model, it also includes: obtaining synthesized historical emotion speech, historical speech information corresponding to the historical emotion speech, and historical text information; inputting the historical emotion speech into the emotion extraction model to be trained, and outputting the historical emotion representation corresponding to the historical emotion speech; inputting the historical text information into a pre-trained text encoder, and outputting the historical text features corresponding to the historical text information; inputting the historical emotion representation and the historical text features into a pre-trained duration predictor, and the duration predictor outputs the historical alignment information corresponding to the historical text features and the historical emotion representation; generating a predicted Mel spectrum based on the historical alignment information, the historical emotion representation, and the historical text features, and generating a predicted emotion speech based on the predicted Mel spectrum; and completing the training of the emotion extraction model based on the historical emotion speech and the predicted emotion speech.

[0115] Exemplarily, before inputting historical text information and historical text features into a pre-trained duration predictor, it also includes: obtaining linear spectrum information corresponding to historical emotional speech, inputting the linear spectrum information into a preset posterior encoder, the posterior encoder parsing the linear spectrum information, and outputting latent variable information and acoustic features corresponding to the linear spectrum information; inputting the latent variable information and historical text information into a preset forced alignment module, the forced alignment module outputs forced alignment information corresponding to the latent variable information and the historical text information; inputting historical emotional representation and historical text features into the duration predictor to be trained, the duration predictor outputs predicted alignment information corresponding to the historical text features and the historical emotional representation, and completing the training of the duration predictor according to the predicted alignment information and the forced alignment information.

[0116] It should be noted that, in some embodiments, before generating emotional speech according to the target Mel spectrum, it also includes: inputting the acoustic features into the speech decoder to be trained, the speech decoder parses the acoustic features, and outputs the predicted speech information corresponding to the acoustic features; completing the training of the speech decoder according to the historical speech information and the predicted speech information, so as to input the target Mel spectrum into the trained speech decoder and output the emotional speech.

[0117] It should be noted that, in some embodiments, a target Mel spectrum is generated according to alignment information, emotion representation and text features, and emotional speech is generated according to the target Mel spectrum, including: inputting the alignment information, emotion representation and text features into a preset stream model, and the stream model outputs the target Mel spectrum; wherein the posterior encoder parses the linear spectrum information, and outputs the latent variable information corresponding to the linear spectrum information as simple latent variable information; before inputting the latent variable information and historical text information into a preset forced alignment module, it also includes: inversely inputting the simple latent variable information into the stream model, and the stream model outputs the complex latent variable information corresponding to the simple latent variable information, so as to input the complex latent variable information into the forced alignment module.

[0118] It should be noted that, in some embodiments, before inputting the historical text information into the pre-trained text encoder, it also includes: inputting the historical text information into the text encoder to be trained, outputting the predicted text features corresponding to the historical text information to obtain the relative entropy information of the acoustic features and the predicted text features; and completing the training of the text encoder according to the relative entropy information.

[0119] Exemplarily, the training of the emotion extraction model is completed based on historical emotional speech and predicted emotional speech, including: obtaining historical Mel spectrum corresponding to the historical emotional speech; obtaining cross-loss information of the historical Mel spectrum and the predicted Mel spectrum; and completing the training of the emotion extraction model based on the cross-loss information, historical emotional speech and predicted emotional speech.

[0120] A computer-readable storage medium is also provided in an embodiment of the present application, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and the processor executes the program instructions to implement the steps of the emotional speech synthesis method provided in the above embodiments of the present application.

[0121] The computer-readable storage medium may be an internal storage unit of the computer device described in the foregoing embodiment, such as a hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a smart memory card (SmartMedia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc., equipped on the computer device.

[0122] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. An emotional speech synthesis method, characterized in that: include: Obtaining voice information and text information to be synthesized; Inputting the speech information into a pre-trained emotion extraction model, the emotion extraction model parses the speech information and outputs the emotion representation corresponding to the speech information; Inputting the text information into a preset text encoder, the text encoder parses the text information and outputs text features corresponding to the text information; Inputting the text features and the sentiment representation into a pre-trained duration predictor, the duration predictor parses the text features and the sentiment representation, and outputs alignment information corresponding to the text features and the sentiment representation; A target Mel spectrum is generated according to the alignment information, the emotion representation and the text feature, and an emotional speech is generated according to the target Mel spectrum, thereby completing the synthesis of the emotional speech.

2. The method according to claim 1, characterized in that Before inputting the speech information into the pre-trained emotion extraction model, the method further includes: Acquire the synthesized historical emotional speech, historical speech information and historical text information corresponding to the historical emotional speech; Inputting the historical emotional speech into the emotion extraction model to be trained, and outputting the historical emotional representation corresponding to the historical emotional speech; Inputting the historical text information into a pre-trained text encoder, and outputting historical text features corresponding to the historical text information; Inputting the historical sentiment representation and the historical text features into a pre-trained duration predictor, the duration predictor outputting historical alignment information corresponding to the historical text features and the historical sentiment representation; Generate a predicted Mel spectrum according to the historical alignment information, the historical emotion representation and the historical text features, and generate a predicted emotional speech according to the predicted Mel spectrum; The emotion extraction model is trained according to the historical emotion speech and the predicted emotion speech.

3. The method according to claim 2, characterized in that Before inputting the historical text information and historical text features into the pre-trained duration predictor, the method further includes: Acquire linear spectrum information corresponding to the historical emotional speech, input the linear spectrum information into a preset posterior encoder, the posterior encoder parses the linear spectrum information, and outputs latent variable information and acoustic features corresponding to the linear spectrum information; Inputting the latent variable information and the historical text information into a preset forced alignment module, and the forced alignment module outputs forced alignment information corresponding to the latent variable information and the historical text information; The historical sentiment representation and the historical text features are input into a duration predictor to be trained, and the duration predictor outputs the prediction alignment information corresponding to the historical text features and the historical sentiment representation The duration predictor is trained according to the prediction alignment information and the forced alignment information.

4. The method according to claim 3, characterized in that Before generating emotional speech according to the target Mel spectrum, the method further includes: Inputting the acoustic features into a speech decoder to be trained, the speech decoder parses the acoustic features and outputs predicted speech information corresponding to the acoustic features; The speech decoder is trained according to the historical speech information and the predicted speech information, so as to input the target Mel spectrum into the trained speech decoder and output the emotional speech.

5. The method according to claim 3, characterized in that: The step of generating a target Mel spectrum according to the alignment information, the emotion representation and the text feature, and generating emotional speech according to the target Mel spectrum includes: Inputting the alignment information, sentiment representation and text features into a preset stream model, and the stream model outputs the target Mel spectrum; The posterior encoder parses the linear spectrum information and outputs the latent variable information corresponding to the linear spectrum information as simple latent variable information; before inputting the latent variable information and the historical text information into a preset forced alignment module, the method further includes: The simple latent variable information is inversely input into the flow model, and the flow model outputs the complex latent variable information corresponding to the simple latent variable information, so as to input the complex latent variable information into the forced alignment module.

6. The method according to claim 3, characterized in that Before inputting the historical text information into the pre-trained text encoder, the method further includes: Input the historical text information into the text encoder to be trained, and output the predicted text features corresponding to the historical text information Obtaining relative entropy information of the acoustic features and the predicted text features; The text encoder is trained based on the relative entropy information.

7. The method according to claim 2, characterized in that The training of the emotion extraction model according to the historical emotion speech and the predicted emotion speech includes: Obtaining a historical Mel spectrum corresponding to the historical emotional speech; Obtaining cross loss information between the historical Mel spectrum and the predicted Mel spectrum; The emotion extraction model is trained according to the cross loss information, the historical emotional speech and the predicted emotional speech.

8. An emotional speech synthesis device, characterized in that: include: An information acquisition unit, which acquires voice information and text information to be synthesized; A representation extraction unit, configured to input the speech information into a pre-trained emotion extraction model, wherein the emotion extraction model analyzes the speech information and outputs an emotion representation corresponding to the speech information; A feature output unit, used for inputting the text information into a preset text encoder, wherein the text encoder parses the text information and outputs text features corresponding to the text information; A duration prediction unit, used for inputting the text features and the sentiment representation into a pre-trained duration predictor, wherein the duration predictor parses the text features and the sentiment representation and outputs alignment information corresponding to the text features and the sentiment representation; The speech synthesis unit is used to generate a target Mel spectrum according to the alignment information, the emotion representation and the text feature, generate emotional speech according to the target Mel spectrum, and complete the synthesis of the emotional speech.

9. A computer device, characterized in that: The computer device includes a memory and a processor; The memory is used to store computer programs; The processor is configured to execute the computer program and implement the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, causes the processor to implement the method according to any one of claims 1 to 7.

Images