Emotion transfer speech synthesis method and system
By combining text encoder and emotion extraction module in speech synthesis, the spokesperson information and emotional information are decoupled and the Mel voice spectrum characteristics are generated, and the problem of emotional encoding vector entanglement information in the prior art is solved, and the clarity of emotional information is achieved. Extraction and control improve the expressiveness of speech synthesis.
Patent Information
- Application Number
- CN202210414220.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-04-20
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2042-04-20
AI Technical Summary
The prior art is difficult to effectively decouple spokesperson information and emotional information in speech synthesis, resulting in the emotional coding vector containing too much entangled information, lack of clear control of emotional information, and it is difficult to create new synthetic speech in the multi-speaker corpus.
Using the emotional transfer speech synthesis method, through the text encoder, emotion extraction module and text speech frame alignment module, the text encoding vector is combined with the emotional style vector to generate the Mel's voice spectrum features, realizing independent extraction and control of emotional information.
It realizes clear extraction and decoupling of emotional information, improves the similarity of emotional coded vectors, and can freely combine spokesperson information to generate emotional voice of the target spokesperson, and can easily control emotional strength.
Smart Images

Figure CN114842825B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of speech synthesis, and in particular to an emotion transfer speech synthesis method and system. Background Art
[0002] A deep learning-based speech synthesis (TTS, text-to-speech) model is usually composed of an encoder-decoder neural network that is trained to successfully map a given text sequence to a speech frame sequence. In addition, the model can effectively model and control speech style (such as the speaker's identity, speaking style, emotion, and prosody).
[0003] Many current TTS application scenarios, such as audiobook narration, news broadcasting, conversation assistants, etc., require single-speaker multi-style speech synthesis. For example, a synthesized speaker can speak with multiple emotions such as happiness, sadness, fear, etc. However, it is difficult to collect multi-style speech data of a single speaker, so the corresponding performance in emotional speech synthesis is insufficient.
[0004] Currently, most neural network-based neural TTS models are trained using expressive single-emotion style corpora (e.g., emotionless, bland speech data). Obtaining a large amount of single-speaker speech data with multiple styles is useful for training a good TTS system, but data collection is usually expensive and time-consuming. In speech emotion synthesis, there are two key elements that need to be addressed: First, the synthesized speech should convey emotions correctly and accurately. Specifically, the emotions conveyed by the synthesized speech should be easily perceived by the listener without confusion. Second, controlling emotional expression should be done in a more flexible and reliable way.
[0005] An effective solution is to use speech style transfer, which allows the target speaker to accurately learn the required emotional information from speech data of the same style but recorded by other speakers, while retaining the target speaker's timbre. In addition, expressive speech can be generated by combining various emotional styles and timbres from a multi-speaker corpus.
[0006] In the field of multi-emotional style synthesis, there are usually the following solutions:
[0007] First of all, the most direct solution is to directly add emotional style labels to the TTS model. This label directly participates in speech synthesis as the input of the TTS model to control the type of emotion in the synthesized speech.
[0008] Secondly, the cross-speaker style transfer model based on encoding vectors is usually based on several general encoding embedding methods (reference encoders [1] Towards end-to-end prosody transfer for expressive speech synthes is with Tacotron [2] Control abstraction transfer for end-to-end speech synthes is, global style tags (GST) [3] Style tokens: Unsupervised style mode limiting, control and transfer in end-to-end speech synthes is and variational autoencoder (VAE) [4] Learning interesting representations for style control and transfer in end-to-end speech synthes is). These methods extract encoding vectors representing emotional information from emotional speech, add them to the TTS system through simple or complex splicing methods, and can meet the requirements of emotional style transfer.
[0009] However, the existing technology has the following disadvantages:
[0010] (1) Adding emotional style labels: This task relies heavily on training data and requires the speaker to have a full range of emotional data. At the same time, this solution cannot combine the timbre and style of different speakers to create new synthesized speech, nor can it directly control the strength of emotions.
[0011] (2) Cross-speaker model based on encoding vector: First, the style encoding vectors obtained by these methods often contain too much entangled information (including speaker identity, emotion, speaking style, speaking speed, etc.), and lack effective interpretation of emotional information.
[0012] Secondly, the emotions expressed in the synthesized audio are often averaged emotions, which are not obvious enough; and it is also difficult to select the appropriate emotional speech to extract the encoding vector. In addition, the control of specific speech features (such as speaker identity and speaking style) is not effective enough.
[0013] Finally, in emotional style transfer, these methods often transfer all the style patterns of the source speaker (including speaking style, speaking speed, emotional expression, etc.), which does not match the need to transfer emotions and seriously affects the generalization ability of the model. When performing style control, it is difficult to find a direct relationship between the target style and the parameters in the encoding vector of the style representation to control the style change.
[0014] In view of the above-mentioned defects of the existing emotional speech synthesis, the present invention proposes an emotional speech synthesis method and system. Summary of the invention
[0015] The purpose of the present invention is to provide an emotion transfer speech synthesis system to solve the problems of the prior art: most of them use a solution of extracting emotion feature sequences in audio using an emotion feature extraction module to represent emotion information. These methods often have too much entangled information, and the independent control of specific speech features is not clear and direct.
[0016] In order to achieve the above object, the present invention adopts the following technical solutions:
[0017] The emotion transfer speech synthesis method comprises the following steps:
[0018] S1. Input the text input sequence with phoneme level into the text encoder to obtain the text encoding vector;
[0019] S2. Obtain the corresponding emotion category number according to the emotion category required by the speech, input it into the emotion extraction module, and obtain the emotion style vector;
[0020] S3. The text encoding vector and the emotional style vector are passed through a text-speech frame alignment module to obtain a text-speech alignment sequence;
[0021] S4. Input the speaker ID into the speech frame decoder, process the text-speech alignment sequence through the speech frame decoder, and decode to obtain the Mel-spectrograph features.
[0022] Further preferably, S1 comprises the following steps:
[0023] The phoneme-level text input sequence is processed through the character embedding layer and encoded into a fixed-dimensional sequence vector;
[0024] The phoneme-level text input sequence is passed through the position encoding layer to construct a position vector;
[0025] The fixed-dimensional sequence vector and position vector are added and sent to the multi-layer FFTB lock module to be converted into a fixed-length text encoding vector.
[0026] Further preferably, the extraction process of the emotion extraction module in S2 includes the following steps:
[0027] Obtain the Mel-spectrogram feature of the emotional speech, and input the Mel-spectrogram feature into the reference encoder layer, and encode it into a vector of fixed length, which is recorded as the reference embedding vector;
[0028] Input the reference embedding vector into the attention module to calculate multiple emotion expression vectors; the emotion expression vectors are labeled by emotion category numbers, and during the extraction process, the emotion expression vectors each represent an emotion category;
[0029] The emotional style vector is formed by weighted combination of multiple emotional expression vectors.
[0030] Further preferably, S2 comprises the following steps:
[0031] The emotion category number is converted into the emotion expression vector through the attention module;
[0032] The emotional expression vectors are weighted combined to form the emotional style vector.
[0033] Further preferably, S3 comprises the following steps:
[0034] By adding the emotion style vector and the text encoding vector correspondingly, a combined text encoding vector is obtained;
[0035] The duration predictor is used to predict the duration of each text phoneme in the combined text encoding vector. The duration predictor uses a two-layer 1D convolutional neural network, and a fully connected layer is connected to the last layer. Its input is the combined text encoding vector. The prediction model predicts the duration value of the speech for each text encoding in the vector through multi-layer convolution operations.
[0036] Through the LR module, each phoneme is aligned to the speech frame length according to the predicted duration, and the speech frame length encoding vector is obtained;
[0037] The speech frame length coding vector is passed through the fundamental frequency predictor and energy predictor in turn to obtain the fundamental frequency prediction value and energy prediction value; the duration predictor, fundamental frequency predictor, and energy predictor use the same structure, all using a 2-layer 1D convolutional neural network, and the last layer is connected to a fully connected layer. The fundamental frequency predictor / energy predictor accepts the speech frame length coding vector and predicts the speech fundamental frequency value / speech energy value corresponding to each frame respectively;
[0038] The fundamental frequency prediction value, energy prediction value and speech frame length coding vector are added together to obtain a text-speech alignment sequence.
[0039] Further preferably, S4 comprises the following steps:
[0040] Pass the text-speech alignment sequence through the position vector layer to construct the position vector encoding;
[0041] The position vector code and the corresponding position of the alignment code sequence are added, and the speaker ID value is concatenated with the position vector code after adding the corresponding position of the alignment code sequence. The concatenated result is decoded by a multi-layer FFTB lock module.
[0042] The obtained decoding feature sequence is passed through a post-processing module to obtain the Mel-spectrogram feature; the post-processing module is a 5-layer 1-dimensional convolutional network stack, which calculates the decoding feature sequence through the convolutional network multiple times, and the obtained 80-dimensional encoding vector is the Mel-spectrogram feature.
[0043] Emotion transfer speech synthesis system, including:
[0044] Text encoder, sentiment extraction module, text-speech frame alignment module, speech frame decoder.
[0045] Furthermore, the text encoder comprises: a character embedding layer, a position encoding layer and a FFTB lock module;
[0046] The emotion extraction module includes: a reference encoder layer and an attention module;
[0047] The text-to-speech frame alignment module includes: a duration predictor, an LR module, a fundamental frequency predictor, and an energy predictor;
[0048] The speech frame decoder comprises: a position coding layer, a FFTB lock module, and a post-processing module.
[0049] The present invention has at least the following beneficial effects:
[0050] The emotion information extraction module proposed in the present invention can, firstly, completely decouple the speaker information and emotion information in the audio features. The emotion coding vector only contains the emotion information in the audio and improves the similarity between the coding vector and the emotion information represented by the vector. Secondly, the emotion coding vector can be freely combined with the speaker information to realize the task of transferring the emotion information from the audio data of the source speaker to the target speaker without emotion. By assigning weights to the emotion coding vector, it can be easier to control the strength of the emotion. BRIEF DESCRIPTION OF THE DRAWINGS
[0051] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying any creative work.
[0052] Figure 1 This is the overall architecture diagram;
[0053] Figure 2 for text encoder workflow;
[0054] Figure 3 Workflow of the text-to-speech frame alignment module;
[0055] Figure 4 It is the workflow of speech frame decoder;
[0056] Figure 5 Extraction and production process for sentiment extraction module;
[0057] Figure 6 The overall execution process of the system. DETAILED DESCRIPTION
[0058] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.
[0059] See also Figure 1 , which is the overall architecture diagram of the model of the present invention.
[0060] The emotion transfer TTS system is a sequence-sequence structure based on a multi-layer neural network. The system inputs a text sequence at the phoneme level, a specified speaker number, and a specified emotion category number, and outputs sequence information at the speech frame level.
[0061] The system mainly consists of four modules: text encoder module, text-speech frame alignment module, speech frame decoder module and emotion information extraction module.
[0062] exist Figure 1 In the model architecture diagram, the text encoder module and the sentiment information extraction module are spliced together into the alignment module, and then the output of the alignment module enters the speech frame decoder module to obtain the final decoded speech frame. The following is a detailed introduction to these four modules.
[0063] (1) Text encoder, see Figure 2 , Figure 2 Demonstrates the text encoder workflow.
[0064] The role of the text encoder is to transform the phoneme-level text sequence into a machine-readable text encoding vector through a neural network.
[0065] The text encoder consists of a character embedding layer, a position encoding layer, and a FFTB lock module (which consists of multiple convolutional layers and a self-attention mechanism layer). The specific operation process is as follows:
[0066] a. Process the phoneme-level text input sequence through the character embedding layer and encode it into a fixed-dimensional sequence vector;
[0067] b. Construct a position vector from the phoneme-level text input sequence through the position encoding layer;
[0068] c. Add the encoded text sequence vector and the corresponding position of the position vector, and send the added result to the multi-layer FFTBlock module to be converted into a text encoding vector of fixed length.
[0069] (2) Text-to-speech frame alignment module, see Figure 3 , Figure 3 The workflow of the alignment fusion module is shown.
[0070] The alignment module changes the text encoding vector from the phoneme length to the speech frame length to obtain the text-speech alignment sequence. This module contains three predictors: duration predictor, fundamental frequency predictor and energy predictor, which respectively predict the duration of each phoneme, the average energy value corresponding to each phoneme and the average fundamental frequency value corresponding to each phoneme for the phoneme encoding sequence. The LR module aligns the phoneme sequence encoding vectors to the speech frame length one by one through the predicted duration value. The output of the alignment module is the sum of the three prediction values. The duration predictor, fundamental frequency predictor and energy predictor all use a 2-layer 1D convolutional neural network, and the fully connected layer is connected to the last layer. The duration predictor predicts the speech duration value for each text encoding in the vector through multi-layer convolution operations; the fundamental frequency predictor / energy predictor accepts the speech frame length encoding vector and predicts the speech fundamental frequency value / speech energy value corresponding to each frame. The purpose of using three predictors in the alignment module is to extract and predict features that can determine the details of speech and the overall perception. For example, the phoneme duration (duration) affects the length of pronunciation and the overall rhythm; the pitch (fundamental frequency value) is a feature that affects emotion and rhythm; and the energy value determines the volume of the audio.
[0071] Since the emotional style vector contains the emotional details in the speech, it needs to be fused with the text encoding vector before the alignment module performs time alignment and fundamental frequency energy information prediction. Therefore, the alignment module first adds the emotional style vector and the text encoding vector.
[0072] The process of aligning text sequences with speech frames is as follows:
[0073] a. Add the sentiment style vector and the text encoding vector accordingly to obtain a combined text encoding vector. This vector combines the information of each text phoneme and the local sentiment style information.
[0074] b. The combined text encoding vector predicts the duration of each text phoneme through the duration predictor;
[0075] b. Use the LR module to align each phoneme to the speech frame length according to the predicted duration, and obtain the speech frame length encoding vector;
[0076] c. Then the encoding vector of the speech frame length passes through the pitch predictor (PitchPred i ctor) and the energy predictor (Energy Pred i ctor) in turn to obtain the pitch prediction value and the energy prediction value;
[0077] d. Add the fundamental frequency prediction value and energy prediction value to the speech frame length coding vector in turn, and the final result is the obtained text-speech alignment sequence.
[0078] (3) Speech frame decoding, see Figure 4 , Figure 4 The speech frame decoder workflow is shown.
[0079] The function of this part is to decode the speech frame-level coded data obtained by the "alignment and fusion module" into speech frame feature data (Mel-spectrogram features). The speech frame-level coded data in the previous step is not the final result of speech synthesis. The text-speech alignment coded data combines text and emotional style information. The decoder is needed to decode the coded data into Mel-spectrogram features to generate an audio file audible to the human ear.
[0080] The structure of the decoder is roughly the same as that of the text encoder, but a post-processing module is connected at the end to synthesize more accurate speech frame-level features. The post-processing module is a stack of 5 layers of 1D convolutional networks. This module calculates the decoded feature sequence through the convolutional network multiple times, and the resulting 80-dimensional encoding vector is the Mel-spectrogram feature. The purpose of the post-processing module is to model the speech frame decoding sequence obtained by the decoder more finely and reduce the information loss in the Mel-spectrogram features generated by the model.
[0081] Figure 4 The working process of the speech frame decoder is shown. The decoding process of the decoder is as follows:
[0082] a. Aligned coding sequence at the speech frame level containing emotional style and text fusion information, and constructing position vector coding through the position vector layer;
[0083] b. Add the position vector code and the corresponding position of the alignment code sequence, and perform front-to-back concatenation of the speaker ID value and the position vector code after adding the corresponding position of the alignment code sequence. The concatenated result is decoded by a multi-layer FFTB lock module. The obtained decoded feature sequence is passed through a post-processing module to obtain the final speech frame feature sequence, namely the Mel-spectrogram feature.
[0084] (4) Sentiment information extraction module, see Figure 5 , Figure 5 Extraction and production process for sentiment extraction module.
[0085] The function of the emotional information extraction module used in the present invention is to extract the emotional style vector from the speech data. The emotional style vector can uniquely represent an emotional style and can be completely decoupled from the speaker information, speaking style and other information in the speech data. Therefore, the speaker information and the emotional style vector can be freely combined to synthesize emotional speech that meets the needs.
[0086] The emotion vector is concatenated with the text encoding vector as the input data of the alignment module, and then decoded by the decoder to obtain the final Mel-spectrogram feature with emotion information. This module is divided into two processing flows: extraction process and production process.
[0087] The extraction process is as follows:
[0088] a. The emotional speech obtains its Mel-spectrogram feature, which is input into the reference encoder layer (ReferenceEncoder). The reference encoder layer encodes the Mel-spectrogram feature into a vector of fixed length, called the reference embedding vector;
[0089] b. Using the reference embedding vector, multiple vectors representing emotional expressions are calculated through the attention module (Attention Mode 1); the emotional expression vectors are labeled with the emotional category number so that during the extraction process, these emotional expression vectors can each represent an emotional category;
[0090] c. The emotional style vector is composed of a weighted combination of these multiple emotional expression vectors.
[0091] The extraction process is performed during the TTS system training process, and multiple emotion expression vectors are finally obtained; in the speech production process, the selected emotion category number and multiple corresponding emotion expression vectors are multiplied to obtain the emotion style vector. The production process is as follows:
[0092] a. Specify the required emotion category and convert the emotion category number into a fixed-length vector (emotional style vector) containing only 0 and 1 through the attention module;
[0093] b. The vector is passed through the attention module (Attent i on Mode 1) and weighted combined with multiple pre-extracted emotion vectors to finally obtain the required emotion style vector.
[0094] Model overall workflow: see Figure 5 , which is the overall execution flow chart of the emotion synthesis TTS system.
[0095] The overall working mode of the TTS synthesis system includes two processes: training process and production process. During the training process, the various parameters in the entire network are automatically adjusted through the labeled and aligned "text phoneme sequence", "text corresponding speech frame data", and "speaker information" data sets; after the training is completed, the various parameters in the network will no longer change. In the production process, the network model with fixed parameters obtained during the training process is used to input the target text sequence and the target speaker to obtain the target synthesized Mel-spectrogram features.
[0096] The input of the TTS system includes "speaker identity ID", "phoneme-level text sequence" and "emotional audio features / emotional tag ID" (in the training process, this part is "emotional audio features + emotion tag ID", and in the production process, this part is "emotional tag ID"). The overall workflow of the system is:
[0097] a. The phoneme-level text sequence is input into the text encoder, and the text encoding vector is generated through the processing of the text encoder network;
[0098] b. The "emotional audio features / emotional tag ID" information is passed through the emotion extraction module to obtain the emotional style vector decoupled from the audio features;
[0099] c. The text encoding vector and the emotional style vector are added together as the input of the text-speech frame alignment module, which can expand the input sequence into a sequence of speech frame length;
[0100] d. The aligned speech frame length sequence is combined with the "speaker ID" information and decoded into Mel-spectrogram features through the speech frame decoder.
[0101] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions only describe the principles of the present invention. The present invention may be subject to various changes and improvements without departing from the spirit and scope of the present invention. These changes and improvements fall within the scope of the present invention. The scope of protection claimed by the present invention is defined by the attached claims and their equivalents.
Claims
1. The emotion transfer speech synthesis method is characterized by: The following steps are involved: S1. Input the text input sequence with phoneme level into the text encoder to obtain the text encoding vector; S2. Obtain the corresponding emotion category number according to the emotion category required by the speech, input it into the emotion extraction module, and obtain the emotion style vector; S3. The text encoding vector and the emotional style vector are passed through a text-speech frame alignment module to obtain a text-speech alignment sequence; S4. Input the speaker ID into the speech frame decoder, process the text-speech alignment sequence through the speech frame decoder, and decode to obtain the Mel-spectrograph features.
2. The emotion transfer speech synthesis method according to claim 1, characterized in that: Wherein, S1 comprises the following steps: The phoneme-level text input sequence is processed through the character embedding layer and encoded into a fixed-dimensional sequence vector; The phoneme-level text input sequence is passed through the position encoding layer to construct a position vector; The dimensional sequence vector and position vector are added and sent to the multi-layer FFTBlock module to be converted into a fixed-length text encoding vector.
3. The emotion transfer speech synthesis method according to claim 1, characterized in that: in, The extraction process of the emotion extraction module in S2 includes the following steps: Obtain the Mel-spectrogram feature of the emotional speech, and input the Mel-spectrogram feature into the reference encoder layer, and encode it into a vector of fixed length, which is recorded as the reference embedding vector; Input the reference embedding vector into the attention module to calculate multiple emotion expression vectors; the emotion expression vectors are labeled by emotion category numbers, and during the extraction process, the emotion expression vectors each represent an emotion category; The emotional style vector is formed by weighted combination of multiple emotional expression vectors.
4. The emotion transfer speech synthesis method according to claim 3, characterized in that: Wherein, S2 includes the following steps: The emotion category number is converted into the emotion expression vector through the attention module; The emotional expression vectors are weighted combined to form the emotional style vector.
5. The emotion transfer speech synthesis method according to claim 1, characterized in that: Among them, S3 includes the following steps: By adding the emotion style vector and the text encoding vector correspondingly, a combined text encoding vector is obtained; Predicting the duration of each combined text phoneme of the combined text encoding vector by a duration predictor; Through the LR module, each phoneme is aligned to the speech frame length according to the predicted duration, and the speech frame length encoding vector is obtained; The speech frame length coding vector is sequentially passed through a fundamental frequency predictor and an energy predictor to obtain a fundamental frequency prediction value and an energy prediction value; The fundamental frequency prediction value, energy prediction value and speech frame length coding vector are added together to obtain a text-speech alignment sequence.
6. The emotion transfer speech synthesis method according to claim 1, characterized in that: Wherein, S4 comprises the following steps: The text-speech alignment sequence containing the emotional style vector and the text encoding vector information is passed through the position vector layer to construct the position vector encoding; The position vector code and the corresponding position of the alignment code sequence are added, and the speaker ID value is front-to-back concatenated with the position vector code after adding the corresponding position of the alignment code sequence. The concatenated result is decoded by a multi-layer FFTBlock module. The obtained decoding feature sequence is passed through a post-processing module to obtain the Mel-spectrogram feature.
7. Emotional transfer speech synthesis system, characterized in that: include: Text encoder, sentiment extraction module, text-speech frame alignment module, speech frame decoder.
8. The emotion transfer speech synthesis system according to claim 7, characterized in that: The text encoder comprises: a character embedding layer, a position encoding layer and a FFTBlock module; The emotion extraction module includes: a reference encoder layer and an attention module; The text-to-speech frame alignment module includes: a duration predictor, an LR module, a fundamental frequency predictor, and an energy predictor; The speech frame decoder comprises: a position coding layer, a FFTBlock module, and a post-processing module.
