Emotion personalized speech synthesis model

Abstract

The invention provides an emotion personalized speech synthesis model, and belongs to the technical field of speech synthesis. The model is divided into five parts: a text information extraction stage, a speaker characterization extraction stage, an emotion characterization extraction stage, a feature fusion stage, and a Mel spectrum and speech generation stage. In the text information extraction stage, firstly, input phonemes are encoded to obtain a phoneme sequence, and the phoneme sequence is converted into a hidden sequence by using an encoder. In the speaker representation extraction stage, speaker representation is extracted mainly through a speaker encoder, and information of a speaker is obtained. In the emotion representation extraction stage, an emotion encoder is mainly used for extracting emotion representation, and emotion information is obtained. In the feature fusion stage, speaker characterization and emotion characterization are fused by using a reference attention mechanism, so that the characterization carries speaker and emotion information, and then some variable information (pitch, energy and duration) is added. And generating a Mel spectrum: generating the Mel spectrum through a decoder. And a voice generation stage: converting the Mel spectrum into an oscillogram to obtain a final voice. Compared with a previous personalized speech synthesis technology, the method has the following advantages: (1) an emotion feature extraction and emotion classification module is added, so that the emotion expressive force of personalized speech is improved; (2) ProbSparse Self-attention is used to replace self-attention, and the dimensionality and the network parameter quantity are reduced, so that the model training speed and the model reasoning speed are improved. Therefore, the method has high innovativeness and practicability and can be applied to multiple fields.

Description

Technical Field

[0001] The invention belongs to the technical field of speech synthesis, and in particular to emotional personalized speech synthesis. The method uses a speech synthesis model and combines multiple speech signal processing technologies to convert text into speech with personalized emotions. Background Art

[0002] Emotional personalized speech synthesis can better simulate the expression and emotional state of human speech, making the synthesized speech more natural, realistic, and more in line with human auditory perception. In addition, emotional personalized speech synthesis can also be applied to a variety of fields, such as speech synthesis, speech recognition, intelligent voice assistants, etc., and has broad application prospects. For example, in the intelligent customer service system, emotional personalized speech synthesis can better simulate the attitude and emotional state of human customer service personnel, improve service quality and user satisfaction; in the field of virtual anchors, emotional personalized speech synthesis can give virtual anchors unique personality and emotions, enhancing their attractiveness and interactivity. Therefore, emotional personalized speech synthesis technology has important significance and application value.

[0003] Personalized speech synthesis is a personalized service based on speech synthesis technology, which aims to generate speech output with personalized characteristics according to the personalized needs of users. At present, there are mainly the following personalized speech synthesis methods.

[0004] Statistical model-based method: This method uses statistical models to analyze the speech characteristics of speech signals, such as fundamental frequency, formant, etc., and the language characteristics of text information, such as phonemes, rhythm, etc. Then, according to the personalized needs of users, the model parameters are adjusted to generate speech output with personalized characteristics.

[0005] Deep learning-based method: This method builds a deep neural network model to learn the mapping relationship between speech and text, as well as the user's personalized needs. Then, based on the user's input text and personalized features, it generates personalized and natural speech output.

[0006] In recent years, the demand for personalized voice in various industries has increased, and the application fields of personalized voice have become more and more extensive, which has greatly promoted the research of personalized speech synthesis. Current research can achieve the customization of voice, but the emotion of voice is insufficient. How to make up for the emotion of personalized voice is also one of the important technical topics in this field. Summary of the invention

[0007] The purpose of the present invention is to make up for the shortcoming of insufficient emotional expression of personalized speech synthesis and propose an emotional personalized speech synthesis model. The present invention not only considers the quality of synthesized speech, but also considers the emotional expression of synthesized speech. In this way, the expression mode and emotional state of human speech are better simulated, making the synthesized speech more natural, realistic, and more in line with human auditory perception.

[0008] The present invention proposes an emotion-personalized speech synthesis model, which includes: a text information extraction stage, a speaker representation extraction stage, an emotion representation extraction stage, a feature fusion stage, a mel-spectrogram generation stage, and a speech generation stage. The specific scheme is as follows:

[0009] 1) The step of text information extraction includes converting the input text into phonemes and using an encoder to convert the phoneme sequence into a latent sequence, thereby extracting the deep information of the text. This method can effectively capture the voice features in the text information and provide a key information basis for the subsequent emotional personalized speech synthesis.

[0010] 2) The speaker representation extraction step includes using a pre-trained network to extract frame-level representations from the reference mel-spectrogram. These frame-level representations are passed through a speaker encoder to extract speaker representations, thereby incorporating speaker information in the reference audio into the speech synthesis process. The speaker feature extraction step also includes using a speaker classifier to classify the speaker. The classifier can accurately classify the features of each speaker, thereby effectively extracting speaker representations with emotional information.

[0011] 3) The step of extracting emotional representation includes using a pre-trained network to extract frame-level representations from the reference mel-spectrogram, and extracting emotional representations through an emotional encoder. The step of extracting emotional features also uses an emotional classifier to classify emotions. The emotional classifier can accurately classify the emotions in speech into five types, including happiness, sadness, anger, surprise, and fear. This method can effectively capture the emotional features in speech and provide an important information basis for subsequent emotional personalized speech synthesis.

[0012] 4) The feature fusion step includes fusing the speaker representation and the emotion representation using the reference attention mechanism so that the representation contains speaker and emotion information. The speaker representation and the emotion representation are fused through dot product attention. Then some information such as pitch, energy, and duration are added through the variance adapter.

[0013] 5) The step of generating a mel-spectrogram includes using a decoder to convert the comprehensive acoustic features processed by the variance adapter into a corresponding mel-spectrogram.

[0014] 6) The step of generating speech includes using a trained vocoder to convert the mel spectrogram into a corresponding waveform. This step is the last step of speech synthesis, which converts the processed mel spectrogram into a sound signal to generate realistic and personalized emotional speech. The step of generating speech also includes using pre-trained adversarial generative network technology for training. This technology can effectively improve the realism and naturalness of the synthesized speech.

[0015] Compared with the prior art, the present invention has the following advantages:

[0016] 1) Added emotion feature extraction and emotion classification modules to improve the emotional expressiveness of personalized voice.

[0017] 2) ProbSparse Self-attention is used instead of self-attention to reduce the dimension and number of network parameters, thereby improving the model training and inference speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 This is the modeling process of the emotion personalized speech synthesis model proposed by the present invention. DETAILED DESCRIPTION

[0019] The present invention proposes an emotion-personalized speech synthesis model, which is further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0020] The process of emotional personalized speech synthesis is as follows Figure 1 As shown, the following steps are included:

[0021] 1) Text information extraction steps: including converting the input text into phonemes and using an encoder to convert the phoneme sequence into a hidden sequence to extract the deep information of the text. This method can effectively capture the voice features in the text information and provide a key information basis for the subsequent emotional personalized speech synthesis.

[0022] 2) The process of speaker feature extraction usually includes two steps: First, use a pre-trained neural network to extract the acoustic features of each frame from the Mel-spectrogram of the reference audio. These acoustic features can include information such as pitch, intensity, harmonics, etc., which are very important for distinguishing the voices of different speakers. Secondly, these frame-level acoustic features are processed by a speaker encoder to extract a unique representation of the speaker. A speaker classifier is used to classify the speaker. This speaker representation can be used to identify the speaker, or to generate speech similar to the speaker in applications such as speech synthesis.

[0023] 3) Emotional feature extraction steps: including using a pre-trained network to extract frame-level representations from the reference mel-spectrogram, and extracting emotional representations through an emotional encoder. Using an emotional classifier to classify emotions, there are five types of emotions, including happiness, sadness, anger, surprise, and fear. This method can effectively capture the emotional features in speech and provide an important information basis for subsequent emotional personalized speech synthesis.

[0024] 4) Feature fusion step: In order to generate speech with speaker emotions, the speaker representation and emotion representation need to be combined. This process is achieved by using the attention mechanism. Specifically, we add the speaker representation and emotion representation to get a reference attention value, and then use the content representation as a key to calculate the weighted sum using this reference attention value to obtain a comprehensive speaker emotion representation. This comprehensive representation can be used to guide the speech synthesis model to generate speech with a specific speaker and emotion. In order to better adapt to the acoustic characteristics of the target speaker, we input this comprehensive feature into the variance adapter. The variance adapter can add some variance information to each frame, such as pitch, duration, and energy, to better control the sound quality and tone of the synthesized speech. In this way, we can make the synthesized speech more natural and fluent while maintaining the characteristics of the speaker's voice.

[0025] 5) The step of generating a mel-spectrogram includes using a decoder to convert the comprehensive acoustic features processed by the variance adapter into a corresponding mel-spectrogram.

[0026] 6) Generate speech: Use the pre-trained HiFi-GAN decoder to convert the Mel spectrogram into a waveform. HiFi-GAN is a speech synthesis model based on a generative adversarial network that can generate high-fidelity, high-quality speech waveforms. When using HiFi-GAN for speech synthesis, we can input the pre-processed Mel spectrogram into the decoder, which will convert the Mel spectrogram into the corresponding speech waveform. In this way, we can generate natural speech that matches the target speaker and emotion.

[0027] The method of the present invention can be implemented by programming by ordinary technicians in the field. Ordinary technicians in the field can understand that the above-mentioned emotion personalized speech synthesis model can be completed by a program, and the program can be stored in a computer-readable storage medium.

[0028] The above is only a specific embodiment of the present invention, which obviously cannot be used to limit the scope of the present invention. Therefore, equivalent changes made according to the claims of the present invention are still within the scope of the present invention.

Claims

1. The present invention proposes an emotional personalized speech synthesis model, characterized in that: The emotion personalized speech synthesis model comprises the following steps: a text information extraction stage, a speaker representation extraction stage, an emotion representation extraction stage, a feature fusion stage, a Mel spectrogram generation stage and a speech generation stage.

2. The text information extraction stage is characterized by: The steps of text information extraction include converting the input text into phonemes, and using an encoder to convert the phoneme sequence into a hidden sequence, thereby extracting the deep information of the text; the structure of the encoder is composed of word embedding and feedforward blocks (self-attention layer and feedforward neural network). This structure effectively captures the speech features in the text information and provides a key information basis for the subsequent emotional personalized speech synthesis.

3. Emotional representation extraction stage, characterized by: The emotion feature extraction step includes extracting frame-level representation from the reference mel-spectrogram using a pre-trained network, and extracting emotion representation through an emotion encoder; the structure of the emotion encoder is composed of a feedforward block and an instance regularization. This method effectively captures the emotion features in speech, providing an important information basis for subsequent emotion-personalized speech synthesis. An emotion classifier is also used to classify emotions; the emotion classifier accurately classifies the emotions in speech into five types, including happiness, sadness, anger, surprise, and fear.

4. Speaker representation extraction stage, characterized by: The speaker representation extraction step includes extracting frame-level representations from the reference mel-spectrogram using a pre-trained network; these frame-level representations are passed through a speaker encoder to extract speaker representations, thereby incorporating speaker information in the reference audio into the speech synthesis process. A speaker classifier is also used to classify speakers; the speaker classifier structure is 1 gradient reversal layer and n hidden linear layers with ReLU activation and 1 output linear layer with Softmax activation. The classifier accurately classifies the features of each speaker, thereby effectively extracting speaker representations with emotional information.

5. Speaker representation extraction stage, characterized by: The speaker representation extraction step includes extracting frame-level representations from the reference mel-spectrogram using a pre-trained network; these frame-level representations are passed through a speaker encoder to extract speaker representations, thereby incorporating speaker information in the reference audio into the speech synthesis process. A speaker classifier is also used to classify speakers; the speaker classifier structure is 1 gradient reversal layer and n hidden linear layers with ReLU activation and 1 output linear layer with Softmax activation. The classifier accurately classifies the features of each speaker, thereby effectively extracting speaker representations with emotional information.

6. Generate a mel-spectrogram, characterized in that: The step of generating a mel-spectrogram includes using a decoder to convert the comprehensive acoustic features processed by the variance adapter into a corresponding mel-spectrogram. The decoder mainly consists of 6 feedforward blocks (self-attention layer and feedforward neural network).

7. The speech generation stage is characterized by: The step of generating speech includes using a trained vocoder to convert the mel-spectrogram into a corresponding waveform; this step is the last step of speech synthesis, converting the processed mel-spectrogram into a sound signal, thereby generating realistic and personalized emotional speech. Pre-trained adversarial generative network technology is also used for training; this technology effectively improves the realism and naturalness of the synthesized speech.

Images