Rhythm-controllable Chinese and English mixed speech synthesis method and system

Abstract

The invention discloses a rhythm-controllable Chinese and English mixed speech synthesis method and system, and belongs to the field of speech synthesis. The method comprises the following steps of: 1) extracting phoneme pronunciation duration, energy, pitch and Mel spectrum from Chinese and English texts and audios with rhythm marks as a training set, and learning a text coding representation aligned with the Mel spectrum in length; and 2) generating predicted energy and pitch through an energy and pitch prediction model to realize energy and pitch control. And 3) combining the predicted energy, pitch and text coding representation, outputting a synthesized Mel spectrum through a decoder, and synthesizing speech through a vocoder. According to the method, the rhythm pause information in the synthetic speech is better controlled by using a jump neural network encoder, the rhythm pronunciation information of each frame in the synthetic speech is finely controlled by using the predicted duration, energy and pitch, and the speech with richer rhythm change is generated. The method is completely completed by one model, and language distinguishing does not need to be carried out on the texts firstly.

Description

technical field [0001] The invention belongs to the field of speech synthesis, and relates to a Chinese-English mixed speech synthesis, in particular to a rhythm-controllable Chinese-English mixed speech synthesis method and a system thereof. Background technique [0002] In recent years, with the development of deep learning, speech synthesis technology has also been greatly improved. Speech synthesis has moved from the traditional parametric method and concatenation method to an end-to-end method. They usually first generate the mel spectrum from the text features, and then use the vocoder image to synthesize the speech from the mel spectrum. According to the structure, these end-to-end methods can be divided into autoregressive models and non-autoregressive models. Autoregressive models usually use the Encoder-Attention-Decoder mechanism for autoregressive generation: to generate the current data point, all previous data points in the time series must be generated as mo...

AI Technical Summary

Problems solved by technology

However, due to its complex network structure and autoregressive structural form, the existing autoregressive speech synthesis model still has some shortcomings in actual production:

[0004] (1) The model is complex and has high requirements for computing resources, so it cannot be used on hardware with low computing resources

[0005] (2) Due to the defect of autoregressive structure, the naturalness of long sentence synthesis decreases

[0006] (3) In terms of voice control, most of them only use duration, energy and pitch to control, and the control of voice rhythm is not comprehensive;

[0007] (4) It is impossible to synthesize texts that are mixed in Chinese and English;

Images