Multi-speaker chinese speech synthesis method based on dense connection delay neural network

Abstract

The application discloses a multi-speaker Chinese speech synthesis method based on a dense connection time delay neural network, wherein a speaker encoder module in a multi-speaker Chinese speech synthesis network based on the dense connection time delay neural network extracts speaker embedding from a reference speech spectrum; the speaker encoder module is simple in structure and small in parameter quantity; the extracted speaker embedding fuses multi-level information; therefore, the speaker embedding can be optimized together with other modules in the multi-speaker Chinese speech synthesis network, the training process is simplified, and the speaker embedding more suitable for a speech synthesis task can be extracted; secondly, the output of a text encoder module of the multi-speaker Chinese speech synthesis network is taken as a key and a value, the output of the speaker encoder module is taken as a query, and the key, the value and the query are input into an encoder's scaling dot product attention mechanism to generate a conditional text representation as an input of a decoder, so that the speaker embedding can effectively control the style in the synthesized speech and improve the naturalness and similarity of the synthesized speech.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of speech synthesis, and particularly relates to a multi-speaker speech synthesis method based on a dense connection time delay neural network. BACKGROUND

[0002] Speech synthesis is a technology that can convert any input text into corresponding speech output, and is one of indispensable technologies in human-computer voice interaction, and has been widely applied in actual life. With the rapid development of science and technology, in order to be suitable for more applications, the demand of users for speech synthesis is increasing. In view of the demand for the diversity of synthesized speech styles, the most intuitive solution is to let each speaker record several hours of high-quality corpus, and then train a speech synthesis model for each speaker, and select the corresponding model according to the user-specified speaker to synthesize in the synthesis stage. However, the shortcomings of this method are also very prominent. Several models need to be trained for several speakers, which increases the memory overhead, and the speaking style of the synthesized speech can only be limited to the fixed known speaker, and cannot imitate the style of the unknown speaker in the training.

[0003] The main purpose of multi-speaker speech synthesis is to synthesize any speech similar to different speaker styles using only one speech synthesis system, without training a separate speech synthesis model for each speaker. The speaker style here specifically refers to the characteristics of the speaker's voice, accent, emotion, etc. in the speech. The multi-speaker speech synthesis technology can effectively utilize the similarity in the pronunciation of the same phonemes by different speakers, and does not need to contain a large-scale parallel corpus of each target speaker in the training set in the training stage, but only needs a large-scale corpus composed of small-scale corpora of multiple different speakers, to imitate the style characteristics of different speakers. Compared with multiple single-speaker speech synthesis models, the multi-speaker speech synthesis model can reduce the storage overhead, and effectively improve the naturalness and diversity of the synthesized speech under the condition of the same data size.

[0004] At present, there are some mainstream multi-speaker Chinese speech synthesis methods that can synthesize speech similar to the style of the target speaker. Most of the existing models need to train an additional voiceprint verification network, then extract the hidden variable from the reference speech as the speaker feature representation, that is, the speaker embedding. And the speaker embedding is spliced with the text content representation obtained by the speech synthesis model to form the conditional text representation. Finally, the conditional text representation is input into the subsequent module of the speech synthesis model for synthesis. In this method, the voiceprint verification network needs to be trained additionally, which makes the training process of the whole multi-speaker speech synthesis method complex and cumbersome. Moreover, the speaker embedding extracted by the network is based on the speaker classification task and may not be suitable for the speech synthesis task. In addition, intuitively, in the multi-speaker speech synthesis task, the text content should play a major role in the synthesis result, controlling the specific semantics expressed in the synthesized speech, and the speaker feature plays an auxiliary role, controlling the timbre, accent and emotion of the synthesized speech. However, in most multi-speaker Chinese speech synthesis methods, the conditional text representation is generated by splicing and fusion, which makes the text content representation and the speaker embedding have similar effects on the speech synthesis result, thus limiting the naturalness and similarity of the synthesized speech. SUMMARY

[0005] The summary portion of the present application is used to introduce the concepts in a brief form, which will be described in detail in the specific embodiments section. The summary portion of the present application is not intended to identify key or essential features of the claimed technical solutions, nor is it intended to limit the scope of the claimed technical solutions.

[0006] In view of the problems and deficiencies in the prior art, the present application aims to provide a multi-speaker Chinese speech synthesis method based on a dense connection time delay neural network. The speaker encoder module in the multi-speaker Chinese speech synthesis network based on the dense connection time delay neural network extracts the speaker embedding from the reference speech spectrum. The speaker encoder module has a simple structure and small parameter quantity, and the extracted speaker embedding fuses multi-level information, so it can be optimized with other modules in the multi-speaker Chinese speech synthesis network, and at the same time simplify the training process and extract speaker embedding more suitable for the speech synthesis task. Secondly, the output of the text encoder module in the multi-speaker Chinese speech synthesis network is input into the encoder's scaling dot product attention mechanism as the key and value, and the output of the speaker encoder module is input as the query to generate the conditional text representation, so that the speaker embedding can effectively control the style of the synthesized speech and effectively improve the naturalness and similarity of the synthesized speech to solve the problems in the background art.

[0007] To achieve the above-mentioned purpose, the present application provides the following technical solutions:

[0008] The application discloses a multi-speaker Chinese speech synthesis method based on a dense connection time delay neural network, and the multi-speaker Chinese speech synthesis comprises the following steps.

[0009] Step 1, preprocessing the inputted Chinese text to be synthesized, dividing all sentences of the Chinese text to be synthesized into multiple sub-sentences according to punctuation marks, and then converting the sub-sentences into sub-sentence phonetic sequence;

[0010] Step 2, preprocessing the target speaker voice to be synthesized, converting the target speaker voice to be synthesized into target voice mel spectrum, and cutting out a continuous spectrum segment from the target voice mel spectrum as a synthesis reference voice spectrum;

[0011] Step 3, training a multi-speaker Chinese speech synthesis system based on a dense connection time delay neural network, taking the sub-sentence phonetic sequence and the synthesis reference voice spectrum as input, and obtaining sub-sentence waveform voice of the sub-sentence synthesis output;

[0012] Step 4, splicing all the sub-sentence waveform voices according to the order of the original sub-sentences to obtain the final generated synthesis voice output.

[0013] Further, the multi-speaker Chinese speech synthesis system based on the dense connection time delay neural network in step 3 comprises a multi-speaker Chinese speech synthesis network and a vocoder, and the training steps of the multi-speaker Chinese speech synthesis system comprise the following steps.

[0014] Step 3.1, preprocessing the training data in the multi-speaker Chinese speech synthesis dataset, converting the training Chinese text into training phonetic sequence, converting the training voice waveform into training mel spectrum, and grouping all training mel spectrums of the same speaker into a training speaker spectrum group;

[0015] Step 3.2, randomly selecting a mel spectrum from the training speaker spectrum group corresponding to the speaker of the training phonetic sequence to cut out a continuous spectrum segment as a training reference voice spectrum;

[0016] Step 3.3, jointly grouping the training phonetic sequence, the training mel spectrum and the training reference voice spectrum into a training sample;

[0017] Step 3.4, defining a multi-speaker Chinese speech synthesis network based on a dense connection time delay neural network, taking the training sample as input to output predicted mel spectrum;

[0018] Step 3.5, training a vocoder using all the training mel spectrums in the multi-speaker Chinese speech synthesis dataset and the corresponding training voice waveform, outputting predicted waveform voice corresponding to the inputted predicted mel spectrum.

[0019] Step 3.6, combining the multi-speaker Chinese speech synthesis network trained and the vocoder to obtain the multi-speaker Chinese speech synthesis system trained.

[0020] Further, the specific operation of intercepting a continuous spectrum segment in steps 2 and 3.2 is as follows:

[0021] If the total length of the target speech mel spectrum or the training mel spectrum is less than the required fixed frame number, it is copied to make the total length of the target speech mel spectrum or the training mel spectrum greater than or equal to the fixed frame number, and then a continuous spectrum segment with a length of the fixed frame number is randomly intercepted;

[0022] If the total length of the target speech mel spectrum or the training mel spectrum is greater than or equal to the fixed frame number, a continuous spectrum segment with a length of the fixed frame number is directly randomly intercepted.

[0023] Further, when the Chinese text to be synthesized and the target speaker voice are synthesized to obtain corresponding voice output, the target speaker can be a known speaker appearing in the multi-speaker Chinese speech synthesis dataset, or an unknown speaker not appearing in the multi-speaker Chinese speech synthesis dataset.

[0024] Further, the multi-speaker Chinese speech synthesis network structure based on the dense connection time delay neural network is as follows:

[0025] e text =Encoder text (x),

[0026] e spk =Encoder spk (y ref ),

[0027] e cond =Attention cond (e text ,e spk ),

[0028] e attn =Attention dec (e cond ),

[0029] e dec =Decoder rnn (e attn ),

[0030]

[0031] Encodertext represents a text encoder module in the encoder, Encoder spk represents a speaker encoder module in the encoder, Attention cond represents a scaled dot-product attention mechanism in the encoder, Attention dec represents a position-sensitive attention mechanism in the decoder, Decoder rnn represents a recurrent neural network in the decoder, Postnet represents a post-processing network in the decoder; x represents an input sequence of sub-sentence pinyins, y ref represents a truncated reference speech spectrum, e text represents a text encoder module in the encoder, Encoder text represents an extracted text content representation, e spk represents a speaker encoder module in the encoder, Encoder spk represents an extracted speaker embedding, e cond represents a scaled dot-product attention mechanism in the encoder, Attention cond represents a conditional text representation obtained by fusing a text content representation and a speaker embedding, e attn represents a position-sensitive attention mechanism in the decoder, Attention dec represents an obtained output, e dec represents a recurrent neural network in the decoder, Decoder rnn represents an output intermediate feature, represents a spectrum output by a post-processing network in the decoder, Postnet.

[0032] Further, a conditional text representation obtained by fusing a text content representation and a speaker embedding using the scaled dot-product attention mechanism is calculated according to the following formula:

[0033] K = W k e text +b k ,

[0034] V = W v e text +b v ,

[0035] Q = W q e spk +b q ,

[0036]

[0037] wherein K, V and Q are respectively a key, a value and a query in the scaled dot-product attention mechanism, W k , W v , Wq , b k , b v and b q are the trainable parameters of the three linear layers in the key, value and query in the scaled dot-product attention mechanism, d represents the dimension of the text content representation e text .

[0038] Further, the attention weight calculation formula of the position-sensitive attention mechanism is:

[0039] e ij = v T tanh(W s s i + W h h j + W f f ij + b),

[0040]

[0041] where s i is the i-th step decoder hidden state, h j is the j-th step encoder hidden state, f ij is the total attention weight α i-1 of the previous step, v, W s , W h , W f and b are trainable parameters, and L is the total length of the encoding.

[0042] Further, the main part structure of the dense connection time delay neural network model in the speaker encoder module is:

[0043] The first layer is a time delay neural network layer, the second to seventh layers are all dense time delay neural network layers, the eighth layer is a forward neural network layer, the ninth to twentieth layers are all dense time delay neural network layers, the twenty-first layer is a forward neural network layer, the twenty-second layer is a pooling layer for calculating the mean and variance of all frames, and the twenty-third layer is a forward neural network layer.

[0044] Further, the dense time delay neural network layer of the speaker encoder module includes a bottleneck layer based on the forward neural network and a time delay neural network layer, and they are connected using a residual connection, that is, the input and the output of the time delay neural network layer are added together as the output of the entire dense time delay neural network layer.

[0045] Further, the loss function used by the trained multi-speaker Chinese speech synthesis network is:

[0046] L total = L before + Lafter +L st +λ1·L spkenc +λ2·R;

[0047] wherein, L before and L after are the mean square error between the input and output of the post-processing network in the decoder and the target spectrum, L st is the cross-entropy loss of the stop symbol, L spkenc is the cross-entropy loss of the speaker classification in the speaker encoder module, R is the parameter regularization term, and λ1 and λ2 are the weights of the speaker classification loss and the regularization term, respectively.

[0048] Compared with the prior art, the beneficial effects of the present application are: the present application provides a multi-speaker Chinese speech synthesis method based on a densely connected time delay neural network, and a multi-speaker Chinese speech synthesis system based on a densely connected time delay neural network finally generates a synthesized speech output. The speaker encoder module of the multi-speaker Chinese speech synthesis network in the multi-speaker Chinese speech synthesis system extracts local features and long-time correlations in the speech in turn, and aggregates multi-stage information of different layers after the two times of extracting information, so that a better speaker embedding can be extracted. Since the speaker encoder module has a small parameter quantity and a simple structure, it can be jointly optimized with other parts in the multi-speaker Chinese speech synthesis network model, simplifying the training process, and can project the reference speech spectrum into a speaker embedding space suitable for the speech synthesis task, so that the extracted speaker embedding can better control the style of the synthesized speech. In addition, the present application also uses the scaling dot product attention mechanism in the encoder to fuse speaker information and text information, and the residual error between the generated conditional text representation and the original text content representation is calculated by the speaker embedding and the text content representation, so that the speaker embedding controls the style of the synthesized speech in a way that is intuitive. Therefore, the method proposed in the present application can imitate the speaker style in the target speaker's voice, synthesize the voice of any text, and can be applied to known speakers and unknown speakers, and the synthesized speech has higher naturalness and similarity compared with the prior art. BRIEF DESCRIPTION OF DRAWINGS

[0049] The accompanying drawings, which form a part of this application, are included to provide a further understanding of the application and are incorporated in and constitute a part of this application. The illustrations, together with the description, serve to explain the application, but do not limit the application. In the drawings:

[0050] Figure 1 : a multi-speaker Chinese speech synthesis step flowchart of the multi-speaker Chinese speech synthesis method based on a densely connected time delay neural network implemented by the present application;

[0051] Figure 2 Figure 1 is a flow chart of a multi-speaker Chinese speech synthesis system training step for the multi-speaker Chinese speech synthesis method based on a densely connected delay neural network according to an embodiment of the present application;

[0052] Figure 3 Figure 2 is a structure flow chart of a multi-speaker Chinese speech synthesis network for the multi-speaker Chinese speech synthesis method based on a densely connected delay neural network according to an embodiment of the present application;

[0053] Figure 4 Figure 3 is a structure block diagram of a speaker encoder module for the multi-speaker Chinese speech synthesis method based on a densely connected delay neural network according to an embodiment of the present application. DETAILED DESCRIPTION

[0054] Embodiments of the present disclosure will be described in more detail with reference to the drawings. Although certain embodiments of the present disclosure are shown in the drawings, it is understood that the present disclosure can be implemented in various forms and should not be interpreted as being limited to the embodiments set forth herein. Rather, these embodiments are provided so that the present disclosure can be more thoroughly and completely understood. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0055] It should also be noted that, for the convenience of description, only the parts related to the present application are shown in the drawings. The embodiments in the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0056] The present application discloses a multi-speaker Chinese speech synthesis method based on a densely connected delay neural network, which will be described in detail below with reference to the drawings and in combination with embodiments.

[0057] Referring to Figure 1 As shown in FIG. 1, the multi-speaker Chinese speech synthesis includes the following steps:

[0058] Step 1, pre-process the input Chinese text to be synthesized, divide all sentences of the Chinese text to be synthesized into multiple sub-sentences according to punctuation marks, and then convert the sub-sentences into sub-sentence phonetic sequence;

[0059] Step 2, pre-process the target speaker voice to be synthesized, convert the target speaker voice into target voice mel spectrum, and cut out a continuous spectrum segment therefrom as a synthesis reference voice spectrum;

[0060] Step 3, train a multi-speaker Chinese speech synthesis system based on a densely connected delay neural network, take the sub-sentence phonetic sequence and the synthesis reference voice spectrum as input, and obtain the sub-sentence waveform voice of the sub-sentence synthesis output;

[0061] Step 4, all clause waveform speech is spliced according to the order of the original clause to obtain the final generated synthesized speech output.

[0062] Specifically, first, for a given Chinese text to be synthesized and a target speaker voice, preprocessing is performed, the Chinese text to be synthesized is divided into multiple clauses according to commas and periods, and each clause is converted into a clause pinyin sequence. At the same time, the target speaker voice is converted into an 80-dimensional target voice mel spectrum, and a fixed frame number of spectrum segments is obtained by copying and intercepting as a synthesized reference voice spectrum. Then, each clause pinyin sequence and the synthesized reference voice spectrum are input into the multi-speaker Chinese speech synthesis system based on the dense connection time delay neural network to obtain the clause waveform speech output of each clause. Finally, all clause waveform speeches are spliced according to the original order of the clauses to obtain the final generated synthesized speech output.

[0063] Further, the multi-speaker Chinese speech synthesis system includes a multi-speaker Chinese speech synthesis network and a vocoder, that is, the clause pinyin sequence and the reference voice spectrum are first input into the multi-speaker Chinese speech synthesis network to obtain the clause mel spectrum output of each clause. Then, all the clause mel spectrums are input into the trained vocoder to obtain the clause waveform speech output corresponding to each clause. The vocoder used in the present application is a HiFi-GAN vocoder, which is based on the method of generative adversarial network, uses a generator and a discriminator for adversarial training, and can quickly synthesize high-quality waveform speech after training.

[0064] Referring to Figure 2 As shown in the figure, the training steps of the multi-speaker Chinese speech synthesis system include:

[0065] Step 3.1, preprocessing the training data in the multi-speaker Chinese speech synthesis dataset, converting the training Chinese text into a training pinyin sequence, converting the training voice waveform into a training mel spectrum, and all training mel spectrums belonging to the same speaker form a training speaker spectrum group;

[0066] Step 3.2, according to the training pinyin sequence, randomly selecting a mel spectrum from the training speaker spectrum group corresponding to the speaker to which it belongs to obtain a continuous spectrum segment by intercepting, as a training reference voice spectrum;

[0067] Step 3.3, the training pinyin sequence, training mel spectrum and training reference voice spectrum together form a training sample;

[0068] Step 3.4, define a multi-speaker Chinese speech synthesis network based on a dense connection time delay neural network, and train the training sample as input to output a predicted mel spectrum;

[0069] Step 3.5, using all the training mel-spectra and their corresponding training speech waveforms in the multi-speaker Chinese speech synthesis dataset to train the vocoder, inputting the predicted mel-spectra to output the predicted waveform speech;

[0070] Step 3.6, combining the trained multi-speaker Chinese speech synthesis network and the vocoder to obtain the trained multi-speaker Chinese speech synthesis system.

[0071] Specifically, first, the speaker's speech waveform and the Chinese text to be trained in the multi-speaker Chinese speech synthesis dataset are preprocessed. The Chinese text to be trained is converted into a training pinyin sequence according to the standard of Chinese pinyin. The speaker's speech waveform is processed by a series of processes such as de-voicing, pre-emphasis, energy normalization, short-time Fourier transform, and mel filter filtering, and the 80-dimensional log training mel-spectrum is obtained. Then, for each pair of parallel data, a spectrum is randomly selected from all the training mel-spectra of the speaker to whom the data belongs, and a fixed frame number of the spectrum is obtained by copying and cutting to serve as a training reference speech spectrum, thereby constructing a training sample. Each training sample is composed of a training pinyin sequence, its corresponding training mel-spectrum, and a training reference speech spectrum. A multi-speaker Chinese speech synthesis network based on a densely connected time delay neural network is defined, and all the training samples are used to train the network. A general vocoder is trained using the training mel-spectrum and its corresponding training speech waveform in the multi-speaker Chinese speech synthesis dataset. The vocoder is a general vocoder for all speakers, and is trained using data of all speakers (i.e., all data in the multi-speaker Chinese speech synthesis dataset), so the training data does not need to be distinguished by speaker. Finally, the output of the multi-speaker Chinese speech synthesis network is used as the input of the vocoder, the multi-speaker Chinese speech synthesis network is used to generate predicted mel-spectra and corresponding predicted waveform speech as a training set to fine-tune the vocoder, and the trained multi-speaker Chinese speech synthesis network and the vocoder are combined to obtain the trained multi-speaker Chinese speech synthesis system.

[0072] Further, the multi-speaker Chinese speech synthesis dataset contains data of multiple speakers, wherein each speaker's data includes multiple text-speech pairs, and each text-speech pair includes a training pinyin sequence converted from training Chinese text and a training mel-spectrum converted from training speech waveform. Each training sample includes a training pinyin sequence, a training mel-spectrum corresponding to the pinyin sequence, and a mel-spectrum segment randomly selected and cut from the speaker's spectrum group to which the pinyin sequence belongs. It can be expressed in symbols as: x i , y i ), i = 1, 2,..., n, where x irepresents a training pinyin sequence, y i represents a corresponding training mel-spectrogram, n represents the number of data of the speaker; each training sample is (x i , y i , y j ), wherein y j ' represents a randomly selected one of all y i . j The intercepted mel-spectrum segment.

[0073] When synthesizing the corresponding speech output from the Chinese text to be synthesized and the target speaker, the target speaker can be a known speaker appearing in the multi-speaker Chinese speech synthesis dataset, or an unknown speaker not appearing in the multi-speaker Chinese speech synthesis dataset.

[0074] In steps 2 and 3.2, a continuous mel-spectrum segment is intercepted, and the specific operation is as follows: the number of intercepted fixed frames is set to 200 frames. If the total length of the target speech mel-spectrum or the training mel-spectrum is less than 200 frames, it is copied to make the total length of the target speech mel-spectrum or the training mel-spectrum greater than or equal to 200 frames, and then a continuous spectrum segment with a length of 200 frames is randomly intercepted; if the total length of the target speech mel-spectrum or the training mel-spectrum is greater than or equal to 200 frames, a continuous spectrum segment with a length of 200 frames is directly randomly intercepted.

[0075] Referring to FIG. 1, the multi-speaker Chinese speech synthesis network structure based on the dense connection time delay neural network is as follows: Figure 3

[0076] e text = Encoder text (x),

[0077] e spk = Encoder spk (y ref ),

[0078] e cond = Attention cond (e text ,e spk ),

[0079] e attn = Attention dec (e cond ),

[0080] e dec = Decoder rnn (e attn ),

[0081] ​

[0082] wherein Encoder text denotes a text encoder module in the encoder, Encoder spk denotes a speaker encoder module in the encoder, Attention cond denotes a scaled dot-product attention mechanism in the encoder, Attention dec denotes a location-sensitive attention mechanism in the decoder, Decoder rnn denotes a recurrent neural network in the decoder, Postnet denotes a post-processing network in the decoder; x denotes an input sequence of pinyin, y ref denotes the truncated reference speech spectrum, e text denotes a text encoder module in the encoder, Encoder text denotes the extracted text content representation, e spk denotes a speaker encoder module in the encoder, Encoder spk denotes the extracted speaker embedding, e cond denotes using a scaled dot-product attention mechanism in the encoder, Attention cond denotes the conditional text representation obtained after fusing the text content representation and the speaker embedding, e attn denotes using a location-sensitive attention mechanism in the decoder, Attention dec denotes the obtained output, e dec denotes a recurrent neural network in the decoder, Decoder rnn denotes the output intermediate feature, denotes the spectrum output by the post-processing network in the decoder, Postnet.

[0083] Specifically, the multi-speaker Chinese speech synthesis network can be divided into an encoder and a decoder.

[0084] The encoder is composed of three parts, namely a text encoder module, a speaker encoder module and an encoder attention module. The text encoder module generates a text content representation according to a pinyin sequence, the speaker encoder module extracts a speaker embedding from an input reference speech, and the encoder attention module generates a conditional text representation according to the text content representation and the speaker embedding for input to the subsequent decoder. The encoder attention module uses a scaled dot-product attention mechanism.

[0085] The text encoder module is composed of a pinyin embedding layer, three convolutional layers and a bidirectional long short-term memory network (Bi-LSTM). It is used to extract the text content representation from the input sentence pinyin sequence. The input sentence pinyin sequence is first obtained by a lookup table to obtain the vector of each pinyin element in the sequence, and each vector has a dimension of 512. Then, after passing through three convolutional layers, the output of the last convolutional layer is input into a bidirectional long short-term memory network to better obtain the context information in the text.

[0086] As shown in Figure 4 , the speaker encoder module based on the dense connection time delay neural network model extracts a 256-dimensional speaker embedding from each input reference speech spectrum. The main part of the dense connection time delay neural network model is as follows: the first layer is a time delay neural network layer, and uses the activation function ReLU for activation; the second to seventh layers are all dense time delay neural network layers with a frame offset of 1; the eighth layer is a forward neural network layer activated by ReLU; the ninth to twentieth layers are dense time delay neural network layers with a frame offset of 3; the twenty-first layer is a forward neural network layer activated by ReLU; the twenty-second layer is a pooling layer for calculating the mean and variance of all frames; and the twenty-third layer is a forward neural network layer with batch normalization.

[0087] In Figure 4 , as shown in the dashed part, a linear classification layer is added at the end during training to classify the speaker embedding, and the classification loss is added to the loss of the entire multi-speaker Chinese speech synthesis network. Compared with other voiceprint verification networks, the dense connection time delay neural network model has fewer parameters and a simpler structure. The speaker encoder module based on the dense connection time delay neural network model can be trained simultaneously with other parts of the multi-speaker Chinese speech synthesis network, without the need for separate training. In addition, the speaker encoder can effectively extract speaker features from the speech by aggregating multi-stage information from different layers, so that the synthesis model can better synthesize speech similar to the target speaker's speech.

[0088] The decoder is divided into a preprocessing network module, a decoder attention module, a recurrent neural network module and a post-processing network module. The preprocessing network module in the decoder includes two fully connected layers. The recurrent neural network module is composed of two one-way long short-term memory neural networks. The post-processing network module is composed of five convolutional layers. The decoder attention module uses a position-sensitive attention mechanism. In each step of operation of the decoder, the input is spliced with the context vector output by the decoder attention module at the previous step after being processed by the preprocessing network module. Then, the current step output is calculated by the recurrent neural network module. The context information of the current step is calculated by the decoder attention module according to the output. Finally, the current step output of the recurrent neural network module and the current step output of the decoder attention module are spliced and mapped by two linear layers to obtain the spectrum and stop symbol results of the current frame.

[0089] Further, the working process of the multi-speaker Chinese speech synthesis network is as follows: the text content representation is extracted by the sub-clause pinyin sequence input text encoder module, the speaker embedding is extracted by the reference speech spectrum input speaker encoder module, and the text content representation and the speaker embedding are input into the scaling dot product attention mechanism to output the conditional text representation. The conditional text representation is input into the position-sensitive attention mechanism to output the context information of the current step, which is spliced with the current step output of the recurrent neural network module and mapped by two linear layers to obtain the spectrum and stop symbol results of the current frame. After obtaining the spectrum of all frames, the obtained spectrum is fine-tuned by the post-processing network module using residual connection to handle the problem that the subsequent frames cannot be predicted in advance when calculated in the one-way recurrent neural network. The sub-clause mel spectrum of each sub-clause synthesis output is obtained.

[0090] In order to make the model converge better and faster, a teacher forcing method is used in the recurrent neural network, that is, a real spectrum is randomly selected as the input of the next frame with a certain probability. The dense delay neural network layer includes a bottleneck layer based on a forward neural network and a delay neural network layer, and they are connected by residual connection, that is, the input and the output of the delay neural network layer are added together as the output of the entire dense delay neural network layer.

[0091] The encoder attention module in the encoder uses the scaling dot product attention mechanism to fuse the conditional text representation obtained after the text content representation and the speaker embedding. The specific calculation formula is as follows:

[0092] K=W k e text +b k ,

[0093] V=W v e text +b v ,

[0094] Q = W q e spk +b q ,

[0095]

[0096] where K, V and Q are the key, value and query in the scaled dot-product attention mechanism, W k , W v , W q , b k , b v and b q are the trainable parameters in the scaled dot-product attention mechanism for calculating the three linear layers in the key, value and query, and d represents the dimension of the text content representation e text .

[0097] The position-sensitive attention mechanism used by the decoder attention module in the decoder has the following formula for calculating the attention weight:

[0098] e ij = v T tanh(W s s i +W h h j +W f f ij +b),

[0099]

[0100] where s i is the i-th step decoder hidden state, h j is the j-th step encoder hidden state, f ij is the attention weight of the previous step, and f i-1 is the position feature obtained after convolution, v, W s , W h , W f and b are all trainable parameters, and L is the total length of the encoding.

[0101] The loss function used to train the multi-speaker Chinese speech synthesis network is:

[0102] L total = L before + L after + L st + λ1·L spkenc + λ2·R;

[0103] where L before and L after are the mean square errors between the input and output of the post-processing network in the decoder and the target spectrum, respectively, and Lst L is the cross-entropy loss for stop symbol spkenc R is the cross-entropy loss for speaker classification in the speaker encoder module, and R is a parameter regularization term, and λ1 and λ2 are weights of the speaker classification loss and the regularization term, respectively.

[0104] The embodiment is applied to a large-scale Chinese multi-speaker speech synthesis dataset AISHELL-3

[0105] The method of the present application is tested on the large-scale Chinese multi-speaker speech synthesis dataset AISHELL-3, which includes parallel corpora recorded by 218 speakers. The effects of the method of the present application and the best existing method are compared. The best existing method is a multi-speaker speech synthesis system using feedback constraints proposed in a paper published in the top conference in the field of speech in 2020. In the experiment, the mean opinion score is used as an evaluation index, and the naturalness and similarity of the synthesized speech of the method of the present application and the best existing method are compared when synthesizing known speaker speech and unknown speaker speech. The experimental results show that the present application can effectively improve the naturalness and similarity of the synthesized speech while simplifying the training process. Especially when the target speaker is an unknown speaker, the improvement of the synthesized speech effect is more obvious.

[0106] The above description is only some of the preferred embodiments of the present disclosure and an explanation of the principles of the technology applied. Those skilled in the art should understand that the scope of the application involved in the embodiments of the present disclosure is not limited to the technical solutions formed by the specific combinations of the above technical features, and should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above inventive concept. For example, the above features are replaced with the technical features disclosed in the embodiments of the present disclosure (but not limited to) having similar functions to form technical solutions.

Claims

1. A multi-speaker Chinese speech synthesis method based on a dense connection time delay neural network, characterized in that, The multi-speaker Chinese speech synthesis comprises the following steps: Step 1, preprocessing the inputted Chinese text to be synthesized, dividing all sentences of the Chinese text to be synthesized into multiple sub-sentences according to punctuation marks, and then converting the sub-sentences into sub-sentence pinyin sequences; Step 2, preprocessing the target speaker voice to be synthesized, converting the target speaker voice into target voice mel spectrum, and cutting out a continuous spectrum segment therefrom as a synthesis reference voice spectrum; Step 3, training a multi-speaker Chinese speech synthesis system based on a dense connection time delay neural network, taking the sub-sentence pinyin sequence and the synthesis reference voice spectrum as inputs, and obtaining a sub-sentence waveform voice output of the sub-sentence; Step 4, splicing all the sub-sentence waveform voices according to the order of the original sub-sentences to obtain a final generated synthesis voice output; The multi-speaker Chinese speech synthesis network based on the dense connection time delay neural network is defined as follows: e text = Encoder text (x), e spk = Encoder spk (y ref ), e cond = Attention cond (e text , e spk ), e attn = Attention dec (e cond ), e dec = Decoder rnn (e attn ), wherein Encoder text denotes a text encoder module in the encoder, Encoder spk denotes a speaker encoder module in the encoder, Attention cond denotes a scaled dot-product attention mechanism in the encoder, Attention dec denotes a location-sensitive attention mechanism in the decoder, Decoder rnn denotes a recurrent neural network in the decoder, Postnet denotes a post-processing network in the decoder; x denotes an input sequence of sub-sentence pinyins, y ref denotes a truncated reference speech spectrum, e text denotes a text encoder module in the encoder, Encoder text denotes an extracted text content representation, e spk denotes a speaker encoder module in the encoder, Encoder spk denotes an extracted speaker embedding, e cond denotes a scaled dot-product attention mechanism in the encoder, Attention cond denotes a conditional text representation obtained after fusing the text content representation and the speaker embedding, e attn denotes a location-sensitive attention mechanism in the decoder, Attention dec denotes an obtained output, e dec denotes a recurrent neural network in the decoder, Decoder rnn denotes an output intermediate feature, denotes a spectrum output by the post-processing network in the decoder, Postnet 2. The method of claim 1, wherein the method is implemented by a dense connection time delay neural network based multi-speaker Chinese speech synthesis method. The multi-speaker Chinese speech synthesis system based on the dense connection time delay neural network in step 3 comprises a multi-speaker Chinese speech synthesis network and a vocoder, and the training steps of the multi-speaker Chinese speech synthesis system comprise: Step 3.1, preprocessing the training data in the multi-speaker Chinese speech synthesis dataset, converting the training Chinese text therein into a training pinyin sequence, converting the training voice waveform into a training mel spectrum, and grouping all training mel spectrums of the same speaker into a training speaker spectrum group; Step 3.2, randomly selecting a mel spectrum from the training speaker spectrum group corresponding to the speaker of the training pinyin sequence to cut out a continuous spectrum segment therefrom as a training reference voice spectrum; Step 3.3, jointly grouping the training pinyin sequence, the training mel spectrum, and the training reference voice spectrum into a training sample; Step 3.4, inputting the training sample into the multi-speaker Chinese speech synthesis network based on the dense connection time delay neural network to train and output a predicted mel spectrum; Step 3.5, training a vocoder using all the training mel spectrums in the multi-speaker Chinese speech synthesis dataset and their corresponding training voice waveforms, and outputting a predicted waveform voice corresponding to the inputted predicted mel spectrum; Step 3.6, combining the trained multi-speaker Chinese speech synthesis network and the vocoder to obtain the trained multi-speaker Chinese speech synthesis system.

3. The method of claim 2, wherein the method is implemented by a dense connection time delay neural network based multi-speaker Chinese speech synthesis method. The operation of cutting out a continuous spectrum segment in steps 2 and 3.2 is as follows: If the total length of the target voice mel spectrum or the training mel spectrum is less than the required fixed frame number, the target voice mel spectrum or the training mel spectrum is copied to make the total length of the target voice mel spectrum or the training mel spectrum greater than or equal to the fixed frame number, and then a continuous spectrum segment with a length of the fixed frame number is randomly cut out therefrom; If the total length of the target voice mel spectrum or the training mel spectrum is greater than or equal to the fixed frame number, a continuous spectrum segment with a length of the fixed frame number is directly randomly cut out therefrom.

4. The method of claim 2, wherein the method is implemented by a dense connection time delay neural network based multi-speaker Chinese speech synthesis method. When the to-be-synthesized Chinese text and the target speaker voice are synthesized to obtain a corresponding voice output, the target speaker can be a known speaker appearing in the multi-speaker Chinese voice synthesis dataset or an unknown speaker not appearing in the multi-speaker Chinese voice synthesis dataset.

5. The method of claim 1, wherein the method is implemented by a dense connection time delay neural network based multi-speaker Chinese speech synthesis method. The conditional text representation obtained after the text content representation and the speaker embedding are fused using the scaling dot product attention mechanism is calculated according to the following specific formula: K = W k e text +b k , V = W v e text +b v , Q = W q e spk +b q , where K, V and Q are the key, value and query in the scaled dot-product attention mechanism, respectively, W k , W v , W q , b k , b v and b q are the trainable parameters for the three linear layers in the scaled dot-product attention mechanism, and d denotes the dimension of the text content representation e text .

6. The method of claim 5, wherein the method is implemented by a dense connection time delay neural network based multi-speaker Chinese speech synthesis method. The attention weight calculation formula of the position-sensitive attention mechanism is as follows: e ij = v T tanh(W s s i + W h h j + W f f ij + b), where s i is the i-th step decoder hidden state, h j is the j-th step encoder hidden state, f ij is the all attention weights of the previous step, a i-1 is the position feature after convolution, v, W s , W h , W f and b are all to-be-trained parameters, and L is the total length of encoding.

7. The method of claim 6, wherein the method is implemented by a dense connection time delay neural network based multi-speaker Chinese speech synthesis method. The main part structure of the speaker encoder module based on the dense connection time delay neural network model is as follows: The first layer is a time delay neural network layer, the second to seventh layers are dense time delay neural network layers, the eighth layer is a forward neural network layer, the ninth to twentieth layers are dense time delay neural network layers, the twenty-first layer is a forward neural network layer, the twenty-second layer is a pooling layer for calculating the mean and variance of all frames, and the twenty-third layer is a forward neural network layer.

8. The method of claim 7, wherein the method is implemented by a dense connection time delay neural network based multi-speaker Chinese speech synthesis method. The dense time delay neural network layer of the speaker encoder module includes a bottleneck layer and a time delay neural network layer based on the forward neural network, and they are connected using a residual connection, that is, the input and the output of the time delay neural network layer are added together to serve as the output of the entire dense time delay neural network layer.

9. The method of claim 8, wherein the method is implemented by a dense connection time delay neural network based multi-speaker Chinese speech synthesis method. The loss function used by the trained multi-speaker Chinese voice synthesis network is as follows: L total = L before + L after + L st + λ1- L spkenc + λ2- R; where L before and L after are the mean square error between the input and output of the post-processing network in the decoder and the target spectrum, respectively, L st is the cross-entropy loss of the stop token, L spkenc is the cross-entropy loss of speaker classification in the speaker encoder module, R is a parameter regularization term, and λ1 and λ2 are the weights of the speaker classification loss and the regularization term, respectively.

Images