Speech synthesis model training method, speech synthesis method, device and medium

By converting the training text into phoneme sequences and using the WaveGlow vocoder for parallel audio conversion, the problem of slow speech synthesis accuracy and speed of the Tacotron2 model during training is solved, and the accuracy and speed of the speech synthesis model is improved.

CN114038447BActive Publication Date: 2024-11-12SHENZHEN RAISOUND TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202111460685.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-12-02
Publication Date
2024-11-12
Estimated Expiration
2041-12-02

AI Technical Summary

Technical Problem

The existing Tacotron2 speech synthesis model requires a large amount of corpus during training, resulting in low accuracy of speech synthesis, and the sample-by-sample point generation mechanism of WaveNet vocoder leads to slow speech synthesis.

Method used

Use a preset speech synthesis model to convert the training text into phoneme sequences, use the encoder, decoder and residual network to process the phoneme sequences, generate the target Mel spectrum, and then use the WaveGlow vocoder to perform parallel audio conversion, adjust the model parameters to improve Voice synthesis accuracy and speed.

Benefits of technology

Reduced the need for training corpus, improved the accuracy of the speech synthesis model, and accelerated the speed of speech synthesis.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114038447B_ABST
    Figure CN114038447B_ABST
Patent Text Reader

Abstract

The present application relates to the field of artificial intelligence technology, and discloses a method for training a speech synthesis model, the method comprising: obtaining a training text, performing phoneme conversion on the training file using a preset speech synthesis model to obtain a phoneme sequence, the speech synthesis model comprising an encoder, a decoder, a residual network, and a WaveGlow vocoder; using the encoder, the decoder, and the residual network to process the phoneme sequence in sequence to obtain a target Mel spectrum; using the WaveGlow vocoder to perform parallel audio conversion on the target Mel spectrum to obtain a target audio; performing loss calculation on the target audio to obtain a training loss value, and adjusting the parameters of the speech synthesis model according to the loss value to obtain a target speech synthesis model. In addition, the present application also relates to a speech synthesis method, apparatus, device, and storage medium. The present application can improve the accuracy of the speech synthesis model and speed up the training speed of the speech synthesis model to synthesize speech.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a training method for a speech synthesis model, a speech synthesis method, a device, an electronic device and a storage medium. Background Art

[0002] Speech synthesis refers to the technology of converting any input text into corresponding speech, and is an important research branch in the field of natural speech processing.

[0003] The most common speech synthesis model at present is the Tacotron2 model. The Tacotron2 model uses character embedding as the input model during training, so a large amount of training corpus is required to ensure that the trained Tacotron 2 model will not have pronunciation errors. However, in actual applications, it is usually impossible to obtain a large amount of training corpus, which leads to low speech synthesis accuracy. Even if a large amount of training corpus is obtained, the speech synthesis speed will be reduced. In addition, the Tacotron2 model uses WaveNet as a vocoder. Since the WaveNet vocoder uses a speech waveform generation mechanism based on sample points, the synthesis speed is very slow and the accuracy is not high. Summary of the invention

[0004] In order to solve the above technical problems or at least partially solve the above technical problems, the present application provides a training method for a speech synthesis model, a speech synthesis method, a device, an electronic device and a storage medium.

[0005] In a first aspect, the present application provides a method for training a speech synthesis model, the method comprising:

[0006] Acquire a training text, and perform phoneme conversion on the training file using a preset speech synthesis model to obtain a phoneme sequence, wherein the speech synthesis model includes an encoder, a decoder, a residual network, and a WaveGlow vocoder;

[0007] Using the encoder, the decoder and the residual network to process the phoneme sequence in sequence to obtain a target Mel spectrum;

[0008] Using the WaveGlow vocoder to perform parallel audio conversion on the target Mel spectrum to obtain target audio;

[0009] A loss calculation is performed on the target audio to obtain a training loss value, and parameters of the speech synthesis model are adjusted according to the loss value to obtain a target speech synthesis model.

[0010] Optionally, the using the encoder, the decoder, and the residual network to sequentially process the phoneme sequence to obtain a target Mel spectrum includes:

[0011] Using the encoder to extract context features from the phoneme sequence to obtain a hidden feature matrix;

[0012] According to the hidden feature matrix, using the decoder to predict the Mel spectrum of the training text to obtain a predicted Mel spectrum;

[0013] The predicted Mel spectrum is residually connected using the residual network to obtain a target Mel spectrum.

[0014] Optionally, the decoder includes an attention network and a post-processing network, and the predicting the Mel spectrum of the training text using the decoder according to the hidden feature matrix to obtain the predicted Mel spectrum includes:

[0015] Extracting a context vector in the hidden feature matrix using the attention network to obtain a context vector for a first current time step;

[0016] The context vector of the first current time step is concatenated with a preset Mel spectrum, and the concatenation result is input into a preset double-layer long short-term memory layer to obtain the context vector of the second current time step;

[0017] Performing a first linear projection on the context vector of the second current time step using the post-processing network to obtain a context scalar of the current time step;

[0018] Using the post-processing network, perform a second linear projection on the context vector of the second current time step and perform Mel spectrum prediction on the context vector after the second linear projection to obtain the Mel spectrum of the second current time step;

[0019] Calculate the probability of Mel spectrum prediction completion using a preset first activation function according to the context scalar of the current time step;

[0020] Determine whether the probability of completing the Mel spectrum prediction is less than a preset threshold;

[0021] When the probability of the completion of the Mel spectrum prediction is less than the threshold, the context vector of the second current time step is connected in series with the Mel spectrum of the second current time step, and the step of inputting into the preset double-layer long short-term memory layer is returned until the probability of the completion of the Mel spectrum prediction is less than the threshold, and the Mel spectrum prediction is ended to obtain the predicted Mel spectrum.

[0022] Optionally, the attention network includes an attention weight value, a linear layer, a second activation function, and a mapping function, and the step of extracting the context vector in the hidden feature matrix using the preset attention network to obtain the context vector of the first current time step includes:

[0023] Performing linear projection on the hidden feature matrix using the linear layer to obtain a key matrix;

[0024] Input the attention weight value into a preset convolutional layer to generate a position feature matrix;

[0025] Performing linear projection on the position feature matrix using the linear layer to obtain an additional feature matrix;

[0026] Adding the additional feature matrix to the key matrix, and processing the addition result by using the second activation function to obtain an attention probability vector;

[0027] Mapping the attention probability vector using the mapping function to obtain a weight vector of current attention;

[0028] The current attention weight vector is multiplied by the hidden feature matrix to obtain a context vector of the first current time step.

[0029] Optionally, the concatenating the context vector of the first current time step with a preset Mel spectrum, and inputting the concatenation result into a preset double-layer long short-term memory layer to obtain the context vector of the second current time step includes:

[0030] The context vector of the first current time step is concatenated with a preset Mel spectrum, and inputted into one of the long short-term memory layers to obtain a query vector;

[0031] Concatenating the query vector and the context vector of the first time step into another layer of the long short-term memory layer to obtain a decoder hidden state;

[0032] The decoder hidden state is concatenated with the context vector of the first time step to obtain the context vector of the second current time step.

[0033] Optionally, the using the residual network to perform residual connection on the predicted Mel spectrum to obtain a target Mel spectrum includes:

[0034] Performing residual calculation on the predicted Mel spectrum using a preset residual network to obtain a residual Mel spectrum;

[0035] The residual Mel spectrum is superimposed on the predicted Mel spectrum to obtain a target Mel spectrum.

[0036] In a second aspect, the present application provides a speech synthesis method, the method comprising:

[0037] Get the speech text to be synthesized;

[0038] The target speech synthesis model is used to perform speech synthesis on the speech text to be synthesized to obtain synthesized speech, wherein the target speech synthesis model is trained using the training method of the above-mentioned speech synthesis model.

[0039] In a fourth aspect, the present application provides a training device for a speech synthesis model, the device comprising:

[0040] A training text conversion module, used to obtain training text, and perform phoneme conversion on the training file using a preset speech synthesis model to obtain a phoneme sequence, wherein the speech synthesis model includes an encoder, a decoder, a residual network and a WaveGlow vocoder;

[0041] A target audio generation module, configured to sequentially process the phoneme sequence using the encoder, the decoder, and the residual network to obtain a target Mel spectrum, and perform parallel audio conversion on the target Mel spectrum using the WaveGlow vocoder to obtain a target audio;

[0042] The model loss value calculation module is used to perform loss calculation on the target audio to obtain a training loss value, and adjust the parameters of the speech synthesis model according to the loss value to obtain a target speech synthesis model.

[0043] In a fifth aspect, the present application provides a speech synthesis device, the device comprising:

[0044] A text acquisition module is used to acquire the speech text to be synthesized;

[0045] The model speech synthesis module is used to perform speech synthesis on the speech text to be synthesized using a target speech synthesis model to obtain synthesized speech, wherein the target speech synthesis model is obtained by training using the training device of the above-mentioned speech synthesis model.

[0046] In a seventh aspect, an electronic device is provided, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0047] Memory, used to store computer programs;

[0048] The processor is used to implement the steps of the speech synthesis model training method described in any embodiment of the first aspect, or implement the steps of the speech synthesis method described in the second aspect when executing the program stored in the memory.

[0049] In an eighth aspect, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the training method of the speech synthesis model as described in any embodiment of the first aspect are implemented, or the steps of the speech synthesis method as described in the second aspect are implemented.

[0050] The above technical solution provided by the embodiment of the present application has the following advantages compared with the prior art:

[0051] The training method, device, equipment and storage medium of the speech synthesis model proposed in the embodiment of the present invention convert the training text into a phoneme sequence to obtain the pronunciation attributes of each word in the training text, thereby avoiding pronunciation errors caused by the problem of polyphony in the training text, thereby reducing the demand for training corpus and improving the accuracy of the speech synthesis model. Furthermore, the phoneme sequence is processed in sequence by an encoder, a decoder and a residual network to obtain a target Mel spectrum. Finally, the target Mel spectrum is converted into a parallel audio by a WaveGlow vocoder to obtain the target audio. Since WaveGlow can realize the parallel conversion of the target Mel spectrum into a speech waveform, the slow speech synthesis speed caused by the sample point collection is avoided, and the training speed of the speech synthesis model for synthesizing speech is accelerated. Therefore, the training method, device, equipment and storage medium of the speech synthesis model proposed in the embodiment of the present invention improve the accuracy of the speech synthesis model and accelerate the training speed of the speech synthesis model for synthesizing speech. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0053] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0054] Figure 1 A flowchart of a method for training a speech synthesis model provided in an embodiment of the present application;

[0055] Figure 2 A flowchart of a speech synthesis method provided in an embodiment of the present application;

[0056] Figure 3 A schematic diagram of a module of a training device for a speech synthesis model provided in an embodiment of the present application;

[0057] Figure 4A schematic diagram of a module of a speech synthesis device provided in an embodiment of the present application;

[0058] Figure 5 A schematic diagram of the internal structure of an electronic device for implementing a method for training a speech synthesis model provided in an embodiment of the present application;

[0059] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0061] Figure 1 A flowchart of a method for training a speech synthesis model provided in an embodiment of the present application. In this embodiment, the method for training the speech synthesis model includes:

[0062] S11. Obtain training text, and use a preset speech synthesis model to perform phoneme conversion on the training file to obtain a phoneme sequence, wherein the speech synthesis model includes an encoder, a decoder, a residual network, and a WaveGlow vocoder.

[0063] In the embodiment of the present invention, the source and text type of the training text can be in various forms, wherein the text type includes Chinese and English, etc. The phoneme sequence can be the smallest speech unit divided according to the natural attributes of speech, and is analyzed according to the pronunciation action in the syllable, where one action constitutes one phoneme, such as the phonemes of Chinese characters can be pinyin and tones.

[0064] In an optional embodiment of the present invention, when an English text is selected as a training text, there is a phenomenon that the same letters in English speech are pronounced differently. These complex pronunciation rules require a large amount of training data to learn. Therefore, when using a neural network to learn these pronunciation rules, if the training data is insufficient, it is difficult to learn all the pronunciation rules. In particular, when some rules appear too few times in the training data, the neural network will not be able to fully learn these rules, which will inevitably lead to pronunciation errors in some synthesized speech. Similarly, Chinese characters also have the phenomenon that the same Chinese characters are pronounced differently, such as polyphones. As the most basic unit of pronunciation, phonemes themselves embody pronunciation properties. Therefore, converting the training text into a phoneme sequence input can avoid pronunciation errors.

[0065] Specifically, use a preset speech synthesis model to perform phoneme conversion on the training file to obtain a phoneme sequence, including:

[0066] Use a language analysis tool to analyze the language of the training text to determine the language of the training text;

[0067] Perform sentence segmentation processing on the training text with the determined language according to the language to obtain segmented sentence texts;

[0068] According to preset text format rules, convert non-characters in the segmented sentence text into characters;

[0069] Perform word segmentation processing on the segmented sentence text to obtain a word-segmented text;

[0070] According to a preset character-phoneme mapping dictionary, map the word-segmented text to obtain phonemes;

[0071] Perform vector conversion on the phonemes to obtain phoneme vectors;

[0072] Encode and sort the phoneme vectors in text order to obtain a phoneme sequence.

[0073] In the embodiment of the present invention, by analyzing the language of the training text, the pronunciation rules of the training text can be determined. Further, perform word segmentation processing on the training text to accurately perform phoneme conversion on the training text to obtain phonemes. Finally, by encoding and sorting the phonemes, a phoneme sequence is obtained. The accuracy of the phoneme sequence is ensured by arrangement, avoiding chaotic pronunciation of the training text and improving the accuracy of speech synthesis.

[0074] In the embodiment of the present invention, the preset text format rules may be that in the obtained training text, if there are Arabic numerals, convert the Arabic numerals into characters, and then standardize the synthesized text according to the set rules. For example, in "There are 123 flowers here", where "123" is an Arabic numeral and needs to be converted into the Chinese characters "one two three", which facilitates subsequent processes such as converting characters into phonemes.

[0075] In another alternative embodiment of the present invention, an open-source grapheme-to-phoneme conversion tool G2P can be used to perform phoneme conversion on the training file to obtain a phoneme sequence.

[0076] S12. Use the encoder, the decoder, and the residual network to process the phoneme sequence in turn to obtain a target Mel spectrogram.

[0077] In an embodiment of the present invention, the encoder includes a convolutional layer and a bidirectional long short-term memory network. The decoder can be an autoregressive recurrent neural network, including an attention network and a post-processing network. The residual network includes a convolutional layer and a series of functions.

[0078] In the embodiment of the present invention, the conversion of text data into audio data is achieved by converting the phoneme sequence into a target Mel-spectrogram.

[0079] In detail, the encoder, the decoder and the residual network are used to process the phoneme sequence in sequence to obtain a target Mel spectrum, including:

[0080] Using the encoder to extract context features from the phoneme sequence to obtain a hidden feature matrix;

[0081] According to the hidden feature matrix, using the decoder to predict the Mel spectrum of the training text to obtain a predicted Mel spectrum;

[0082] The predicted Mel spectrum is residually connected using the residual network to obtain a target Mel spectrum.

[0083] In the embodiment of the present invention, the hidden feature matrix includes information such as the context vector of the phoneme sequence.

[0084] In the embodiment of the present invention, since the meaning of each word in the training text is often closely related to the context, for example, in the sentence "I love China so much", the word "good" has two pronunciations. Analyzing the word "good" alone cannot determine the pronunciation of the word "good", which can easily cause the problem of pronunciation errors. Therefore, it is necessary to extract the context feature information of each word.

[0085] In detail, the encoder includes a convolutional layer and a bidirectional long short-term memory network, and the encoder is used to extract context features from the phoneme sequence to obtain a hidden feature matrix, including:

[0086] Performing convolution processing on the phoneme sequence using a preset number of convolution layers to obtain a feature matrix of the phoneme sequence;

[0087] Performing a modified linear unit activation process and a batch normalization process on the feature matrix to obtain an optimized feature matrix;

[0088] The optimized feature matrix is ​​calculated using a preset bidirectional long short-term memory network to obtain a hidden feature matrix.

[0089] In an embodiment of the present invention, the bidirectional long short-term memory network may be used to obtain and store the context vector of the phoneme sequence.

[0090] In an embodiment of the present invention, the encoder is used to extract features from the phoneme sequence to obtain a hidden feature matrix. Since the hidden feature matrix contains information such as the context vector of the phoneme sequence, the context features of the phoneme sequence can be obtained by obtaining the hidden feature matrix, thereby improving the influence of the context features of the phoneme sequence on the phoneme sequence and improving the accuracy of the pronunciation of the speech synthesis model.

[0091] Furthermore, the decoder includes an attention network and a post-processing network, and the method of predicting the Mel spectrum of the training text using the decoder according to the hidden feature matrix to obtain the predicted Mel spectrum includes:

[0092] Extracting a context vector in the hidden feature matrix using the attention network to obtain a context vector for a first current time step;

[0093] The context vector of the first current time step is concatenated with a preset Mel spectrum, and the concatenation result is input into a preset double-layer long short-term memory layer to obtain the context vector of the second current time step;

[0094] Performing a first linear projection on the context vector of the second current time step using the post-processing network to obtain a context scalar of the current time step;

[0095] Using the post-processing network, perform a second linear projection on the context vector of the second current time step and perform Mel spectrum prediction on the context vector after the second linear projection to obtain the Mel spectrum of the second current time step;

[0096] Calculate the probability of Mel spectrum prediction completion using a preset first activation function according to the context scalar of the current time step;

[0097] Determine whether the probability of completing the Mel spectrum prediction is less than a preset threshold;

[0098] When the probability of the completion of the Mel spectrum prediction is less than the threshold, the context vector of the second current time step is connected in series with the Mel spectrum of the second current time step, and the step of inputting into the preset double-layer long short-term memory layer is returned until the probability of the completion of the Mel spectrum prediction is less than the threshold, and the Mel spectrum prediction is ended to obtain the predicted Mel spectrum.

[0099] In an embodiment of the present invention, the attention network includes the position-sensitive attention mechanism and the double-layer long short-term memory layer, which is mainly used to determine which part of the encoder input needs to be paid attention to. The first activation function can be a sigmoid function.

[0100] The method of extracting the context vector in the hidden feature matrix by using the position-sensitive attention mechanism in the preset attention network to obtain the context vector of the first current time step includes:

[0101] Performing linear projection on the hidden feature matrix using the linear layer to obtain a key matrix;

[0102] Input the attention weight value into a preset convolutional layer to generate a position feature matrix;

[0103] Performing linear projection on the position feature matrix using the linear layer to obtain an additional feature matrix;

[0104] Adding the additional feature matrix to the key matrix, and processing the addition result by using the second activation function to obtain an attention probability vector;

[0105] Mapping the attention probability vector using the mapping function to obtain a weight vector of current attention;

[0106] The current attention weight vector is multiplied by the hidden feature matrix to obtain a context vector of the first current time step.

[0107] The attention weight value may be a value obtained by concatenating the attention weight of the previous time step with the accumulation of all previous attention weights. The second activation function may be a Tanh function. The mapping function may be a softmax function.

[0108] For example, the hidden feature matrix is ​​input into a linear layer of 128 units for linear projection to generate a key matrix, and then the attention weight of the previous time step is connected in series with the accumulation of all previous attention weights and sent to another convolution layer, and the position feature is calculated by 32 one-dimensional convolution kernels with a length of 31. The position feature is processed by a linear layer of 128 units and added to the key matrix as an additional feature, and the addition result is processed by the Tanh function together, and then processed by a linear layer to generate an attention probability vector. Finally, the attention probability vector is processed by the softmax function to obtain the current attention weight vector.

[0109] In addition, the context vector of the first current time step is concatenated with a preset Mel spectrum, and the concatenation result is input into a preset double-layer long short-term memory layer to obtain the context vector of the second current time step, including:

[0110] The context vector of the first current time step is concatenated with a preset Mel spectrum, and inputted into one of the long short-term memory layers to obtain a query vector;

[0111] Concatenate the query vector and the context vector of the first time step and input them into another layer of the long short-term memory layer to obtain a decoder hidden state;

[0112] The decoder hidden state is concatenated with the context vector of the first time step to obtain the context vector of the second current time step.

[0113] The first current time step can be understood as the previous time step in the current loop operation process, and the second current time step can be understood as the current time step in the current loop operation process.

[0114] For example, the context vector of the previous time step is obtained, the context vector of the previous time step is concatenated with a preset Mel spectrum, and the concatenation result is sent to the first long short-term memory layer to obtain a query vector, the query vector is concatenated with the context vector of the previous time step, and the concatenation result is input into the second long short-term memory layer to generate a decoder hidden state, and the decoder hidden state is concatenated with the context vector of the previous time step to obtain the context vector of the current time step.

[0115] In the embodiment of the present invention, the mel spectrum is predicted by fusing the context vector of the previous time step to obtain the predicted mel spectrum. Therefore, the predicted mel spectrum is prone to lose its own characteristics, resulting in errors in the understanding of the meaning of the word, thereby causing pronunciation errors. Therefore, it is necessary to perform residual linking on the predicted mel spectrum to deepen the influence of the word itself on the word, thereby improving the accuracy of understanding the meaning of the word and reducing pronunciation errors.

[0116] Therefore, the residual connection of the predicted Mel spectrum to obtain the target Mel spectrum includes:

[0117] Performing residual calculation on the predicted Mel spectrum using a preset residual network to obtain a residual Mel spectrum;

[0118] The residual Mel spectrum is superimposed on the predicted Mel spectrum to obtain a target Mel spectrum.

[0119] In an optional embodiment of the present invention, the predicted Mel spectrum obtained through N decoding steps is sent to the residual network to generate a residual and superimpose it with itself to generate the target Mel spectrum. The residual network is composed of 5 convolution layers, each convolution layer is composed of 512 convolution kernels with a shape of 5*1, and each convolution layer is followed by batch normalization. Except for the last convolution layer, the other four convolution layers are activated by Tanh activation function.

[0120] S13: Use the WaveGlow vocoder to perform parallel audio conversion on the target Mel spectrum to obtain target audio.

[0121] In the embodiment of the present invention, the traditional speech synthesis model uses WaveNet as a vocoder to convert the target Mel spectrum into the target audio. The main structure of the WaveNet vocoder is a series of dilated convolutions stacked together, which increases the receptive field of the entire network. Since the WaveNet vocoder is an autoregressive structure, the WaveNet vocoder needs the previously generated sound waveform as a condition when predicting the sound waveform of the current time step. Therefore, the speed of speech synthesis is very slow.

[0122] In the embodiment of the present invention, WaveGlow is selected as a vocoder to convert the target Mel spectrum into the target audio. The WaveGlow vocoder is a stream-based model that can generate high-quality audio samples in parallel, thereby improving the speed of speech synthesis.

[0123] In detail, the use of the WaveGlow vocoder in the speech synthesis model to perform parallel audio conversion on the target Mel spectrum to obtain the target audio includes:

[0124] Using the WaveGlow vocoder to perform parallel speech waveform conversion on the target Mel spectrum to obtain a target speech waveform;

[0125] Performing audio conversion on the target speech waveform to obtain target audio.

[0126] In an embodiment of the present invention, the audio conversion of the target speech waveform includes: sampling, quantizing and encoding the target speech waveform signal to obtain the target audio. Sampling the target speech waveform signal is the process of discretizing the continuous target speech waveform signal on the time axis on the time axis, and quantizing the sampled target speech waveform signal is to convert each sample of the continuous value in amplitude into a discrete value representation.

[0127] S14. Perform loss calculation on the target audio to obtain a training loss value, and adjust the parameters of the speech synthesis model according to the loss value to obtain a target speech synthesis model.

[0128] In an embodiment of the present invention, the loss function in the speech synthesis model is used to calculate the loss of the target audio to obtain a training loss value, and the parameters of the speech synthesis model are adjusted according to the loss value until the loss value is less than a preset threshold, thereby obtaining a target speech synthesis model.

[0129] The training method of the speech synthesis model proposed in the embodiment of the present invention converts the training text into a phoneme sequence, obtains the pronunciation attributes of each word in the training text, avoids pronunciation errors caused by the polyphony of a word in the training text, thereby reducing the demand for training corpus and improving the accuracy of the speech synthesis model. Furthermore, the phoneme sequence is processed in sequence by an encoder, a decoder and a residual network to obtain a target Mel spectrum. Finally, the target Mel spectrum is converted into a parallel audio by a WaveGlow vocoder to obtain a target audio. Since WaveGlow can realize the parallel conversion of the target Mel spectrum into a speech waveform, the slow speech synthesis speed caused by the sample point collection is avoided, and the training speed of the speech synthesis model for synthesizing speech is accelerated. Therefore, the training method of the speech synthesis model proposed in the embodiment of the present invention improves the accuracy of the speech synthesis model and accelerates the training speed of the speech synthesis model for synthesizing speech.

[0130] Figure 2 A flowchart of a speech synthesis method provided in an embodiment of the present application is provided. In this embodiment, the speech synthesis method includes:

[0131] S21. Obtain the speech text to be synthesized.

[0132] In this embodiment, the speech text to be synthesized may be obtained through any channel, for example, the speech text to be synthesized is input by a user, or obtained from a database.

[0133] The type of the speech text to be synthesized may be Chinese, English, or the like.

[0134] The speech text to be synthesized may be a paragraph text that needs to be abstracted, and the paragraph text includes a plurality of sentences.

[0135] S22. Perform speech synthesis on the speech text to be synthesized using a target speech synthesis model to obtain synthesized speech.

[0136] In this embodiment, the target speech synthesis model is trained using the speech synthesis model training method described in the aforementioned method embodiment.

[0137] In this embodiment, the speech text to be synthesized may be first cleaned, that is, unpronounceable characters in the speech text to be synthesized may be deleted, and then the speech text to be synthesized may be processed using a target speech synthesis model to obtain synthesized speech.

[0138] In this embodiment, since the target speech synthesis model is trained by the model training method described in the aforementioned method embodiment, the target speech synthesis model can be used to perform speech synthesis on the speech text to be synthesized to obtain synthesized speech.

[0139] like Figure 3 As shown, an embodiment of the present application provides a module schematic diagram of a training device 30 for a speech synthesis model, and the training device 30 for the speech synthesis model includes: a training text conversion module 31, a target audio generation module 32 and a model loss value calculation module 33.

[0140] The training text conversion module 31 is used to obtain the training text and perform phoneme conversion on the training file using a preset speech synthesis model to obtain a phoneme sequence. The speech synthesis model includes an encoder, a decoder, a residual network and a WaveGlow vocoder;

[0141] The target audio generation module 32 is used to process the phoneme sequence in sequence using the encoder, the decoder and the residual network to obtain a target Mel spectrum, and perform parallel audio conversion on the target Mel spectrum using the WaveGlow vocoder to obtain a target audio;

[0142] The model loss value calculation module 33 is used to perform loss calculation on the target audio to obtain a training loss value, and adjust the parameters of the speech synthesis model according to the loss value to obtain a target speech synthesis model.

[0143] In detail, each module in the training device 30 of the speech synthesis model described in the embodiment of the present application adopts the same method as above when used. Figure 1 The training method of the speech synthesis model described in the text is the same as the technical means and can produce the same technical effect, so I will not go into details here.

[0144] like Figure 4 As shown, an embodiment of the present application provides a module schematic diagram of a speech synthesis device 40, and the speech synthesis device 40 includes: a text acquisition module 41 and a model speech synthesis module 42.

[0145] The text acquisition module 41 is used to acquire the speech text to be synthesized;

[0146] The model speech synthesis module 42 is used to perform speech synthesis on the speech text to be synthesized using a target speech synthesis model to obtain synthesized speech, wherein the target speech synthesis model is obtained by training using the training device of the aforementioned speech synthesis model.

[0147] In detail, each module in the speech synthesis device 40 described in the embodiment of the present application adopts the same method as described above when used. Figure 2 The present invention has the same technical means as the speech synthesis method described in , and can produce the same technical effects, which will not be repeated here.

[0148] like Figure 5As shown, an embodiment of the present application provides an electronic device, including a processor 111, a communication interface 112, a memory 113 and a communication bus 114, wherein the processor 111, the communication interface 112, and the memory 113 communicate with each other through the communication bus 114.

[0149] The memory 113 is used to store computer programs.

[0150] In one embodiment of the present application, the processor 111 is used to implement the training method of the speech synthesis model provided by any one of the aforementioned method embodiments, or implement the speech synthesis method provided by any one of the aforementioned method embodiments when executing the program stored in the memory 113.

[0151] The training method of the speech synthesis model includes:

[0152] Acquire a training text, and perform phoneme conversion on the training file using a preset speech synthesis model to obtain a phoneme sequence, wherein the speech synthesis model includes an encoder, a decoder, a residual network, and a WaveGlow vocoder;

[0153] Using the encoder, the decoder and the residual network to process the phoneme sequence in sequence to obtain a target Mel spectrum;

[0154] Using the WaveGlow vocoder to perform parallel audio conversion on the target Mel spectrum to obtain target audio;

[0155] A loss calculation is performed on the target audio to obtain a training loss value, and parameters of the speech synthesis model are adjusted according to the loss value to obtain a target speech synthesis model.

[0156] The speech synthesis method comprises:

[0157] Get the speech text to be synthesized;

[0158] The speech text to be synthesized is subjected to speech synthesis using a target speech synthesis model to obtain synthesized speech, wherein the target speech synthesis model is obtained by training using the speech synthesis model training method described in any of the aforementioned method embodiments.

[0159] The communication bus 114 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus 114 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0160] The communication interface 112 is used for communication between the above electronic device and other devices.

[0161] The memory 113 may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory 113 may also be at least one storage device located away from the processor 111 .

[0162] The above-mentioned processor 111 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.

[0163] An embodiment of the present application also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the computer program implements the steps of a training method for a speech synthesis model as provided in any one of the aforementioned method embodiments, or implements the steps of a speech synthesis method as provided in any one of the aforementioned method embodiments.

[0164] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loading and executing computer program instructions on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server, a data center, etc. that contains one or more available media integrated. The available medium can be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk SolidStateDisk (SSD)), etc.

[0165] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0166] The foregoing is merely a specific embodiment of the present invention, which enables those skilled in the art to understand or implement the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A method for training a speech synthesis model, characterized in that: The method comprises: Acquire a training text, and perform phoneme conversion on the training text using a preset speech synthesis model to obtain a phoneme sequence, wherein the speech synthesis model includes an encoder, a decoder, a residual network, and a WaveGlow vocoder; Using the encoder, the decoder and the residual network to process the phoneme sequence in sequence to obtain a target Mel spectrum; Using the WaveGlow vocoder to perform parallel audio conversion on the target Mel spectrum to obtain target audio; Performing loss calculation on the target audio to obtain a training loss value, and adjusting parameters of the speech synthesis model according to the loss value to obtain a target speech synthesis model; The encoder, the decoder and the residual network are used to process the phoneme sequence in sequence to obtain a target Mel spectrum, including: using the encoder to extract context features of the phoneme sequence to obtain a hidden feature matrix; using the decoder to predict the Mel spectrum of the training text according to the hidden feature matrix to obtain a predicted Mel spectrum; using the residual network to perform residual connection on the predicted Mel spectrum to obtain a target Mel spectrum; The decoder includes an attention network and a post-processing network. The decoder is used to predict the Mel spectrum of the training text according to the hidden feature matrix to obtain the predicted Mel spectrum, including: using the attention network to extract the context vector in the hidden feature matrix to obtain the context vector of the first current time step; concatenating the context vector of the first current time step with a preset Mel spectrum, and inputting the concatenation result into a preset double-layer long short-term memory layer to obtain the context vector of the second current time step; using the post-processing network to perform a first linear projection on the context vector of the second current time step to obtain the context scalar of the current time step; using the post-processing network to perform a linear projection on the context vector of the second current time step to obtain the context scalar of the current time step; The context vector of the second current time step is subjected to a second linear projection and the context vector after the second linear projection is subjected to a Mel spectrum prediction to obtain the Mel spectrum of the second current time step; according to the context scalar of the current time step, the probability of the completion of the Mel spectrum prediction is calculated using a preset first activation function; it is determined whether the probability of the completion of the Mel spectrum prediction is less than a preset threshold; when the probability of the completion of the Mel spectrum prediction is less than the threshold, the context vector of the second current time step is connected in series with the Mel spectrum of the second current time step, and the step of inputting into the preset double-layer long short-term memory layer is returned until the probability of the completion of the Mel spectrum prediction is less than the threshold, the Mel spectrum prediction is terminated, and a predicted Mel spectrum is obtained; The attention network includes an attention weight value, a linear layer, a second activation function, and a mapping function. The method of extracting a context vector from the hidden feature matrix using the preset attention network to obtain a context vector of the first current time step includes: performing linear projection on the hidden feature matrix using the linear layer to obtain a key matrix; Input the attention weight value into a preset convolutional layer to generate a position feature matrix; perform linear projection on the position feature matrix using the linear layer to obtain an additional feature matrix; add the additional feature matrix to the key matrix, and process the addition result using the second activation function to obtain an attention probability vector; Mapping the attention probability vector using the mapping function to obtain a weight vector of current attention; The current attention weight vector is multiplied by the hidden feature matrix to obtain a context vector of the first current time step.

2. The method for training a speech synthesis model according to claim 1, wherein: The context vector of the first current time step is concatenated with a preset Mel spectrum, and the concatenation result is input into a preset double-layer long short-term memory layer to obtain the context vector of the second current time step, including: the context vector of the first current time step is concatenated with a preset Mel spectrum, and the result is input into one of the long short-term memory layers to obtain a query vector; the query vector and the context vector of the first current time step are concatenated and input into another layer of the long short-term memory layer to obtain a decoder hidden state; the decoder hidden state is concatenated with the context vector of the first current time step to obtain the context vector of the second current time step.

3. The method for training a speech synthesis model according to claim 1, wherein: The using the residual network to perform residual connection on the predicted Mel spectrum to obtain a target Mel spectrum includes: Performing residual calculation on the predicted Mel spectrum using a preset residual network to obtain a residual Mel spectrum; The residual Mel spectrum is superimposed on the predicted Mel spectrum to obtain a target Mel spectrum.

4. A speech synthesis method, characterized in that: The method comprises: Get the speech text to be synthesized; The speech text to be synthesized is subjected to speech synthesis using a target speech synthesis model to obtain synthesized speech, wherein the target speech synthesis model is obtained by training using the speech synthesis model training method described in any one of claims 1 to 3.

5. A training device for a speech synthesis model, characterized in that: include: A training text conversion module, used to obtain training text, and perform phoneme conversion on the training text using a preset speech synthesis model to obtain a phoneme sequence, wherein the speech synthesis model includes an encoder, a decoder, a residual network and a WaveGlow vocoder; A target audio generation module, configured to sequentially process the phoneme sequence using the encoder, the decoder, and the residual network to obtain a target Mel spectrum, and perform parallel audio conversion on the target Mel spectrum using the WaveGlow vocoder to obtain a target audio; A model loss value calculation module, used to perform loss calculation on the target audio to obtain a training loss value, and adjust the parameters of the speech synthesis model according to the loss value to obtain a target speech synthesis model; The encoder, the decoder and the residual network are used to process the phoneme sequence in sequence to obtain a target Mel spectrum, including: using the encoder to extract context features of the phoneme sequence to obtain a hidden feature matrix; using the decoder to predict the Mel spectrum of the training text according to the hidden feature matrix to obtain a predicted Mel spectrum; using the residual network to perform residual connection on the predicted Mel spectrum to obtain a target Mel spectrum; The decoder includes an attention network and a post-processing network. The decoder is used to predict the Mel spectrum of the training text according to the hidden feature matrix to obtain the predicted Mel spectrum, including: using the attention network to extract the context vector in the hidden feature matrix to obtain the context vector of the first current time step; concatenating the context vector of the first current time step with a preset Mel spectrum, and inputting the concatenation result into a preset double-layer long short-term memory layer to obtain the context vector of the second current time step; using the post-processing network to perform a first linear projection on the context vector of the second current time step to obtain the context scalar of the current time step; using the post-processing network to perform a linear projection on the context vector of the second current time step to obtain the context scalar of the current time step; The context vector of the second current time step is subjected to a second linear projection and the context vector after the second linear projection is subjected to a Mel spectrum prediction to obtain the Mel spectrum of the second current time step; according to the context scalar of the current time step, the probability of the completion of the Mel spectrum prediction is calculated using a preset first activation function; it is determined whether the probability of the completion of the Mel spectrum prediction is less than a preset threshold; when the probability of the completion of the Mel spectrum prediction is less than the threshold, the context vector of the second current time step is connected in series with the Mel spectrum of the second current time step, and the step of inputting into the preset double-layer long short-term memory layer is returned until the probability of the completion of the Mel spectrum prediction is less than the threshold, the Mel spectrum prediction is terminated, and a predicted Mel spectrum is obtained; The attention network includes an attention weight value, a linear layer, a second activation function, and a mapping function. The method of extracting the context vector from the hidden feature matrix using the preset attention network to obtain the context vector of the first current time step includes: using the linear layer to linearly project the hidden feature matrix to obtain a key matrix; inputting the attention weight value into a preset convolutional layer to generate a position feature matrix; using the linear layer to linearly project the position feature matrix to obtain an additional feature matrix; adding the additional feature matrix to the key matrix, and processing the addition result using the second activation function to obtain an attention probability vector; using the mapping function to map the attention probability vector to obtain a current attention weight vector; multiplying the current attention weight vector with the hidden feature matrix to obtain the context vector of the first current time step.

6. A speech synthesis device, characterized in that: The device comprises: A text acquisition module is used to acquire the speech text to be synthesized; A model speech synthesis module is used to perform speech synthesis on the speech text to be synthesized using a target speech synthesis model to obtain synthesized speech, wherein the target speech synthesis model is trained using the speech synthesis model training device as described in claim 5.

7. An electronic device, characterized in that: The electronic device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores computer program instructions that can be executed by the at least one processor, and the computer program instructions are executed by the at least one processor so that the at least one processor can execute the training method of the speech synthesis model as described in any one of claims 1 to 3, or execute the speech synthesis method as described in claim 4.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, it implements the training method of the speech synthesis model as described in any one of claims 1 to 3, or executes the speech synthesis method as described in claim 4.