Speech synthesis method, device, equipment and storage medium

Abstract

A speech synthesis method, apparatus, device, and storage medium. The speech synthesis method comprises: providing training corpus, the training corpus including speaker information, language information, and text information; performing feature extraction processing on the speaker information to obtain speaker features; performing language encoding processing on the language information to obtain a language encoding result; performing text encoding processing on the text information based on parameters updated after decoupling the speaker information and the language information to obtain a text encoding result; decoding the text encoding result, speaker features, and language encoding result to obtain an acoustic spectrum; and performing audio synthesis processing based on the acoustic spectrum to generate speech audio corresponding to the text information. Embodiments of the present invention improve the ability of speech synthesis to cross-translate across multiple languages ​​and reduce the cost of speech synthesis.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of speech processing, and in particular to a speech synthesis method and apparatus, device, and storage medium. Background Art

[0002] Speech synthesis, also known as text-to-speech (TTS) technology, is a key research area in speech processing, aiming to enable machines to generate natural-sounding human speech. Speech synthesis technology can be applied independently in different scenarios or integrated into an overall voice interaction solution as a final step.

[0003] Traditional speech synthesis technology is divided into front-end and back-end. The front-end is primarily responsible for language analysis and processing of text, including language, word segmentation, part-of-speech prediction, polyphone processing, prosody prediction, and sentiment. After predicting the pronunciation of the text, this information is sent to the TTS back-end system, which then uses this information to convert the content into speech. The back-end acoustic system has a long history of development, from the first generation of speech concatenation synthesis, to the second generation of speech parameter synthesis, to the third generation of end-to-end synthesis. The intelligence of the back-end acoustic system has gradually increased, and the level of detail and difficulty required for labeling has gradually decreased.

[0004] However, the language transfer capability of speech synthesis still needs to be improved. Summary of the Invention

[0005] The problem solved by the embodiments of the present invention is to provide a speech synthesis method and apparatus, a device and a storage medium, which improve the ability of speech synthesis to cross-migrate multiple languages ​​and save the cost of speech synthesis.

[0006] To solve the above problems, an embodiment of the present invention provides a speech synthesis method, comprising: providing training corpus, the training corpus including speaker information, language information, and text information; performing feature extraction processing on the speaker information to obtain speaker features; performing language encoding processing on the language information to obtain a language encoding result; performing text encoding processing on the text information based on parameters updated after decoupling processing of the speaker information and the language information to obtain a text encoding result; performing decoding processing on the text encoding result, speaker features, and the language encoding result to obtain an acoustic spectrum; and performing audio synthesis processing based on the acoustic spectrum to generate speech audio corresponding to the text information.

[0007] Correspondingly, an embodiment of the present invention also provides a speech synthesis device, including: a training corpus providing module, used to provide training corpus, the training corpus including speaker information, language information and text information; a speaker feature extractor, used to perform feature extraction processing on the speaker information to obtain speaker features; a language encoder, used to perform language encoding processing on the language information to obtain a language encoding result; a text encoder, used to perform text encoding processing on the text information based on parameters updated after decoupling processing from the speaker information and the language information to obtain a text encoding result; a decoder, used to decode the encoding result, speaker features and the language encoding result to obtain an acoustic spectrum; and an audio synthesis module, used to perform audio synthesis processing based on the acoustic spectrum to generate speech audio corresponding to the text information.

[0008] Correspondingly, an embodiment of the present invention also provides a device comprising at least one memory and at least one processor, wherein the memory stores one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement the speech synthesis method provided in an embodiment of the present invention.

[0009] Correspondingly, an embodiment of the present invention further provides a storage medium, wherein the storage medium stores one or more computer instructions, and the one or more computer instructions are used to implement the speech synthesis method provided by the embodiment of the present invention.

[0010] Compared with the prior art, the technical solution of the embodiment of the present invention has the following advantages:

[0011] The speech synthesis method provided by the embodiment of the present invention performs text encoding processing on the text information based on the parameters updated after decoupling the speaker information and the language information to obtain a text encoding result, which is conducive to separating the text encoding result from the language information and the speaker information, thereby improving the independence of the text encoding process from the language information and the speaker information, and correspondingly improving the generalization ability of the text encoding process, which is conducive to improving the ability of speech synthesis to cross-migrate to multiple languages. Accordingly, only a single language corpus is required to synthesize speech audio in different languages, saving the cost of speech synthesis.

[0012] In the speech synthesis device provided by the embodiment of the present invention, the text encoder performs text encoding processing on the text information based on the updated parameters after decoupling processing from the speaker information and the language information to obtain a text encoding result, which is conducive to separating the text encoder from the language information and the speaker information, thereby improving the independence between the text encoder and the language information and the speaker information, and correspondingly improving the generalization ability of the text encoder, which is conducive to improving the ability of the speech synthesis device to cross-migrate to multiple languages. Accordingly, only a single language corpus is required to synthesize speech audio in different languages, saving the cost of speech synthesis. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 1 is a flow chart of an embodiment of a speech synthesis method of the present invention;

[0014] Figure 2 yes Figure 1 Schematic diagram of the process of an embodiment of step S4;

[0015] Figure 3 is a functional block diagram of an embodiment of a speech synthesis device of the present invention;

[0016] Figure 4 This is a hardware structure diagram of a device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0017] As can be seen from the background technology, the language transfer capability of current speech synthesis still needs to be improved.

[0018] Specifically, speech synthesis typically uses phonemes or other linguistic features as input, limiting the model's capabilities to the linguistic characteristics of the training corpus. Typically, data from only one language per speaker is available, leading to a lack of relevant training data for the model and, consequently, poor language transfer capabilities.

[0019] If you want to synthesize text in multiple languages, you need to include cross-language training data during training. This training data usually requires recording by professionals who are proficient in multiple languages, making it extremely costly and difficult to obtain training data, which in turn increases the cost of speech synthesis.

[0020] In order to solve the above technical problems, an embodiment of the present invention provides a speech synthesis method. Figure 1 , which shows a flow chart of an embodiment of a speech synthesis method of the present invention.

[0021] In this embodiment, the speech synthesis method includes the following basic steps:

[0022] Step S1: providing training corpus, wherein the training corpus includes speaker information, language information, and text information;

[0023] Step S2: performing feature extraction processing on the speaker information to obtain speaker features;

[0024] Step S3: performing language coding processing on the language information to obtain a language coding result;

[0025] Step S4: Based on the updated parameters after the text encoder is decoupled from the speaker information and the language information, the text information is encoded to obtain a text encoding result;

[0026] Step S5: decoding the encoding result, speaker characteristics, and language encoding result to obtain an acoustic spectrum;

[0027] Step S6: Perform audio synthesis processing based on the acoustic spectrum to generate speech audio corresponding to the text information.

[0028] The speech synthesis method provided by the embodiment of the present invention performs text encoding processing on the text information based on the updated parameters after decoupling the speaker information and the language information to obtain a text encoding result. This is conducive to separating the text encoding process from the language information and the speaker information, thereby improving the independence of the text encoding process from the language information and the speaker information, and correspondingly improving the generalization ability of the text encoding process, which is conducive to improving the ability of speech synthesis to cross-migrate to multiple languages. Accordingly, only a single language corpus is required to synthesize speech audio in different languages, saving the cost of speech synthesis.

[0029] In order to make the above-mentioned objects, features and advantages of the embodiments of the present invention more obvious and easy to understand, specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0030] refer to Figure 1 , execute step S1: provide training corpus, the training corpus includes speaker information, language information and text information.

[0031] The training corpus is used as training data to provide speaker information, language information and text information, so that feature extraction processing can be performed on the speaker information, language encoding processing can be performed on the language information, and text encoding processing can be performed on the text information.

[0032] In this embodiment, the training corpus may be only monolingual corpus, and there is no need to record cross-language corpus, which is beneficial to reducing the cost of speech synthesis.

[0033] The training corpus is typically a large amount of spoken speech data recorded by one or more speakers, including transcript files and audio files. The transcript files contain text information and corresponding language information, while the audio files contain speaker information.

[0034] Specifically, the speaker information is used to identify the identity of the speaker, and may include any one or more of the speaker's timbre information, age information, and gender information.

[0035] As an example, speaker information may include tag information representing timbre information, age information, and gender information. For example, speaker information may include "young male," "young female," "middle-aged male," "middle-aged female," "old male," "old female," and "child." The timbre information may be represented by voiceprint information.

[0036] The text information is the input text corresponding to the speech audio to be synthesized. In this embodiment, the text information uses the International Phonetic Alphabet as input, and in order to achieve better cross-language migration, the phonemes of each language are also shared and uniformly mapped.

[0037] As an example, Chinese phonemes originally have 66 (excluding tones), which are mapped to 38 International Phonetic Symbols. Furthermore, to support Chinese, the corresponding pinyin tones are also included in the text message. In this embodiment, the text message contains five pinyin tones: the first, second, third, fourth, and light tones.

[0038] The language information can be obtained based on the text information. For example, based on the text information, the corresponding language can be determined, for example, whether the language is Chinese, English, French, German, etc. For example, based on the word form and tone information contained in the text input, the language information corresponding to the text information can be determined to be Chinese.

[0039] Continue to refer Figure 1 , executing step S2: performing feature extraction processing on the speaker information to obtain speaker features. Specifically, in this embodiment, the speaker information is input into a speaker feature extractor for feature extraction processing.

[0040] The speaker information is subjected to feature extraction processing to obtain speaker features, thereby being able to support multi-person speech synthesis.

[0041] As an embodiment, the plurality of speaker information is embedded in a corresponding speaker table; performing feature extraction processing on the speaker information to obtain the speaker features includes: using the speaker's identity ID, searching for the corresponding speaker information in the speaker table, and then obtaining the speaker's features.

[0042] Correspondingly, the speaker characteristics include: any one or more of the speaker's timbre information, age information, and gender information.

[0043] It should be noted that the above feature extraction process for the speaker information is only an example, and the manner of feature extraction process for the speaker information is not limited to this.

[0044] Continue to refer Figure 1 , executing step S3: performing language encoding processing on the language information to obtain a language encoding result. The language encoding processing is performed on the language information so that the language encoding result, the text encoding result and the speaker characteristics can be subsequently decoded to obtain an acoustic spectrum.

[0045] In this embodiment, the language information is input into the language encoder for language encoding. The language encoding result obtained by inputting the language information into the language encoder for language encoding is not input into the text encoder, thereby ensuring that the text encoder is independent of the language information.

[0046] As an embodiment, the step of performing language encoding processing on the language information includes: embedding multiple language information into a corresponding language table; processing the language information, searching for corresponding language information in the language table, and then obtaining the characteristics of the language.

[0047] Specifically, the language encoding result is language information.

[0048] Continue to refer Figure 1 , executing step S4: based on the updated parameters after decoupling the speaker information and the language information, performing text encoding processing on the text information to obtain a text encoding result.

[0049] Based on the parameters updated after decoupling the speaker information and the language information, the text information is subjected to text encoding processing to obtain a text encoding result. This is conducive to separating the text encoding process from the language information and the speaker information, thereby improving the independence of the text encoding process from the language information and the speaker information, and correspondingly improving the generalization ability of the text encoding process, which is conducive to improving the ability of speech synthesis to cross-migrate to multiple languages. Accordingly, only a single language corpus is required to synthesize speech audio in different languages, saving the cost of speech synthesis.

[0050] Combined with reference Figure 2 , showing Figure 1 FIG. 1 is a flow chart of an embodiment of step S1 in FIG.

[0051] In this embodiment, step S4 of performing text encoding processing on the text information based on the updated parameters after decoupling the speaker information and the language information includes:

[0052] refer to Figure 2 , execute step S41: input the text information into the text encoder for text encoding processing.

[0053] The text encoder is used to perform text encoding processing on text information to obtain a text encoding result.

[0054] In a specific implementation, the text encoder is used to extract sequence information contained in text information and compress it into a vector of fixed length as an encoded representation of the input text.

[0055] Specifically, in this embodiment, the text encoder converts the sequence information contained in the text information into a hidden layer representation, so that the hidden layer representation is subsequently decoded to obtain an acoustic spectrum.

[0056] As an embodiment, the text encoder may include: a character embedding layer (CharacterEmbedding), 3 convolutional layers and a recurrent neural network (RNN) layer connected in sequence.

[0057] The character embedding layer is used to encode the input text information into a character vector.

[0058] The convolutional layer is used to extract features from the input layer data, that is, to extract the implicit features of character embedding.

[0059] The RNN is a type of network with a recurrent connection structure. It can memorize the information input at the previous moment and store it in the memory unit. During calculation, it will comprehensively process the input sequence at the current moment and the previous memory content and then output it, completing the modeling of the input sequence to the output sequence, so that it can focus on the sequence information of a long time span and further consider the contextual content.

[0060] As one embodiment, the RNN may be a bidirectional LSTM (Long Short-Term Memory) layer. By introducing the concept of memory cells, the bidirectional LSTM layer can address the vanishing gradient problem of the RNN, allowing input information from long periods of time to influence the current output, thereby increasing the scope of captured context.

[0061] It should be noted that, in this embodiment, the above description of the text encoding process is only an example, and the implementation of the text encoder is not limited to this.

[0062] Continue to refer Figure 2 , executing step S42: performing a first decoupling process between the text encoder and the language information.

[0063] Performing a first decoupling process on the text encoder and the language information is beneficial to improving the independence between the text encoder and the language information, so that the parameters of the updated text encoder carry less language information, thereby improving the generalization ability of the text encoder and correspondingly enabling cross-migration between multiple languages.

[0064] In one embodiment, performing a first decoupling process between the text encoder and the language information includes: providing a first language classifier; and performing adversarial training on the text encoder and the first language classifier. In the field of natural language processing, adversarial training, as a regularization method, can improve the generalization ability of the model and also improve the robustness.

[0065] In other embodiments, other methods may be used to perform the first decoupling process between the text encoder and the language information.

[0066] As an embodiment, adversarial training of the text encoder and the first language classifier includes: based on the text encoding result and language information, using a loss function to update the parameters of the first language classifier, and passing the inverted gradient to the text encoder to perform parameter update processing on the text encoder.

[0067] That is, in this embodiment, a gradient reversal layer is provided between the text encoder and the first language classifier, and adversarial training is performed between the text encoder and the first language classifier using the gradient reversal layer.

[0068] Specifically, the text encoding result is output to the first language classifier, the language classification loss (the difference between the predicted value and the true value) is calculated, and the gradient is calculated based on the language classification loss. After the gradient is reversed, it is passed to the text encoder to update the parameters of the text encoder, thereby making the training objectives of the first language classifier and the text encoder before and after the gradient reversal layer opposite to each other, so as to achieve the effect of adversarial training.

[0069] In this embodiment, the loss function is a cross-entropy loss function, and the language classification loss is calculated using formula (1):

[0070] L language (φ;t i)=crossentropy(D(t i ),l i ) (I)

[0071] Among them, l i is the language label, t i is the text encoding result output by the text encoder, and D(.) is a fully connected mapping layer.

[0072] The language label can be obtained through language information. The fully connected mapping layer is used to map the text encoding result to the same dimension as the language label, and update the parameter φ of the first language classifier by calculating the cross entropy loss function.

[0073] The cross entropy loss function is a commonly used loss function. Therefore, in this embodiment, the cross entropy loss function is used as an example for description. The type of loss function is not limited to this. For example, in other embodiments, the loss function can also be a minimum mutual information loss function.

[0074] In this embodiment, after calculating the language classification loss of the first language classifier, the inverted gradient is transferred to the text encoder to perform parameter update processing on the text encoder.

[0075] Specifically, the gradient reversal layer is used to reversely update the parameters of the text encoder to achieve adversarial training.

[0076] In this embodiment, after adversarial training of the first language classifier and the text encoder, the loss of the text encoder is updated as formula (II) to update the parameters of the text encoder:

[0077]

[0078] Among them, O is the output after decoding (i.e., acoustic spectrum), x i For input text, l i is the language label, L mel (O|x i ,l i ,s) is the loss of acoustic spectrum, is the parameter of the text encoder, t i The text encoding result output by the text encoder.

[0079] Therefore, in this embodiment, when updating the parameters of the original text encoder When training the text encoder, the reverse gradient of the first language classifier is added, so that the text encoder can learn a language-independent text representation space.

[0080] It should be noted that the implementation method of performing the first decoupling process between the text encoder and the language information is not limited to this.

[0081] For example, in other embodiments, performing the first decoupling process on the text encoder and the language information may further include minimizing the mutual information between the text encoder and the language information.

[0082] Mutual information is used to represent the correlation between two variables. By minimizing the mutual information between the text encoder and the language information, the correlation between the text encoder and the language information is minimized, thereby improving the independence between the text encoder and the language information, and correspondingly improving the generalization ability of the text encoder.

[0083] In this embodiment, after minimizing the mutual information between the text encoder and the language information, the loss of the text encoder is updated as formula (III), and the parameters of the text encoder are then updated:

[0084]

[0085] Among them, O is the output after decoding (i.e., acoustic spectrum), x i For input text, l i is the language label, φ contains the language encoding parameters and the corresponding fully connected structure parameters, L mel (O|x i ,l i ,s) is the loss of acoustic spectrum, is the parameter of the text encoder, t i is the text encoding result output by the text encoder, I CLUBlanguage (φ; l i ;t i ) is the mutual information between the text encoder and the language information. As an example, φ includes a language table and corresponding deep neural network (DNN) parameters.

[0086] In this embodiment, the I CLUBlanguage (φ; l i ;t i ) is calculated by formula (IV):

[0087] I CLUBlanguage (φ; l i ,t i )=E p(l,t) [logp(t i |l i )]-E p(l) E p(t)[logp(t i |l i )] (IV)

[0088] Among them, t i is the text result output by the text encoder, l i is the language label, E p is the expected probability distribution, and p(.) is the probability distribution.

[0089] It should be noted that the above minimization process of the mutual information between the text encoder and the language information is only an example, and the manner of minimizing the mutual information is not limited thereto.

[0090] Continue to refer Figure 2 , executing step S43: performing a second decoupling process between the text encoder and the speaker information. Performing the second decoupling process between the text encoder and the speaker information helps improve the independence between the text encoder and the speaker information, so that the updated text encoder parameters carry less speaker information, improving the generalization capability of the text encoder and enabling cross-language transfer.

[0091] In one embodiment, performing the second decoupling process between the text encoder and the speaker information includes: providing a first speaker classifier; and performing adversarial training on the text encoder and the first speaker classifier. In the field of natural language processing, adversarial training, as a regularization method, can improve the generalization ability of the model while also enhancing its robustness.

[0092] In other embodiments, other methods may be used to perform the second decoupling process between the text encoder and the speaker information.

[0093] As an embodiment, adversarial training between the text encoder and the first speaker classifier includes: based on the text encoding result and speaker information, using a loss function to update the parameters of the first speaker classifier, and passing the inverted gradient to the text encoder to perform parameter update processing on the text encoder.

[0094] That is, in this embodiment, a gradient reversal layer is also provided between the text encoder and the first speaker classifier, and adversarial training is performed between the text encoder and the first speaker classifier using the gradient reversal layer.

[0095] Specifically, the text encoding result is output to the first speaker classifier, the loss of speaker classification (the difference between the predicted value and the true value) is calculated, and the gradient is calculated based on the loss of speaker classification. After reversing the gradient, it is passed to the text encoder to update the parameters of the text encoder, thereby making the training objectives of the first speaker classifier and the text encoder before and after the gradient reversal layer opposite to each other, so as to achieve the effect of adversarial training.

[0096] In this embodiment, the loss function is a cross-entropy loss function, and the speaker classification loss is calculated using formula (V):

[0097] L speaker (ω;t i )=crossentropy(D(t i ),s) (V)

[0098] Among them, s is the speaker label, t i is the text encoding result output by the text encoder, and D(.) is a fully connected mapping layer.

[0099] The speaker label can be obtained through training corpus, and the fully connected mapping layer is used to map the text encoding result to the same dimension as the type of the training label, and update the parameter ω of the first speaker classifier by calculating the cross entropy loss function.

[0100] In other embodiments, the loss function may also be a mutual information loss function that is minimized.

[0101] In this embodiment, after calculating the speaker classification loss of the first speaker classifier, the inverted gradient is transferred to the text encoder, and parameter update processing is performed on the text encoder.

[0102] Specifically, the gradient reversal layer is used to reversely update the parameters of the text encoder to achieve adversarial training.

[0103] In this embodiment, after adversarial training is performed between the first language classifier and the text encoder, and adversarial training is performed between the first speaker classifier and the text encoder, the loss functions of the text encoder, the first speaker classifier, and the first language classifier are updated to Formula (VI), and the parameters of the text encoder, the first speaker classifier, and the first language classifier are then updated:

[0104]

[0105] Among them, O is the output after decoding (i.e., acoustic spectrum), xi For input text, l i is the language label, L mel (O|x i ,l i ,s) is the loss of acoustic spectrum, are the parameters of the text encoder, φ are the parameters of the first language classifier, and ω are the parameters of the first speaker classifier.

[0106] Therefore, in this embodiment, when updating the parameters of the original text encoder When training the text encoder, the reverse gradient of the first language classifier and the first speaker classifier are added, so that the text encoder and the first language classifier are trained adversarially, and the text encoder and the first speaker classifier are trained adversarially, so that the text encoder can learn a text representation space independent of the language and speaker.

[0107] It should be noted that the implementation of the second decoupling process between the text encoder and the speaker information is not limited to this. For example, in other embodiments, the second decoupling process between the text encoder and the speaker information may further include minimizing the mutual information between the text encoder and the speaker information.

[0108] Mutual information is used to represent the correlation between two variables. By minimizing the mutual information between the text encoder and the speaker information, the correlation between the text encoder and the speaker information is minimized, thereby improving the independence between the text encoder and the speaker information, and correspondingly improving the generalization ability of the text encoder.

[0109] In this embodiment, after minimizing the mutual information between the text encoder and the language information, and minimizing the mutual information between the text encoder and the speaker information, the loss of the text encoder, the speaker information parameters, and the language information parameters are updated to Formula (VII):

[0110]

[0111] Among them, L mel (O|x i ,l i ,s) is the loss of acoustic spectrum, is the parameter of the text encoder, φ includes the language encoding parameters and the corresponding fully connected structure parameters, ω includes the speaker encoding parameters and the corresponding fully connected structure parameters, I CLUBspeaker (ω;s,t i) is the mutual information between the text encoder and the speaker information. As an example, φ includes a language table and corresponding deep neural network (DNN) parameters. As an example, ω is a speaker table and corresponding deep neural network (DNN) parameters.

[0112] In this embodiment, the I CLUBspeaker (ω;s,t i ) is calculated by formula (VIII):

[0113] I CLUBspeaker (ω;s,t i )=E p(s,t) [logp(t i |s)]-E p(s) E p(t) [logp(t i |s)] (VIII)

[0114] Among them, t i is the text encoding result output by the text encoder, s is the speaker label, E p is the expected probability distribution, and p(.) is the probability distribution.

[0115] It should be noted that the above minimization process of the mutual information between the text encoder and the first speaker information is only an example, and the manner of minimizing the mutual information is not limited to this.

[0116] It should also be noted that, continue to refer to Figure 1 In this embodiment, the speech synthesis method further includes: executing step S7: obtaining speech residual information of the training corpus based on the acoustic spectrum.

[0117] The speech residual information of the training corpus is obtained, and then the text results, speaker characteristics and language encoding results can be decoded based on the speech residual information to obtain an acoustic spectrum, so that the style and rhythm of the speech audio synthesized based on the acoustic spectrum can be controlled. For example, factors such as the stress, pitch, rhythm, and emotion of the speech can be changed to make the synthesized speech audio more natural and expressive, thereby correspondingly improving the control ability of the synthesized speech audio.

[0118] In this embodiment, the speech residual information includes one or more of speech speed, pitch, tone, volume, stress, rhythm and emotion.

[0119] In this embodiment, obtaining speech residual information of the training corpus based on the acoustic spectrum includes: inputting the acoustic spectrum into a residual coding module to perform residual coding processing to obtain the speech residual information.

[0120] Specifically, the residual coding module can capture some unsupervised information as speech residual information.

[0121] In this embodiment, the residual coding module includes a variational auto-encoder (VAE). Variational auto-encoders are an important type of generative model. Unlike traditional autoencoders that describe the latent space numerically, they describe the observation of the latent space in a probabilistic way, showing great application value in data generation. In the speech synthesis model, training optimization will guide the VAE to capture some unsupervised information, such as speaking rate, pitch, volume, etc.

[0122] In other embodiments, the residual coding module may also be other types of encoders, such as Deterministic Auto-Encoders (DAE) or a normalizing flow module.

[0123] In this embodiment, the speech synthesis method further includes: providing a second language classifier; and performing a third decoupling process on the residual coding module and the second language classifier.

[0124] Performing a third decoupling process on the residual coding module and the second language classifier is beneficial to separating the residual coding module from the language information, thereby improving the independence between the speech residual information and the language information, and correspondingly helping to reduce the influence of the language information on the style and rhythm of the synthesized speech audio.

[0125] Specifically, the step of performing a third decoupling process on the residual coding module and the second language classifier may include: performing adversarial training on the residual coding module and the second language classifier; or minimizing the mutual information between the residual coding module and the second language classifier.

[0126] Among them, adversarial training of the residual coding module and the second language classifier includes: based on the speech residual information and language information, using the loss function to update the parameters of the second language classifier, and passing the inverted gradient to the residual coding module to perform parameter update processing on the residual coding module.

[0127] In this embodiment, the speech synthesis method further includes: providing a second speaker classifier; and performing a fourth decoupling process on the residual coding module and the second speaker classifier.

[0128] Performing a fourth decoupling process on the residual coding module and the second speaker classifier is beneficial for separating the residual coding module from the speaker information, thereby improving the independence between the speech residual information and the speaker information, and correspondingly helping to reduce the influence of the speaker information on the style and rhythm of the synthesized speech audio.

[0129] Specifically, the step of performing fourth decoupling processing on the residual coding module and the second speaker classifier includes: performing adversarial training on the residual coding module and the second speaker classifier; or, minimizing the mutual information between the residual coding module and the second speaker classifier.

[0130] Among them, adversarial training of the residual coding module and the second speaker classifier may include: based on the speech residual information and speaker information, using a loss function to update the parameters of the second speaker classifier, and passing the inverted gradient to the residual coding module to perform parameter update processing on the residual coding module.

[0131] For specific details of the third decoupling process and the fourth decoupling process, reference may be made to the aforementioned description of the first decoupling process and the second decoupling process, which will not be repeated in this embodiment.

[0132] It should also be noted that in this embodiment, the speech synthesis method further includes performing a fifth decoupling process on the speaker feature extractor and the language encoder. By performing the fifth decoupling process on the speaker feature extractor and the language encoder, the independence between speaker information and language information is improved, further reducing the difficulty of cross-language transfer in speech synthesis.

[0133] In this embodiment, performing the fifth decoupling process on the speaker feature extractor and the language encoder includes: performing a minimization process on the mutual information between the speaker feature extractor and the language encoder.

[0134] By minimizing the mutual information between the speaker feature extractor and the language encoder, the fifth decoupling process is made independent of data labels, thereby facilitating independence between the speaker information and the language information.

[0135] In other embodiments, the step of performing the fifth decoupling process may further include: providing a third speaker classifier; performing adversarial training between the third speaker classifier and the language encoder; providing a third language classifier; and performing adversarial training between the third language classifier and the speaker feature extractor.

[0136] Among them, the adversarial training between the third speaker classifier and the language encoder may include: based on the language encoding result and speaker information, using a loss function to update the parameters of the third speaker classifier, and transferring the inverted gradient to the language encoder to perform parameter update processing on the language encoder.

[0137] The adversarial training between the third language classifier and the speaker feature extractor includes: based on the speaker features and language information, using a loss function to update the parameters of the third language classifier, and transferring the inverted gradient to the speaker feature extractor to perform parameter update processing on the speaker feature extractor.

[0138] Continue to refer Figure 1 , executing step S5: decoding the text encoding result, speaker characteristics, and language encoding result to obtain an acoustic spectrum. The acoustic spectrum is used as an acoustic representation for speech synthesis input of the speech audio, so that a speech waveform can be subsequently synthesized based on the acoustic spectrum.

[0139] In this embodiment, decoding the text encoding result, speaker characteristics, and language encoding result includes: decoding the text encoding result, speaker characteristics, and language encoding result based on the speech residual information, thereby being able to control the style and rhythm of the speech audio synthesized based on the acoustic spectrum, making the synthesized speech audio more natural and expressive, and correspondingly improving the control ability of the synthesized speech audio.

[0140] Specifically, in this embodiment, the speech residual information, the text encoding result, the speaker characteristics and the language encoding result are input into a decoder for decoding processing.

[0141] The decoder predicts the output acoustic spectrum from the encoded input sequence, and uses the context vector obtained by encoding to obtain the target output sequence after certain transformations.

[0142] In this embodiment, the acoustic spectrum is a Mel spectrogram. As the acoustic representation of the model, the Mel spectrogram can enhance the details of low-frequency information in speech data, thereby improving the intelligibility of synthesized speech. Furthermore, after the frequency axis undergoes a nonlinear transformation using the Mel scale, the spectral content can be represented with relatively fewer dimensions, which facilitates information compression. Furthermore, because the Mel spectrogram is phase-invariant, it is smoother than a waveform representation, making it easier to train using the mean squared error (MSE) loss.

[0143] Continue to refer Figure 1, executing step S6: performing audio synthesis processing based on the acoustic spectrum to generate speech audio corresponding to the text information. Specifically, a vocoder can be used to perform audio synthesis processing. The vocoder converts the acoustic spectrum into a speech waveform (i.e., speech audio). The vocoder can be Wavenet, WaveRNN, Parallel WaveNet, WaveGlow, etc.

[0144] In order to solve the above problem, the present invention also provides a speech synthesis device. Figure 3 The following is a functional block diagram of an embodiment of a speech synthesis device according to the present invention.

[0145] refer to Figure 3 The speech synthesis device includes: a training corpus providing module 10, for providing training corpus, wherein the training corpus includes speaker information, language information and text information; a speaker feature extractor 20, for performing feature extraction processing on the speaker information to obtain speaker features; a language encoder 30, for performing language encoding processing on the language information to obtain a language encoding result; a text encoder 40, for performing text encoding processing on the text information based on parameters updated after decoupling processing from the speaker information and the language information to obtain a text encoding result; a decoder 50, for decoding the encoding result, speaker features and the language encoding result to obtain an acoustic spectrum; and an audio synthesis module 60, for performing audio synthesis processing based on the acoustic spectrum to generate speech audio corresponding to the text information.

[0146] In the speech synthesis device, the text encoder 40 performs text encoding on the text information based on the updated parameters after decoupling from the speaker information and the language information to obtain a text encoding result, which is conducive to separating the text encoder 40 from the language information and the speaker information, thereby improving the independence between the text encoder 40 and the language information and the speaker information, and correspondingly improving the generalization ability of the text encoder 40, which is conducive to improving the ability of the speech synthesis device to cross-migrate to multiple languages. Accordingly, only a single language corpus is required to synthesize speech audio in different languages, saving the cost of speech synthesis.

[0147] The training corpus providing module 10 is used to provide training corpus, which includes speaker information, language information and text information.

[0148] The training corpus is used as training data to provide speaker information, language information and text information, so that the speaker feature extractor 20 can perform feature extraction processing on the speaker information, the language encoder 30 can perform language encoding processing on the language information, and the text encoder 40 can perform text encoding processing on the text information.

[0149] In this embodiment, the training corpus may be only monolingual corpus, and there is no need to record cross-language corpus, which is beneficial to reducing the cost of speech synthesis.

[0150] The training corpus is typically a large amount of spoken speech data recorded by one or more speakers, including transcript files and audio files. The transcript files contain text information and corresponding language information, while the audio files contain speaker information.

[0151] Specifically, the speaker information is used to identify the identity of the speaker, and may include any one or more of the speaker's timbre information, age information, and gender information.

[0152] As an example, speaker information may include tag information representing timbre information, age information, and gender information. For example, speaker information may include "young male," "young female," "middle-aged male," "middle-aged female," "old male," "old female," and "child." The timbre information may be represented by voiceprint information.

[0153] The text information is the input text corresponding to the voice audio to be synthesized.

[0154] In this embodiment, the text information uses the International Phonetic Alphabet as input, and in order to achieve better cross-language migration, the phonemes of various languages ​​are shared and uniformly mapped.

[0155] As an example, Chinese phonemes originally have 66 (excluding tones), which are mapped to 38 International Phonetic Symbols. Furthermore, to support Chinese, the corresponding pinyin tones are also included in the text message. In this embodiment, the text message contains five pinyin tones: the first, second, third, fourth, and light tones.

[0156] The language information can be obtained based on the text information. For example, based on the text information, the corresponding language can be determined, for example, whether the language is Chinese, English, French, German, etc. For example, based on the word form and tone information contained in the text input, the language information corresponding to the text information can be determined to be Chinese.

[0157] The speaker feature extractor 20 is used to perform feature extraction processing on the speaker information to obtain speaker features, thereby supporting multi-person speech synthesis.

[0158] In one embodiment, the speaker information is embedded in a corresponding speaker table. Feature extraction is performed on the speaker information, and the speaker feature extractor 20 obtains the speaker features by searching the speaker table for the corresponding speaker information using the speaker's ID, thereby obtaining the speaker's features. Accordingly, the speaker features include any one or more of the speaker's timbre, age, and gender.

[0159] It should be noted that the above feature extraction process for the speaker information is only an example, and the manner of feature extraction process for the speaker information is not limited to this.

[0160] The language encoder 30 is used to perform language encoding processing on the language information to obtain a language encoding result, so that the decoder 50 can decode the language encoding result, the text encoding result and the speaker characteristics to obtain an acoustic spectrum.

[0161] In this embodiment, the language information is input to the language encoder 30 for language encoding, and the obtained language encoding result is not input to the text encoder 40, thereby ensuring that the text encoder 40 is independent of the language information.

[0162] As an embodiment, the language encoder 30 embeds the plurality of language information into a corresponding language table, processes the language information, searches for the corresponding language information in the language table, and thereby obtains the characteristics of the language. Specifically, the language encoding result includes the language information.

[0163] The text encoder 40 is used to perform text encoding processing on the text information based on the parameters updated after the speaker information and the language information are decoupled. This is beneficial for separating the text encoder 40 from the language information and the speaker information, thereby improving the independence between the text encoder 40 and the language information and the speaker information, and correspondingly improving the generalization ability of the text encoder 40, which is beneficial for improving the ability of speech synthesis to cross-migrate to multiple languages. Accordingly, only a single language corpus is required to synthesize speech audio in different languages, saving the cost of speech synthesis.

[0164] The text encoder 40 is used to perform text encoding processing on the text information to obtain a text encoding result.

[0165] In a specific implementation, the text encoder 40 is used to extract sequence information contained in the text information and compress it into a vector of fixed length as the encoded representation of the input text.

[0166] Specifically, in this embodiment, the text encoder 40 converts the sequence information contained in the text information into a hidden layer representation, so as to subsequently perform decoding processing on the hidden layer representation to obtain an acoustic spectrum.

[0167] As an embodiment, the text encoder 40 may include: a character embedding layer (Character Embedding), 3 convolutional layers and a recurrent neural network (RNN) layer connected in sequence.

[0168] The character embedding layer is used to encode the input text information into a character vector.

[0169] The convolutional layer is used to extract features from the input layer data, that is, to extract the implicit features of character embedding.

[0170] The RNN is a type of network with a recurrent connection structure. It can memorize the information input at the previous moment and store it in the memory unit. During calculation, it will comprehensively process the input sequence at the current moment and the previous memory content and then output it, completing the modeling of the input sequence to the output sequence, so that it can focus on the sequence information of a long time span and further consider the contextual content.

[0171] As one embodiment, the RNN may be a bidirectional LSTM (Long Short-Term Memory) layer. By introducing the concept of memory cells, the bidirectional LSTM layer can address the vanishing gradient problem of the RNN, allowing input information from long periods of time to influence the current output, thereby increasing the scope of captured context.

[0172] It should be noted that, in this embodiment, the above description of the text encoding process is only an example, and the implementation method of the text encoding process performed by the text encoder 40 is not limited to this.

[0173] In this embodiment, the text encoder 40 further includes: a first decoupling module (not shown) for performing a first decoupling process between the encoding module and the language information.

[0174] The first decoupling module performs a first decoupling process on the text encoder 40 and the language information, which is beneficial to improving the independence between the text encoder 40 and the language information, so that the parameters of the updated text encoder 40 carry less language information, thereby improving the generalization ability of the text encoder 40 and correspondingly enabling cross-migration between multiple languages.

[0175] As an embodiment, the first decoupling module includes: a first language classifier; and a first adversarial trainer for performing adversarial training on the text encoder 40 and the first language classifier.

[0176] In the field of natural language processing, adversarial training, as a regularization method, can improve the generalization ability of the model and also improve the robustness. In other embodiments, other methods can also be used to perform the first decoupling process between the text encoder and the language information.

[0177] As an embodiment, the first adversarial trainer updates the parameters of the first language classifier using a loss function based on the text encoding result and language information, and passes the inverted gradient to the text encoder 40 to perform parameter update processing on the text encoder 40.

[0178] That is, in this embodiment, a gradient reversal layer is provided between the text encoder 40 and the first language classifier, and adversarial training is performed between the text encoder 40 and the first language classifier using the gradient reversal layer.

[0179] Specifically, the text encoding result is output to the first language classifier, the language classification loss (the difference between the predicted value and the true value) is calculated, and the gradient is calculated based on the language classification loss. After the gradient is reversed, it is passed to the text encoder 40 to update the parameters of the text encoder 40, thereby making the training objectives of the first language classifier and the text encoder 40 before and after the gradient reversal layer opposite to each other, so as to achieve the effect of adversarial training.

[0180] In this embodiment, the loss function is a cross-entropy loss function, and the language classification loss is calculated using formula (1):

[0181] L language (φ;t i )=crossentropy(D(t i ),l i ) (I)

[0182] Among them, l i is the language label, t i is the text encoding result output by the text encoder, and D(.) is a fully connected mapping layer.

[0183] The language label can be obtained through language information. The fully connected mapping layer is used to map the text encoding result to the same dimension as the language label, and update the parameter φ of the first language classifier by calculating the cross entropy loss function.

[0184] The cross entropy loss function is a commonly used loss function. Therefore, in this embodiment, the cross entropy loss function is used as an example for description. The type of loss function is not limited to this. For example, in other embodiments, the loss function can also be a minimum mutual information loss function.

[0185] In this embodiment, after calculating the language classification loss of the first language classifier, the first adversarial trainer transfers the inverted gradient to the text encoder 40 to perform parameter update processing on the text encoder 40.

[0186] Specifically, the gradient reversal layer is used to inversely update the parameters of the text encoder 40 to achieve adversarial training.

[0187] In this embodiment, after adversarial training is performed between the first language classifier and the text encoder 40, the loss of the text encoder 40 is updated to Formula (II) to update the parameters of the text encoder 40:

[0188]

[0189] Among them, O is the output after decoding (i.e., acoustic spectrum), x i For input text, l i is the language label, L mel (O|x i ,l i ,s) is the loss of acoustic spectrum, is the parameter of the text encoder, t i The text encoding result output by the text encoder.

[0190] Therefore, in this embodiment, when updating the parameters of the original text encoder When the text encoder 40 is trained, the reverse gradient of the first language classifier is added, so that the text encoder 40 and the first language classifier are trained adversarially, so that the text encoder 40 can learn a text representation space independent of the language.

[0191] It should be noted that the implementation manner in which the first decoupling module performs the first decoupling process between the text encoder 40 and the language information is not limited thereto.

[0192] For example, in other embodiments, the first decoupling module may also be a first mutual information processing module, configured to minimize the mutual information between the text encoder and the language information.

[0193] Mutual information is used to represent the correlation between two variables. By minimizing the mutual information between the text encoder and the language information, the correlation between the text encoder and the language information is minimized, thereby improving the independence between the text encoder and the language information, and correspondingly improving the generalization ability of the text encoder.

[0194] In this embodiment, after minimizing the mutual information between the text encoder and the language information, the loss of the text encoder is updated as formula (III), and the parameters of the text encoder are then updated:

[0195]

[0196] Among them, O is the output after decoding (i.e., acoustic spectrum), x i For input text, l i is the language label, φ contains the language encoding parameters and the corresponding fully connected structure parameters, L(O|x i ,l i ,s) is the loss of acoustic spectrum, is the parameter of the text encoder, t i is the text encoding result output by the text encoder, I CLUBlanguage (φ; l i ;t i ) is the mutual information between the text encoder and the language information. As an example, φ includes a language table and corresponding deep neural network (DNN) parameters.

[0197] In this embodiment, the I CLUBlanguage (φ; l i ;t i ) is calculated by formula (IV):

[0198] I CLUBlanguage (φ; l i ,t i )=E p(l,t) [logp(t i |l i )]-E p(l) E p(t) [logp(t i |l i )] (IV)

[0199] Among them, t i is the text result output by the text encoder, l i is the language label, E p is the expected probability distribution, and p(.) is the probability distribution.

[0200] It should be noted that the above minimization process of the mutual information between the text encoder and the language information is only an example, and the manner of minimizing the mutual information is not limited thereto.

[0201] In this embodiment, the text encoder 40 also includes: a second decoupling module (not shown), which is used to perform a second decoupling process between the encoding module and the speaker information, which is beneficial to improving the independence between the text encoder 40 and the speaker information, so that the parameters of the updated text encoder 40 carry less speaker information, thereby improving the generalization ability of the text encoder 40, and correspondingly enabling cross-migration between multiple languages.

[0202] As an embodiment, the second decoupling module includes: a first speaker classifier; and a second adversarial trainer, configured to perform adversarial training on the text encoder 40 and the first speaker classifier.

[0203] In the field of natural language processing, adversarial training, as a regularization method, can improve the generalization ability of the model while also improving its robustness.

[0204] In other embodiments, the second decoupling module may also use other methods to perform the second decoupling process between the text encoder and the speaker information.

[0205] As an embodiment, the second adversarial trainer updates the parameters of the first speaker classifier based on the text encoding result and speaker information using a loss function, and passes the inverted gradient to the text encoder 40 to perform parameter update processing on the text encoder 40.

[0206] That is, in this embodiment, the second adversarial trainer also sets a gradient reversal layer between the text encoder 40 and the first speaker classifier, and uses the gradient reversal layer to perform adversarial training between the text encoder 40 and the first speaker classifier.

[0207] Specifically, the text encoding result is output to the first speaker classifier, the loss of speaker classification (the difference between the predicted value and the true value) is calculated, and the gradient is calculated based on the loss of speaker classification. After the gradient is reversed, it is passed to the text encoder 40 to update the parameters of the text encoder 40, thereby making the training objectives of the first speaker classifier and the text encoder 40 before and after the gradient reversal layer opposite to each other, so as to achieve the effect of adversarial training.

[0208] In this embodiment, the loss function is a cross entropy loss function, and the speaker classification loss is calculated using formula (V):

[0209] Lspeaker (ω;t i )=crossentropy(D(t i ),s) (V)

[0210] Among them, s is the speaker label, t i is the text encoding result output by the text encoder, and D(.) is a fully connected mapping layer.

[0211] The speaker label can be obtained through training corpus, and the fully connected mapping layer is used to map the text encoding result to the same dimension as the type of the training label, and update the parameter ω of the first speaker classifier by calculating the cross entropy loss function.

[0212] In other embodiments, the loss function may also be a mutual information loss function that is minimized.

[0213] In this embodiment, after calculating the speaker classification loss of the first speaker classifier, the second adversarial trainer passes the inverted gradient to the text encoder 40 to perform parameter update processing on the text encoder 40. Specifically, the parameters of the text encoder 40 are inversely updated using the gradient reversal layer to implement adversarial training.

[0214] In this embodiment, after adversarial training is performed between the first language classifier and the text encoder 40, and adversarial training is performed between the first speaker classifier and the text encoder, the loss functions of the text encoder 40, the first speaker classifier, and the first language classifier are updated to Formula (VI), and the parameters of the text encoder 40, the first speaker classifier, and the first language classifier are then updated:

[0215]

[0216] Among them, O is the output after decoding (i.e., acoustic spectrum), x i For input text, l i is the language label, L mel (O|x i ,l i ,s) is the loss of acoustic spectrum, are the parameters of the text encoder, φ are the parameters of the first language classifier, and ω are the parameters of the first speaker classifier.

[0217] Therefore, in this embodiment, when updating the parameters of the original text encoder 40 When the text encoder 40 is trained, the reverse gradient of the first language classifier and the first speaker classifier are added, so that the text encoder 40 can be trained adversarially with the first language classifier, and the text encoder 40 can be trained adversarially with the first speaker classifier, so that the text encoder 40 can learn a text representation space that is independent of the language and the speaker.

[0218] It should be noted that the implementation manner in which the second decoupling module performs the second decoupling process between the text encoder 40 and the speaker information is not limited thereto.

[0219] For example, in other embodiments, the second decoupling module may also be a second mutual information processing module, configured to minimize the mutual information between the text encoder and the speaker information.

[0220] Mutual information is used to represent the correlation between two variables. By minimizing the mutual information between the text encoder and the speaker information, the correlation between the text encoder and the speaker information is minimized, thereby improving the independence between the text encoder and the speaker information, and correspondingly improving the generalization ability of the text encoder.

[0221] In this embodiment, after minimizing the mutual information between the text encoder and the language information, and minimizing the mutual information between the text encoder and the speaker information, the loss of the text encoder, the speaker information parameter, and the language information parameter are updated to Formula (VII):

[0222]

[0223] Among them, L mel (O|x i ,l i ,s) is the loss of acoustic spectrum, is the parameter of the text encoder, φ includes the language encoding parameters and the corresponding fully connected structure parameters, ω includes the speaker encoding parameters and the corresponding fully connected structure parameters, I CLUBspeaker (ω;s,t i ) is the mutual information between the text encoder and the speaker information. As an example, φ includes a language table and corresponding deep neural network (DNN) parameters. As an example, ω is a speaker table and corresponding deep neural network (DNN) parameters.

[0224] In this embodiment, the I CLUBspeaker (ω;s,t i ) is calculated by formula (VIII):

[0225] I CLUBspeaker (ω;s,t i )=E p(s,t) [logp(t i |s)]-E p(s) E p(t) [logp(t i |s)] (VIII)

[0226] Among them, t i is the text encoding result output by the text encoder, s is the speaker label, E p is the expected probability distribution, and p(.) is the probability distribution.

[0227] It should be noted that the above minimization processing of the mutual information between the text encoder and the first speaker information by the second mutual information processing module is only an example, and the manner in which the second mutual information processing module minimizes the mutual information is not limited thereto.

[0228] It should also be noted that, continue to refer to Figure 3 In this embodiment, the speech synthesis device further includes: a residual coding module 70 for obtaining speech residual information of the training corpus based on the acoustic spectrum, and inputting the speech residual information into the decoder 50.

[0229] The residual coding module 70 obtains the speech residual information of the training corpus, so that the text encoder 40 can decode the text result, speaker characteristics and the language encoding result based on the speech residual information to obtain the acoustic spectrum, so that the speech synthesis device can control the style and rhythm of the speech audio synthesized based on the acoustic spectrum, for example: changing the stress, pitch, rhythm, emotion and other factors of the speech, so that the synthesized speech audio is more natural and expressive, and the control ability of the synthesized speech audio is correspondingly improved.

[0230] In this embodiment, the speech residual information includes one or more of speech speed, pitch, tone, volume, stress, rhythm and emotion.

[0231] Specifically, the residual encoding module 70 can capture some unsupervised information as the residual information.

[0232] In this embodiment, the residual encoding module 70 includes a variational autoencoder (VAE). Variational autoencoders are an important type of generative model. Unlike traditional autoencoders that describe the latent space numerically, they describe the observation of the latent space probabilistically, showing great application value in data generation. In speech synthesis models, training optimization guides the VAE to capture some unsupervised information, such as speech rate, pitch, and volume.

[0233] In other embodiments, the residual coding module may also be other types of encoders, such as a deterministic autoencoder (DAE) or a normalizing flow module.

[0234] In this embodiment, the speech synthesis apparatus further includes: a second language classifier; and a third decoupling module configured to perform a third decoupling process on the residual coding module 70 and the second language classifier.

[0235] The third decoupling module performs a third decoupling process on the residual coding module 70 and the second language classifier, which is beneficial to separate the residual coding module 70 from the language information, thereby improving the independence between the speech residual information and the language information, and correspondingly helping to reduce the influence of the language information on the style and rhythm of the synthesized speech audio.

[0236] Specifically, the third decoupling module may include: a third adversarial trainer, configured to perform adversarial training on the residual coding module and the second language classifier; or a third mutual information processing module, configured to minimize the mutual information between the residual coding module and the second language classifier.

[0237] Among them, the third adversarial trainer can update the parameters of the second language classifier based on the speech residual information and language information using the loss function, and pass the inverted gradient to the residual coding module to perform parameter update processing on the residual coding module.

[0238] In this embodiment, the speech synthesis apparatus further includes: a second speaker classifier; and a fourth decoupling module configured to perform a fourth decoupling process on the residual coding module 70 and the second speaker classifier.

[0239] The fourth decoupling module performs a fourth decoupling process on the residual coding module and the second speaker classifier, which is beneficial to separating the residual coding module 70 from the speaker information, thereby improving the independence between the speech residual information and the speaker information, and correspondingly helping to reduce the influence of the speaker information on the style and rhythm of the synthesized speech audio.

[0240] Specifically, the fourth decoupling module may include: a fourth adversarial trainer, used to perform adversarial training on the residual coding module 70 and the second speaker classifier; or a fourth mutual information processing module, used to minimize the mutual information between the residual coding module 70 and the second speaker classifier.

[0241] Among them, the fourth adversarial trainer can update the parameters of the second speaker classifier based on the speech residual information and speaker information using the loss function, and pass the inverted gradient to the residual coding module to perform parameter update processing on the residual coding module.

[0242] For specific details of the third decoupling module and the fourth decoupling module, reference may be made to the aforementioned description of the first decoupling module and the second decoupling module, which will not be repeated herein in this embodiment.

[0243] It should also be noted that, in this embodiment, the speech synthesis device also includes: a fifth decoupling module (not shown), which is used to perform fifth decoupling processing on the speaker feature extractor 20 and the language encoder 30, thereby improving the independence between the speaker information and the language information, and further reducing the difficulty of cross-language cross-migration of speech synthesis.

[0244] In this embodiment, the fifth decoupling module may be a fifth mutual information processing module, configured to minimize the mutual information between the speaker feature extractor 20 and the language encoder 30 .

[0245] By minimizing the mutual information between the speaker feature extractor 20 and the language encoder 30, the fifth decoupling process is made independent of data labels, thereby facilitating independence between the speaker information and the language information.

[0246] In other embodiments, the fifth decoupling module may further include: a third speaker classifier; a fifth adversarial trainer for performing adversarial training between the third speaker classifier and the language encoder; a third language classifier; and a sixth adversarial trainer for performing adversarial training between the third language classifier and the speaker feature extractor.

[0247] Among them, the fifth adversarial trainer can update the parameters of the third speaker classifier using a loss function based on the language encoding result and speaker information, and pass the inverted gradient to the language encoder to perform parameter update processing on the language encoder.

[0248] The sixth adversarial trainer can update the parameters of the third language classifier using a loss function based on the speaker features and language information, and pass the inverted gradient to the speaker feature extractor to perform parameter update processing on the speaker feature extractor.

[0249] The decoder 50 is used to decode the text encoding result, speaker characteristics, and language encoding result to obtain an acoustic spectrum. The acoustic spectrum is used as an acoustic representation for speech synthesis input of the speech audio, so that a speech waveform can be synthesized based on the acoustic spectrum.

[0250] In this embodiment, the decoder 50 decodes the text encoding results, speaker characteristics, and language encoding results based on the speech residual information, thereby being able to control the style and rhythm of the speech audio synthesized based on the acoustic spectrum, making the synthesized speech audio more natural and expressive, and correspondingly improving the control ability of the synthesized speech audio.

[0251] The decoder 50 predicts the output acoustic spectrum from the encoded input sequence, and uses the context vector obtained by encoding to obtain the target output sequence through certain transformations.

[0252] In this embodiment, the acoustic spectrum is a mel-spectrogram. As the acoustic representation of the model, the mel-spectrogram can enhance the details of low-frequency information in the speech data, thereby improving the intelligibility of the synthesized speech. Furthermore, after the frequency axis undergoes a nonlinear transformation using the mel-scaling method, the spectral content can be represented with relatively fewer dimensions, which facilitates information compression. Furthermore, due to its phase invariance, the mel-spectrogram is smoother than a waveform representation, making it easier to train using the mean square error (MSE) loss.

[0253] The audio synthesis module 60 is configured to perform audio synthesis processing based on the acoustic spectrum to generate speech audio corresponding to the text information. Specifically, the audio synthesis module 60 may be a vocoder. The vocoder converts the acoustic spectrum into a speech waveform (i.e., speech audio). The vocoder may be a Wavenet, WaveRNN, ParallelWaveNet, WaveGlow, or the like.

[0254] In order to solve the above problem, an embodiment of the present invention further provides a device, which can implement the speech synthesis method provided by the embodiment of the present invention by loading the above speech synthesis method in the form of a program.

[0255] An optional hardware structure of the device provided by the embodiment of the present invention is as follows Figure 4 As shown, it includes: at least one processor 01, at least one communication interface 02, at least one memory 03 and at least one communication bus 04.

[0256] In this embodiment, the number of the processor 01 , the communication interface 02 , the memory 03 , and the communication bus 04 is at least one, and the processor 01 , the communication interface 02 , and the memory 03 communicate with each other via the communication bus 04 .

[0257] Optionally, the communication interface 02 may be an interface of a communication module for network communication, such as an interface of a GSM module.

[0258] Optionally, the processor 01 may be a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention.

[0259] Optionally, the memory 03 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0260] The memory 03 stores one or more computer instructions, and the one or more computer instructions are executed by the processor 01 to implement the speech synthesis method provided by the embodiment of the present invention.

[0261] It should be noted that the above-mentioned terminal device may also include other devices (not shown) that may not be necessary for understanding the contents disclosed in the embodiments of the present invention; given that these other devices may not be necessary for understanding the contents disclosed in the embodiments of the present invention, the embodiments of the present invention will not introduce them one by one.

[0262] Correspondingly, an embodiment of the present invention further provides a storage medium, wherein the storage medium stores one or more computer instructions, and the one or more computer instructions are used to implement the speech synthesis method described in the embodiment of the present invention.

[0263] The storage medium is a computer-readable storage medium, and the storage medium can be various media that can store program codes, such as read-only memory (ROM), random access memory (RAM), USB flash drive, mobile hard disk, magnetic disk or optical disk.

[0264] The embodiments of the present invention described above are combinations of elements and features of the present invention. Unless otherwise mentioned, the elements or features may be considered as optional. Each element or feature may be put into practice without being combined with other elements or features. In addition, the embodiments of the present invention may be constructed by combining some elements and / or features. The order of operations described in the embodiments of the present invention may be rearranged. Some configurations of any one embodiment may be included in another embodiment and may be replaced by the corresponding configuration of another embodiment. It is obvious to those skilled in the art that claims that do not have a clear reference relationship to each other in the appended claims may be combined into embodiments of the present invention, or may be included as new claims in amendments after submitting this application.

[0265] The embodiments of the present invention may be implemented by various means such as hardware, firmware, software, or a combination thereof. In a hardware configuration, the method according to the exemplary embodiment of the present invention may be implemented by one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, etc.

[0266] In a firmware or software configuration, the embodiments of the present invention may be implemented in the form of modules, procedures, functions, and the like. Software codes may be stored in a memory unit and executed by a processor. The memory unit may be located inside or outside the processor and may send and receive data to and from the processor via various known means.

[0267] Although the present invention is disclosed as above, the present invention is not limited thereto. Any person skilled in the art can make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, the scope of protection of the present invention should be based on the scope defined by the claims.

Claims

1. A speech synthesis method, characterized in that: include: Providing training corpus, wherein the training corpus includes speaker information, language information, and text information, wherein the speaker information includes any one or more of the speaker's timbre information, age information, and gender information; Performing feature extraction processing on the speaker information to obtain speaker features; Performing language coding processing on the language information to obtain a language coding result; Based on the updated parameters after decoupling the speaker information and the language information, performing text encoding processing on the text information to obtain a text encoding result; Decoding the text encoding result, speaker characteristics, and language encoding result to obtain an acoustic spectrum; Performing audio synthesis processing based on the acoustic spectrum to generate speech audio corresponding to the text information; The step of encoding the text information based on the updated parameters after decoupling the speaker information and the language information comprises: inputting the text information into a text encoder for encoding; and minimizing the mutual information between the text encoder and the language information. The speech synthesis method further includes: obtaining speech residual information of a training corpus based on the acoustic spectrum; wherein the step of decoding the encoding result, speaker characteristics, and language encoding result includes: decoding the text encoding result, speaker characteristics, and language encoding result based on the speech residual information; The method of obtaining speech residual information of a training corpus based on the acoustic spectrum includes: inputting the acoustic spectrum into a residual coding module for residual coding processing to obtain the speech residual information; providing a second language classifier, and performing a third decoupling process on the residual coding module and the second language classifier.

2. The speech synthesis method according to claim 1, wherein: Based on the parameters updated after decoupling the speaker information and the language information, performing text encoding processing on the text information further includes: performing a second decoupling process between the text encoder and the speaker information.

3. The speech synthesis method according to claim 2, wherein: Performing a first decoupling process on the text encoder and the language information includes: providing a first language classifier; and performing adversarial training on the text encoder and the first language classifier.

4. The speech synthesis method according to claim 3, wherein: The adversarial training of the text encoder and the first language classifier includes: based on the text encoding result and language information, using a loss function to update the parameters of the first language classifier, and transferring the inverted gradient to the text encoder to perform parameter update processing on the text encoder.

5. The speech synthesis method according to claim 2, wherein: The second decoupling process between the text encoder and the speaker information includes: providing a first speaker classifier; and performing adversarial training on the text encoder and the first speaker classifier.

6. The speech synthesis method according to claim 5, wherein: The adversarial training between the text encoder and the first speaker classifier includes: based on the text encoding result and speaker information, using the loss function to update the parameters of the first speaker classifier, and transferring the inverted gradient to the text encoder to perform parameter update processing on the text encoder.

7. The speech synthesis method according to claim 4 or 6, wherein: The loss function is a cross entropy loss function or a minimum mutual information loss function.

8. The speech synthesis method according to claim 2, wherein: The second decoupling process between the text encoder and the speaker information includes: minimizing the mutual information between the text encoder and the speaker information.

9. The speech synthesis method according to claim 1, wherein: The acoustic spectrum is input into a residual coding module for residual coding processing to obtain the speech residual information; the speech synthesis method further includes: providing a second speaker classifier; and performing a fourth decoupling process on the residual coding module and the second speaker classifier.

10. The speech synthesis method according to claim 1 or 9, wherein: The residual coding module includes a variational autoencoder.

11. The speech synthesis method according to claim 1, wherein: The speech residual information includes one or more of speech speed, pitch, tone, volume, stress, rhythm and emotion.

12. The speech synthesis method according to claim 1, wherein: Performing feature extraction processing on the speaker information includes: inputting the speaker information into a speaker feature extractor for feature extraction processing; Performing language encoding processing on the language information includes: inputting the language information into a language encoder for language encoding processing; The speech synthesis method further includes: performing a fifth decoupling process on the speaker feature extractor and the language encoder.

13. The speech synthesis method according to claim 12, wherein: Performing a fifth decoupling process on the speaker feature extractor and the language encoder includes: performing a minimization process on the mutual information between the speaker feature extractor and the language encoder.

14. A speech synthesis device, characterized in that: include: A training corpus providing module is used to provide training corpus, wherein the training corpus includes speaker information, language information, and text information, wherein the speaker information includes any one or more of the speaker's timbre information, age information, and gender information; A speaker feature extractor, configured to perform feature extraction processing on the speaker information to obtain speaker features; A language encoder, configured to perform language encoding processing on the language information to obtain a language encoding result; A text encoder, configured to perform text encoding processing on the text information based on the updated parameters after decoupling the speaker information and the language information, to obtain a text encoding result; A decoder, configured to decode the encoding result, speaker characteristics, and language encoding result to obtain an acoustic spectrum; An audio synthesis module, configured to perform audio synthesis processing based on the acoustic spectrum to generate speech audio corresponding to the text information; The text encoder is configured to: input the text information into the text encoder for text encoding; minimize the mutual information between the text encoder and the language information; The device is further configured to execute: obtaining speech residual information of a training corpus based on the acoustic spectrum; wherein the step of decoding the encoding result, speaker characteristics, and the language encoding result includes: decoding the text encoding result, speaker characteristics, and language encoding result based on the speech residual information; wherein obtaining speech residual information of a training corpus based on the acoustic spectrum includes: inputting the acoustic spectrum into a residual coding module for residual coding to obtain the speech residual information; providing a second language classifier, and performing a third decoupling process on the residual coding module and the second language classifier.

15. A device, characterized in that The method comprises at least one memory and at least one processor, wherein the memory stores one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement the speech synthesis method according to any one of claims 1 to 13.

16. A storage medium, characterized in that The storage medium stores one or more computer instructions, and the one or more computer instructions are used to implement the speech synthesis method according to any one of claims 1 to 13.

Images