A speech synthesis method based on variational autoencoder

Abstract

The application provides a speech synthesis method based on a variational autoencoder, and relates to the technical field of natural language processing. The method comprises the following steps: obtaining character identifiers corresponding to respective characters in a target text, phonemes included in the target text, and speech duration corresponding to the target text; inputting the character identifiers, the phonemes, and the speech duration into a prior encoding module of a speech synthesis model to obtain prior latent variables corresponding to the target text, wherein the speech synthesis model is a model obtained by training a variational autoencoder; mapping the prior latent variables according to a preset mapping relationship to obtain prior latent variable mapping results, and inputting the prior latent variable mapping results into a decoding module of the speech synthesis model to obtain voiceprint data corresponding to the target text; resampling the voiceprint data to obtain voiceprint features corresponding to the voiceprint data, and generating speech data corresponding to the target text according to the voiceprint features.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of natural language processing, and particularly relates to a speech synthesis method based on a variational autoencoder. BACKGROUND

[0002] With the development of artificial intelligence technology, the speech synthesis technology is also developing faster and faster. In the prior art, the main way of speech synthesis is: first, converting text into linear spectrum corresponding to the text; second, converting the linear spectrum into voiceprint data; and finally, generating speech by resampling the voiceprint.

[0003] However, in the above synthesis method, since the voiceprint data for generating speech by resampling is obtained based on the original linear spectrum, the timbre of the generated speech is consistent with the timbre of the input speech of the text, which may cause a sound copyright problem. SUMMARY

[0004] In order to solve the problem that the timbre of the synthesized speech is consistent with the timbre of the input speech of the text when synthesizing speech based on the prior method, the present application provides a speech synthesis method based on a variational autoencoder, a device, an electronic equipment and a storage medium.

[0005] In a first aspect, the present application provides a speech synthesis method based on a variational autoencoder, comprising:

[0006] obtaining character identifiers corresponding to respective characters in a target text, phonemes included in the target text, and speech duration corresponding to the target text;

[0007] inputting the character identifiers, the phonemes, and the speech duration into a prior encoding module of a speech synthesis model to obtain a prior latent variable corresponding to the target text, the speech synthesis model being a model obtained by training a variational autoencoder;

[0008] mapping the prior latent variable according to a preset mapping relationship to obtain a prior latent variable mapping result, and inputting the prior latent variable mapping result into a decoding module of the speech synthesis model to obtain voiceprint data corresponding to the target text;

[0009] resampling the voiceprint data to obtain voiceprint features corresponding to the voiceprint data, and generating speech data corresponding to the target text according to the voiceprint features.

[0010] As an optional implementation manner of the embodiment of the present application, before inputting the character identifiers, the phonemes, and the speech duration into the prior encoding module of the speech synthesis model to obtain the prior latent variable mapping result, the method further comprises:

[0011] obtaining a training data set, the training data set comprising training speech data, a speech duration corresponding to the training speech data, training text corresponding to the training speech data, and phonemes corresponding to the training speech data;

[0012] training the variational autoencoder based on the training data set to obtain the speech synthesis model.

[0013] As an optional implementation of the embodiment of the present application, the training of the variational autoencoder based on the training data set to obtain the speech synthesis model comprises:

[0014] obtaining a linear spectrum corresponding to the training speech data, and inputting the linear spectrum into a posterior encoding module of the variational autoencoder to obtain a posterior latent variable;

[0015] inputting the speech duration corresponding to the training speech data, the training text corresponding to the training speech data, and the phonemes corresponding to the training speech data into a prior encoding module of the variational autoencoder to obtain a prior latent variable corresponding to the training text;

[0016] adjusting parameters of the variational autoencoder based on the posterior latent variable and the prior latent variable corresponding to the training text until a KL divergence is less than a preset threshold to obtain the speech synthesis model.

[0017] As an optional implementation of the embodiment of the present application, the adjusting of the parameters of the variational autoencoder based on the posterior latent variable and the prior latent variable corresponding to the training text until the KL divergence is less than the preset threshold to obtain the speech synthesis model comprises:

[0018] performing reversible transformation on the posterior latent variable to obtain a posterior latent variable reversible transformation result;

[0019] performing mapping processing on the prior latent variable corresponding to the training text to obtain a prior latent variable mapping result;

[0020] calculating a KL divergence of the posterior latent variable reversible transformation result and the prior latent variable mapping result;

[0021] if the KL divergence is less than the preset threshold, determining that the variational autoencoder is the speech synthesis model.

[0022] As an optional implementation of the embodiment of the present application, the method further comprises:

[0023] inputting the posterior latent variable into a decoding module of the variational autoencoder to obtain a voiceprint feature corresponding to the training speech data;

[0024] Input the voiceprint feature corresponding to the training voice data into a generator of a generative adversarial network to generate a voice signal;

[0025] Input the voice signal into a discriminator of the generative adversarial network, and optimize a decoding module of the variational autoencoder based on a discrimination result output by the discriminator.

[0026] As an optional implementation of the embodiment of the present application, the resampling of the voiceprint data to obtain the voiceprint feature corresponding to the voiceprint data, and the generation of the voice data corresponding to the target text according to the voiceprint feature, comprises:

[0027] Input the prior latent variable corresponding to the target text and the voice duration corresponding to the target text into a duration prediction module of the voice synthesis model to obtain a target voice duration;

[0028] Generate the voice data corresponding to the target text according to the voiceprint feature and the target voice duration.

[0029] As an optional implementation of the embodiment of the present application, the obtaining of the character identifier corresponding to each character in the target text comprises:

[0030] Obtain a target text, and determine whether the target text is a mixed text of Chinese characters and English characters;

[0031] If yes, separate the English characters from the Chinese characters to form an English character sequence and a Chinese character sequence;

[0032] Convert all characters in the English character sequence and the Chinese character sequence into the character identifier corresponding to each character based on a mapping dictionary.

[0033] In a second aspect, the present application provides a voice synthesis device based on a variational autoencoder, comprising:

[0034] An obtaining module is configured to obtain a character identifier corresponding to each character in a target text, phonemes included in the target text, and a voice duration corresponding to the target text;

[0035] An input module is configured to input the character identifier, the phonemes, and the voice duration into a prior encoding module of a voice synthesis model to obtain a prior latent variable corresponding to the target text, the voice synthesis model being a model obtained by training a variational autoencoder;

[0036] A processing module is configured to map the prior latent variable according to a preset mapping relationship to obtain a prior latent variable mapping result, and input the prior latent variable mapping result into a decoding module of the voice synthesis model to obtain voiceprint data corresponding to the target text.

[0037] The generating module is configured to resample the voiceprint data to obtain voiceprint features corresponding to the voiceprint data, and generate speech data corresponding to the target text according to the voiceprint features.

[0038] As an optional implementation of the embodiment of the present application, the device further comprises:

[0039] The training module is configured to obtain a training data set, the training data set comprising training speech data, a speech duration corresponding to the training speech data, training text corresponding to the training speech data, and phonemes corresponding to the training speech data.

[0040] The variational autoencoder is trained based on the training data set to obtain the speech synthesis model.

[0041] As an optional implementation of the embodiment of the present application, the training module is specifically configured to obtain a linear spectrum corresponding to the training speech data, and input the linear spectrum into a posterior encoding module of the variational autoencoder to obtain a posterior latent variable.

[0042] The speech duration corresponding to the training speech data, the training text corresponding to the training speech data, and the phonemes corresponding to the training speech data are input into a prior encoding module of the variational autoencoder to obtain a prior latent variable corresponding to the training text.

[0043] The parameters of the variational autoencoder are adjusted based on the posterior latent variable and the prior latent variable corresponding to the training text until the KL divergence is less than a preset threshold value, and the speech synthesis model is obtained.

[0044] As an optional implementation of the embodiment of the present application, the training module is specifically configured to perform reversible transformation on the posterior latent variable to obtain a posterior latent variable reversible transformation result.

[0045] The prior latent variable corresponding to the training text is mapped to obtain a prior latent variable mapping result.

[0046] The KL divergence between the posterior latent variable reversible transformation result and the prior latent variable mapping result is calculated.

[0047] If the KL divergence is less than a preset threshold value, the variational autoencoder is determined to be the speech synthesis model.

[0048] As an optional implementation of the embodiment of the present application, the device further comprises:

[0049] The optimization module is configured to input the posterior latent variable into a decoding module of the variational autoencoder to obtain voiceprint features corresponding to the training speech data.

[0050] input the voiceprint features corresponding to the training voice data into a generator of a generative adversarial network to generate a voice signal;

[0051] input the voice signal into a discriminator of the generative adversarial network, and optimize a decoding module of the variational autoencoder based on a discrimination result output by the discriminator.

[0052] As an optional implementation of the embodiment of the present application, the generation module is specifically configured to input the prior latent variable corresponding to the target text and the voice duration corresponding to the target text into a duration prediction module of the voice synthesis model to obtain a target voice duration.

[0053] generate voice data corresponding to the target text according to the voiceprint features and the target voice duration.

[0054] As an optional implementation of the embodiment of the present application, the obtaining module is specifically configured to obtain a target text, and determine whether the target text is a mixed text of Chinese characters and English characters.

[0055] If yes, separate the English characters from the Chinese characters to form an English character sequence and a Chinese character sequence.

[0056] convert all characters in the English character sequence and the Chinese character sequence into character identifiers corresponding to the respective characters based on a mapping dictionary.

[0057] In a third aspect, the embodiment of the present application provides an electronic device, comprising a memory and a processor, the memory is configured to store a computer program, and the processor is configured to execute the variational autoencoder-based voice synthesis method of the first aspect or any optional implementation of the first aspect when the computer program is invoked.

[0058] In a fourth aspect, the embodiment of the present application provides a computer readable storage medium having a computer program stored thereon, and the computer program is executed by a processor to implement the variational autoencoder-based voice synthesis method of the first aspect or any optional implementation of the first aspect.

[0059] The technical solution provided by the embodiment of the present application has the following advantages compared with the prior art:

[0060] The embodiment of the present application provides a speech synthesis method and device based on a variational autoencoder, electronic equipment and a storage medium, and the method comprises the following steps: obtaining character identifiers respectively corresponding to characters in a target text, phonemes included in the target text, and speech duration corresponding to the target text; inputting the character identifiers, the phonemes and the speech duration into a prior encoding module of a speech synthesis model to obtain prior hidden variables corresponding to the target text, wherein the speech synthesis model is a model obtained by training a variational autoencoder; mapping the prior hidden variables according to a preset mapping relationship to obtain a prior hidden variable mapping result, and inputting the prior hidden variable mapping result into a decoding module of the speech synthesis model to obtain voiceprint data corresponding to the target text; resampling the voiceprint data to obtain voiceprint features corresponding to the voiceprint data, and generating speech data corresponding to the target text according to the voiceprint features. In the embodiment of the present application, the voiceprint features used for generating the speech data corresponding to the target text are obtained by processing the prior hidden variable mapping result by the decoding module of the speech synthesis model, and the prior hidden variables are obtained according to the character identifiers respectively corresponding to the characters in the target text, the phonemes and the speech duration corresponding to the target text, and are not obtained based on the original input speech of the target text. Therefore, the timbre of the generated speech data is different from the timbre of the original input speech of the target text, so that the problem of sound copyright is avoided. BRIEF DESCRIPTION OF DRAWINGS

[0061] The accompanying drawings, which are incorporated herein and constitute part of the specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0062] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the accompanying drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, for those skilled in the art, other drawings can also be obtained based on these drawings without creative labor.

[0063] Figure 1 The step flow chart of the speech synthesis method based on the variational autoencoder provided by an embodiment of the present application is shown in the figure.

[0064] Figure 2 The step flow chart of the speech synthesis method based on the variational autoencoder provided by another embodiment of the present application is shown in the figure.

[0065] Figure 3 The structural schematic diagram of the speech device based on the variational autoencoder provided by an embodiment of the present application is shown in the figure.

[0066] Figure 4A structural schematic diagram of a speech device based on a variational autoencoder is provided for another embodiment of the present application.

[0067] Figure 5 An internal structural diagram of an electronic device is provided for an embodiment of the present application. DETAILED DESCRIPTION

[0068] For the purposes of making the purposes, embodiments and advantages of the present application clearer, the following will combine the drawings in the exemplary embodiments of the present application to clearly and completely describe the exemplary embodiments of the present application. Obviously, the described exemplary embodiments are only some of the embodiments of the present application, but not all the embodiments.

[0069] Based on the exemplary embodiments described in the present application, all other embodiments obtained by those of ordinary skill in the art without making creative efforts fall within the scope of the claims of the present application. In addition, although the disclosure in the present application is introduced according to one or several examples, it should be understood that each aspect of these disclosures can also constitute a complete embodiment. It should be noted that the brief description of the terms in the present application is only for the convenience of understanding the subsequently described embodiments, and is not intended to limit the embodiments of the present application. Unless otherwise specified, these terms should be understood according to their ordinary and general meanings.

[0070] The speech synthesis method based on the variational autoencoder provided by the embodiments of the present application can be executed by a speech synthesis device based on the variational autoencoder or an electronic device provided by the embodiments of the present application. The electronic device can be a terminal device or other types of electronic devices, and the specific type of the electronic device is not limited in the present application.

[0071] The speech synthesis method based on the variational autoencoder provided by the present application is exemplarily described through several specific embodiments. In order to make the above purposes, features and advantages of the present application more obvious and easy to understand, the following optional embodiments are specifically described below, and the accompanying drawings are described in detail.

[0072] Figure 1 A flowchart of the speech synthesis method based on the variational autoencoder is provided for an embodiment of the present application, as shown in Figure 1 The speech synthesis method based on the variational autoencoder provided by the present embodiment includes the following steps:

[0073] S110, obtaining character identifiers corresponding to each character in a target text, phonemes included in the target text, and speech duration corresponding to the target text.

[0074] The target text can be text obtained based on input voice or directly input text. When the target text is text obtained based on input voice, the voice duration corresponding to the target text is the duration of the input voice. When the target text is directly input text, audio data corresponding to the target text is first obtained, and then the voice duration is determined.

[0075] In this embodiment, the target text can be all Chinese text, all English text, or mixed Chinese and English text.

[0076] For example, obtaining character identifiers corresponding to respective characters in the target text can be achieved by the following method: obtaining the target text, determining whether the target text is mixed Chinese and English text, if so, separating the English characters from the Chinese characters to form an English character sequence and a Chinese character sequence, and converting all characters in the English character sequence and the Chinese character sequence into respective character identifiers based on a mapping dictionary.

[0077] After obtaining the target text, characters that are neither Chinese nor English in the target text are first filtered out, and punctuation marks in the target text are preprocessed to normalize the punctuation marks, that is, only target punctuation marks are retained, and other punctuation marks are replaced with the target punctuation marks. For example, the target punctuation marks can be punctuation marks that require prosodic breaks in the voice corresponding to the target text, such as one or more of a comma, a period, and a semicolon. Punctuation marks that do not require prosodic breaks, such as “#”, are converted into one of the target punctuation marks. The specific conversion into which target punctuation mark can be converted according to a specific correspondence, for example, “#” is converted into a comma. Here, specific examples are not given one by one.

[0078] Converting all characters in the English character sequence and the Chinese character sequence into respective character identifiers based on a mapping dictionary includes: obtaining a phoneme identifier corresponding to each character based on a mapping dictionary, wherein the mapping dictionary includes a correspondence between a phoneme and a phoneme identifier; and constructing a character identifier corresponding to the character based on the phoneme identifier corresponding to the character.

[0079] Exemplarily, each English character in an English character sequence is converted into a phonetic symbol corresponding to the English character, each phoneme included in the phonetic symbol is converted into a phoneme identifier corresponding to the phoneme based on a mapping dictionary, and one or more phoneme identifiers corresponding to one English character constitute a character identifier corresponding to the English character; each Chinese character in a Chinese character sequence is converted into a corresponding pinyin and a corresponding number corresponding to a tone, for example, the character "zhong" is converted into "zhong1", the character "he" is converted into "he2", the character "ma" is converted into "ma3", and the character "shi" is converted into "shi4", and so on, each phoneme included in the pinyin is converted into a phoneme identifier corresponding to the phoneme based on a mapping dictionary, and one or more phoneme identifiers included in the pinyin of one Chinese character and the number corresponding to the tone of the Chinese character constitute a character identifier corresponding to the Chinese character.

[0080] S120, input the character identifier, the phoneme, and the speech duration into a prior encoding module of a speech synthesis model to obtain a prior latent variable corresponding to the target text.

[0081] The speech synthesis model is a model obtained by training a variational autoencoder, and the prior encoding module is an encoder composed of a transformer.

[0082] The prior latent variable can be understood as text feature data corresponding to the target text.

[0083] The prior latent variable corresponding to the target text and the speech duration corresponding to the target text can be input into a duration prediction module of the speech synthesis model to obtain a target speech duration, which is used to control the duration of the speech data corresponding to the target text and indirectly control the speech speed of the output speech data corresponding to the target text.

[0084] It should be noted that in the case of a target text being a mixed text of Chinese characters and English characters, the order of the character identifiers input into the speech synthesis model is consistent with the order of the characters in the target text. That is, before the character identifiers are input into the speech synthesis model, the character identifiers corresponding to the English characters and the character identifiers corresponding to the Chinese characters need to be sorted and arranged according to the order of the characters in the target text. Exemplarily, the third character in the target text is "my" and the fifth character is "and", so in the sequence of character identifiers input into the speech synthesis model, the third character identifier is the character identifier corresponding to "my", and the fifth character identifier is the character identifier corresponding to "and".

[0085] S130, according to a preset mapping relationship, the prior latent variable is mapped to obtain a prior latent variable mapping result, and the prior latent variable mapping result is input into a decoding module of the speech synthesis model to obtain voiceprint data corresponding to the target text.

[0086] The mapping of the prior latent variable according to the preset mapping relationship can be performed according to a latent variable in a TTS (Text to Speech) -Flow standard stream.

[0087] The decoding module of the speech synthesis model is a decoder optimized by using a GAN model (Generative Adversarial Network), so that the speech signal output by the optimized decoder is almost the same as the speech signal input to the posterior encoding module.

[0088] S140, resampling the voiceprint data to obtain voiceprint features corresponding to the voiceprint data, and generating speech data corresponding to the target text according to the voiceprint features.

[0089] Exemplarily, the prior latent variable corresponding to the target text and the target speech duration are input into a duration prediction module of the speech synthesis model to obtain a target speech duration; and the speech data corresponding to the target text is generated according to the voiceprint features and the target speech duration.

[0090] Exemplarily, a mel-frequency spectrum is generated according to the voiceprint features and the target speech duration, and the speech data corresponding to the target text is generated according to the mel-frequency spectrum.

[0091] After the speech data corresponding to the target text is generated, it is output in the form of speech.

[0092] The speech synthesis method based on the variational autoencoder provided in the embodiments of the present application comprises: obtaining character identifiers respectively corresponding to characters in a target text, phonemes included in the target text, and speech duration corresponding to the target text; inputting the character identifiers, the phonemes, and the speech duration into a prior encoding module of a speech synthesis model to obtain prior hidden variables corresponding to the target text, the speech synthesis model being a model obtained by training a variational autoencoder; mapping the prior hidden variables according to a preset mapping relationship to obtain prior hidden variable mapping results, and inputting the prior hidden variable mapping results into a decoding module of the speech synthesis model to obtain voiceprint data corresponding to the target text; resampling the voiceprint data to obtain voiceprint features corresponding to the voiceprint data, and generating speech data corresponding to the target text according to the voiceprint features. In the embodiments of the present application, the voiceprint features used to generate the speech data corresponding to the target text are obtained by processing the prior hidden variable mapping results by the decoding module of the speech synthesis model, and the prior hidden variables are obtained according to the character identifiers respectively corresponding to the characters in the target text, the phonemes, and the speech duration corresponding to the target text, and are not obtained based on the input speech of the target text originally, so that the timbre of the generated speech data is different from the timbre of the input speech of the target text originally, thereby avoiding the problem of sound copyright.

[0093] In step S110, before the character identifiers, the phonemes, and the speech duration are input into the prior encoding module of the speech synthesis model to obtain the prior hidden variable mapping results, the variational autoencoder needs to be trained to obtain the speech synthesis model, and the model training process can comprise steps S210 to S220 as shown in Figure 2 The same or similar steps in the embodiments shown in Figure 2 will not be described in detail, and the specific description and explanation can be referred to the description and explanation in the embodiments shown in Figure 1 .

[0094] In S210, a training data set is obtained, and the training data set comprises training speech data, speech duration corresponding to the training speech data, training text corresponding to the training speech data, and phonemes corresponding to the training speech data.

[0095] The training speech data is the audio duration corresponding to the training text, and the training data set comprises a plurality of training data. One piece of training data can be a sentence, speech corresponding to the sentence, audio duration of the speech, and phonemes included in the sentence. Alternatively, one piece of training data can be a paragraph, speech corresponding to the paragraph, audio duration of the speech, and phonemes included in the paragraph.

[0096] The training text in the training data set can comprise Chinese character text and English character text.

[0097] S220, training the variational autoencoder based on the training data set to obtain the speech synthesis model.

[0098] Exemplarily, the variational autoencoder can be trained in the following manner:

[0099] The linear spectrum corresponding to the training speech data is obtained, and the linear spectrum is input into the posterior encoding module of the variational autoencoder to obtain the posterior latent variable; the speech duration corresponding to the training speech data, the training text corresponding to the training speech data, and the phoneme corresponding to the training speech data are input into the prior encoding module of the variational autoencoder to obtain the prior latent variable corresponding to the training text; the parameters of the variational autoencoder are adjusted based on the posterior latent variable and the prior latent variable corresponding to the training text until the KL divergence is less than a preset threshold, and the speech synthesis model is obtained.

[0100] The posterior encoding module is an encoder composed of a residual network wavnet, and the posterior latent variable is the spectral feature data of the linear spectrum of the training speech data.

[0101] Exemplarily, the variational autoencoder can be trained in the following manner:

[0102] The posterior latent variable is reversibly transformed to obtain a posterior latent variable reversible transformation result; the prior latent variable corresponding to the training text is mapped to obtain a prior latent variable mapping result; the KL divergence of the posterior latent variable reversible transformation result and the prior latent variable mapping result is calculated; if the KL divergence is less than a preset threshold, the variational autoencoder is determined to be the speech synthesis model.

[0103] In this embodiment, the posterior latent variable obtained by training the speech data (audio of the training text) is the spectral feature data of the linear spectrum of the training speech data, and the posterior latent variable is reversibly transformed to obtain a posterior latent variable reversible transformation result; the prior latent variable obtained by training the text is the text feature data, and the prior latent variable is mapped to obtain a prior latent variable mapping result. Since the KL divergence of the posterior latent variable reversible transformation result and the prior latent variable mapping result is less than a preset threshold, the posterior latent variable reversible transformation result tends to approach the prior latent variable mapping result, but they are not equal, so that the voiceprint data output by the decoder of the speech synthesis model is inconsistent with the voiceprint data input into the posterior encoding module, thereby ensuring that the timbre of the speech data generated based on the speech synthesis model is different from the timbre of the original input speech of the target text.

[0104] In the model training process, the decoding module of the variational autoencoder needs to be optimized. For example, the decoding module of the variational autoencoder can be optimized by a GAN model: the posterior latent variable is input into the decoding module of the variational autoencoder to obtain the voiceprint feature corresponding to the training speech data; the voiceprint feature corresponding to the training speech data is input into the generator of the generative adversarial network to generate a speech signal; the speech signal is input into the discriminator of the generative adversarial network, and the decoding module of the variational autoencoder is optimized based on the discrimination result output by the discriminator.

[0105] The decoding module is optimized to ensure that the speech signal output by the optimized decoder maintains the timbre of the speech signal input into the posterior encoding module.

[0106] Based on the same inventive concept, as an implementation of the above method, the embodiments of the present application also provide a speech synthesis device based on a variational autoencoder, which can perform the speech synthesis method based on the variational autoencoder corresponding to the above method embodiments. For the sake of reading, the details of the above method embodiments will not be described one by one, but it should be clear that the speech synthesis device based on the variational autoencoder in the present embodiment can correspondingly implement all the contents in the above method embodiments.

[0107] Figure 3 The structure diagram of the speech synthesis device based on the variational autoencoder provided by the embodiments of the present application is shown in FIG. 3. Figure 3 As shown in FIG. 3, the speech synthesis device 300 based on the variational autoencoder provided by the present embodiment comprises:

[0108] The obtaining module 310 is configured to obtain character identifiers corresponding to respective characters in a target text, phonemes included in the target text, and speech duration corresponding to the target text.

[0109] The input module 320 is configured to input the character identifiers, the phonemes, and the speech duration into a prior encoding module of a speech synthesis model to obtain a prior latent variable corresponding to the target text, the speech synthesis model being a model obtained by training a variational autoencoder.

[0110] The processing module 330 is configured to map the prior latent variable according to a preset mapping relationship to obtain a prior latent variable mapping result, and input the prior latent variable mapping result into a decoding module of the speech synthesis model to obtain voiceprint data corresponding to the target text.

[0111] The generating module 340 is configured to resample the voiceprint data to obtain voiceprint features corresponding to the voiceprint data, and generate speech data corresponding to the target text according to the voiceprint features.

[0112] Referring toFigure 4 as shown, Figure 4 Another structure schematic diagram of a voice synthesis device based on a variational autoencoder provided by an embodiment of the present application is shown, which is based on the device shown in Figure 3 The device shown further comprises:

[0113] The training module 410 is configured to obtain a training data set, the training data set comprising training voice data, a voice duration corresponding to the training voice data, training text corresponding to the training voice data, and phonemes corresponding to the training voice data; and train the variational autoencoder based on the training data set to obtain the voice synthesis model.

[0114] As an optional implementation of the embodiment of the present application, the training module 410 is specifically configured to obtain a linear spectrum corresponding to the training voice data, and input the linear spectrum into a posterior encoding module of the variational autoencoder to obtain a posterior latent variable; input the voice duration corresponding to the training voice data, the training text corresponding to the training voice data, and the phonemes corresponding to the training voice data into a prior encoding module of the variational autoencoder to obtain a prior latent variable corresponding to the training text; and adjust parameters of the variational autoencoder based on the posterior latent variable and the prior latent variable corresponding to the training text until a KL divergence is less than a preset threshold value, to obtain the voice synthesis model.

[0115] As an optional implementation of the embodiment of the present application, the training module 410 is specifically configured to perform reversible transformation on the posterior latent variable to obtain a posterior latent variable reversible transformation result; perform mapping processing on the prior latent variable corresponding to the training text to obtain a prior latent variable mapping result; calculate a KL divergence of the posterior latent variable reversible transformation result and the prior latent variable mapping result; and if the KL divergence is less than a preset threshold value, determine that the variational autoencoder is the voice synthesis model.

[0116] As an optional implementation of the embodiment of the present application, the device further comprises:

[0117] The optimization module 420 is configured to input the posterior latent variable into a decoding module of the variational autoencoder to obtain a voiceprint feature corresponding to the training voice data; input the voiceprint feature corresponding to the training voice data into a generator of a generative adversarial network to generate a voice signal; input the voice signal into a discriminator of the generative adversarial network; and optimize the decoding module of the variational autoencoder based on a discrimination result output by the discriminator.

[0118] As an optional implementation of the embodiment of the present application, the generation module 340 is specifically configured to input the prior latent variable corresponding to the target text and the speech duration corresponding to the target text into a duration prediction module of the speech synthesis model, to obtain a target speech duration; and generate the speech data corresponding to the target text according to the voiceprint feature and the target speech duration.

[0119] As an optional implementation of the embodiment of the present application, the acquisition module 310 is specifically configured to determine whether the target text is a mixed text of Chinese characters and English characters; if yes, separate the English characters from the Chinese characters to form an English character sequence and a Chinese character sequence; and convert all characters in the English character sequence and the Chinese character sequence into character identifiers corresponding to the respective characters based on a mapping dictionary.

[0120] The speech synthesis apparatus based on the variational autoencoder provided in the embodiment can execute the speech synthesis method based on the variational autoencoder provided in the method embodiment, and has similar implementation principles and technical effects, which will not be described here again. Each module in the speech synthesis apparatus based on the variational autoencoder can be realized by software, hardware, or a combination thereof, in whole or in part. Each module can be embedded in or independent of a processor in a computer device in a hardware form, or can be stored in a memory in a computer device in a software form, so as to be called and executed by a processor to perform the operations corresponding to each module.

[0121] In one embodiment, an electronic device is provided, including a memory and a processor, the memory stores a computer program, and the processor executes the computer program to implement any one of the steps of the speech synthesis based on the variational autoencoder described in the method embodiment.

[0122] An exemplary, Figure 5 The structural schematic diagram of the electronic device provided in the embodiment of the present application is shown in FIG. 51. Figure 5 As shown in FIG. 51, the electronic device provided in the embodiment of the present application includes a memory 51 and a processor 52, the memory 51 is configured to store a computer program, and the processor 52 is configured to execute the steps of the speech synthesis method based on the variational autoencoder provided in the method embodiment when the computer program is called. The implementation principles and technical effects are similar, which will not be described here again. Those skilled in the art can understand that, Figure 5 The structure shown in FIG. 51 is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the electronic device to which the scheme of the present application is applied. The specific electronic device can include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement.

[0123] In an embodiment, a computer readable storage medium is provided, having stored thereon a computer program, which, when executed by a processor, implements the steps of any of the speech synthesis methods based on variational autoencoder described in the above method embodiments.

[0124] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer readable storage medium, and when executed, can include the processes of the above-mentioned embodiments. Any reference to a memory, database, or other medium used in the embodiments provided in the present application can include at least one of a non-volatile and volatile memory. The non-volatile memory can include a read-only memory (ROM), a magnetic tape, a floppy disk, a flash memory, or an optical memory. The volatile memory can include a random access memory (RAM) or an external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static random access memory (SRAM) and dynamic random access memory (DRAM).

[0125] It should be noted that, in this document, the terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. Moreover, the terms "include", "contain" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. Without more limitations, the element defined by the statement "including a" does not exclude the presence of additional identical elements in the process, method, article or device including the element.

[0126] The above description is merely one specific implementation of the present application, enabling a person skilled in the art to understand or implement the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to these embodiments described herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A speech synthesis method based on variational autoencoder, characterized in that: include: Obtaining character identifiers corresponding to respective characters in a target text, phonemes included in the target text, and speech duration corresponding to the target text; Inputting the character identifier, the phoneme, and the speech duration into a priori encoding module of a speech synthesis model to obtain a priori latent variables corresponding to the target text, wherein the speech synthesis model is a model obtained by training a variational autoencoder; Mapping the priori latent variables according to a preset mapping relationship to obtain a priori latent variable mapping result, and inputting the priori latent variable mapping result into a decoding module of the speech synthesis model to obtain voiceprint data corresponding to the target text; Resampling the voiceprint data to obtain voiceprint features corresponding to the voiceprint data, and generating speech data corresponding to the target text based on the voiceprint features; The training process of the speech synthesis model includes: Obtaining a linear spectrum corresponding to the training speech data, and inputting the linear spectrum into the posterior encoding module of the variational autoencoder to obtain a posterior latent variable; Inputting the speech duration corresponding to the training speech data, the training text corresponding to the training speech data, and the phonemes corresponding to the training speech data into the prior encoding module of the variational autoencoder to obtain the prior latent variables corresponding to the training text; Adjusting the parameters of the variational autoencoder based on the posterior latent variables and the prior latent variables corresponding to the training text until the KL divergence is less than a preset threshold, thereby obtaining the speech synthesis model; The method of adjusting the parameters of the variational autoencoder based on the posterior latent variables and the prior latent variables corresponding to the training text until the KL divergence is less than a preset threshold to obtain the speech synthesis model includes: Performing a reversible transformation on the posterior latent variable to obtain a reversible transformation result of the posterior latent variable; Performing mapping processing on the prior latent variables corresponding to the training text to obtain a priori latent variable mapping result; Calculating the KL divergence of the reversible transformation result of the posterior latent variable and the mapping result of the prior latent variable; If the KL divergence is less than a preset threshold, the variational autoencoder is determined to be the speech synthesis model.

2. The method according to claim 1, characterized in that Before inputting the character identifier, the phoneme, and the speech duration into a priori encoding module of a speech synthesis model to obtain a priori latent variable mapping result, the method further includes: Acquire a training data set, the training data set including training speech data, speech duration corresponding to the training speech data, training text corresponding to the training speech data, and phonemes corresponding to the training speech data; The variational autoencoder is trained based on the training data set to obtain the speech synthesis model.

3. The method according to claim 1, characterized in that The method further comprises: Inputting the posterior latent variable into the decoding module of the variational autoencoder to obtain the voiceprint features corresponding to the training speech data; Inputting the voiceprint features corresponding to the training voice data into a generator of a generative adversarial network to generate a voice signal; The speech signal is input into the discriminator of the generative adversarial network, and the decoding module of the variational autoencoder is optimized based on the discrimination result output by the discriminator.

4. The method according to claim 1, wherein The resampling of the voiceprint data to obtain voiceprint features corresponding to the voiceprint data, and generating speech data corresponding to the target text according to the voiceprint features, includes: Inputting the prior latent variable corresponding to the target text and the speech duration corresponding to the target text into the duration prediction module of the speech synthesis model to obtain the target speech duration; The voice data corresponding to the target text is generated according to the voiceprint feature and the target speech duration.

5. The method according to any one of claims 1 to 4, characterized in that The step of obtaining character identifiers corresponding to respective characters in the target text includes: Get the target text; Preprocessing the punctuation marks in the target text to obtain a normalized target text; Each character included in the normalized target text is obtained, and each character is converted into a character identifier corresponding to each character based on a mapping dictionary.

6. A speech synthesis device based on a variational autoencoder, characterized in that: include: An acquisition module, configured to acquire character identifiers corresponding to respective characters in a target text, phonemes included in the target text, and speech duration corresponding to the target text; An input module, configured to input the character identifier, the phoneme, and the speech duration into a priori encoding module of a speech synthesis model to obtain a priori latent variables corresponding to the target text, wherein the speech synthesis model is a model obtained by training a variational autoencoder; a processing module, configured to map the prior latent variables according to a preset mapping relationship to obtain a priori latent variable mapping result, and input the priori latent variable mapping result into a decoding module of the speech synthesis model to obtain voiceprint data corresponding to the target text; a generating module, configured to resample the voiceprint data to obtain voiceprint features corresponding to the voiceprint data, and generate speech data corresponding to the target text according to the voiceprint features; The device also includes a training module for obtaining a linear spectrum corresponding to the training speech data, and inputting the linear spectrum into the posterior encoding module of the variational autoencoder to obtain a posterior latent variable; Inputting the speech duration corresponding to the training speech data, the training text corresponding to the training speech data, and the phonemes corresponding to the training speech data into the prior encoding module of the variational autoencoder to obtain the prior latent variables corresponding to the training text; adjusting the parameters of the variational autoencoder based on the posterior latent variables and the prior latent variables corresponding to the training text until the KL divergence is less than a preset threshold, thereby obtaining the speech synthesis model; The training module is specifically used to perform a reversible transformation on the posterior latent variables to obtain a reversible transformation result of the posterior latent variables; and perform mapping processing on the priori latent variables corresponding to the training text to obtain a priori latent variable mapping result; Calculate the KL divergence of the reversible transformation result of the posterior latent variable and the mapping result of the prior latent variable; if the KL divergence is less than a preset threshold, determine that the variational autoencoder is the speech synthesis model.

7. An electronic device comprising: A memory and a processor, wherein the memory stores a computer program, and wherein the processor implements the speech synthesis method based on the variational autoencoder according to any one of claims 1 to 5 when executing the computer program.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the speech synthesis method based on the variational autoencoder according to any one of claims 1 to 5 is implemented.

Images