Speech synthesis method and device based on reference speech, and electronic equipment

Abstract

The invention discloses a reference voice-based voice synthesis method and device, and electronic equipment. The method comprises the steps of obtaining target text data and target reference voice data of a target speaker; and inputting the target text data and the target reference voice data into a pre-trained voice synthesis model, and outputting target synthesized voice data corresponding to the target speaker, the pre-trained voice synthesis model being obtained by performing adversarial training on an initial voice synthesis model based on an SLM discriminator, the SLM discriminator is used for discriminating the SLM features of the sample synthesis voice data and the SLM features of the sample adversarial voice data of the initial voice synthesis model, and the SLM features are generated by the SLM discriminator according to the sample synthesis voice data and the sample adversarial voice data. According to the invention, under the condition that the voice data of the specific duration of the target speaker cannot be obtained, it can be ensured that the synthesized voice data finally output by the voice synthesis model is consistent with the voice features of the target speaker.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a speech synthesis method, device and electronic device based on reference speech. Background Art

[0002] In recent years, with the advancement of artificial intelligence technology, text to speech (TTS) technology has also developed rapidly. Speech synthesis technology can convert text data into natural, fluent and expressive synthetic speech data.

[0003] At present, in the process of speech synthesis based on a pre-trained speech synthesis model, it is also necessary to further optimize the speech synthesis model based on speech data of a specific length of time (usually 30 minutes to 60 minutes) of the target speaker, so as to ensure that the speech synthesis model can learn the voice characteristics of the target speaker, and then ensure that the speech synthesis model ultimately outputs synthesized speech data close to the target speaker.

[0004] However, if the speech data of a specific length of the target speaker cannot be obtained, it will make it impossible to optimize the training of the pre-trained speech synthesis model, which will cause the synthesized speech data ultimately output by the speech synthesis model to deviate significantly from the sound characteristics of the target speaker. Summary of the invention

[0005] In order to solve the above problems, the embodiments of the present application provide a speech synthesis method, device and electronic device based on reference speech, which can reduce the deviation between the synthesized speech data finally output by the speech synthesis model and the sound characteristics of the target speaker when the speech data of a specific duration of the target speaker cannot be obtained. Specifically, the embodiments of the present application disclose the following technical solutions:

[0006] A first aspect of an embodiment of the present application provides a speech synthesis method based on a reference speech, the method comprising:

[0007] Obtain target text data and target reference speech data of a target speaker;

[0008] The target text data and the target reference speech data are input into a pre-trained speech synthesis model, and the target synthesized speech data corresponding to the target speaker is output, wherein the target synthesized speech data is consistent with the content of the target text data, and the pre-trained speech synthesis model is obtained by adversarially training an initial speech synthesis model based on a speech language model SLM discriminator, wherein the SLM discriminator is used to discriminate the SLM features of the sample synthesized speech data and the SLM features of the sample adversarial speech data of the initial speech synthesis model, and the SLM features are generated by the SLM discriminator according to the sample synthesized speech data and the sample adversarial speech data, respectively.

[0009] In some embodiments, the speech synthesis model includes a phoneme encoder, a text encoder, a text style encoder, an audio style encoder, a variance adapter, and a text decoder; the inputting the target text data and the target reference speech data into the pre-trained speech synthesis model and outputting the target synthesized speech data corresponding to the target speaker includes:

[0010] Inputting the target text data into the phoneme encoder and outputting the phoneme features of the target text data;

[0011] Inputting the phoneme features into the text encoder and the text style encoder respectively, and outputting the first text feature and the first text style feature of the target text data respectively;

[0012] Inputting the target reference speech data into the audio style encoder, and outputting the audio style features of the target reference speech data;

[0013] Input the first text feature, the first text style feature and the audio style feature into the variance adapter, and output a second text feature of the target text data; wherein the second text feature is a text feature after adding a target style feature to the first text feature, and the target style feature is a style feature predicted by the variance adapter based on the input first text style feature and the audio style feature;

[0014] The second text feature is input into the text decoder, and the target synthesized speech data corresponding to the target speaker is output.

[0015] In some embodiments, the text style encoder includes an encoding module, a noise module, a diffusion noise reduction module and a decoding module; the inputting the phoneme feature into the text style encoder and outputting the first text style feature of the target text data includes:

[0016] Input the phoneme features into the encoding module, and output the phoneme features after dimensionality reduction;

[0017] Inputting the reduced-dimensional phoneme features and noise data into the diffusion noise reduction module, and outputting the second text style features of the target text data; the noise data is generated by the noise module;

[0018] The second text style feature is input into the decoding module, and the first text style feature of the target text data is output.

[0019] In some embodiments, the method further comprises:

[0020] Acquire at least one training sample, wherein the training sample includes sample text data, sample reference voice data, and sample adversarial voice data;

[0021] Inputting the sample text data and the sample reference speech data into the initial speech synthesis model, and outputting the sample synthesized speech data corresponding to the sample text data;

[0022] Inputting the sample synthesized speech data and the sample adversarial speech data into a first discriminator, and outputting a first discrimination result;

[0023] Inputting the sample synthesized speech data and the sample adversarial speech data into a second discriminator, and outputting a second discrimination result, wherein the second discriminator is an SLM discriminator;

[0024] According to the first discrimination result and the second discrimination result, adversarial training is performed on the initial speech synthesis model to obtain a trained speech synthesis model.

[0025] In some embodiments, the second discriminator includes an SLM model and a discriminant head; the step of inputting the sample synthesized speech data and the sample adversarial speech data into the second discriminator and outputting a second discriminant result includes:

[0026] Input the sample synthesized speech data and the sample adversarial speech data into the SLM model, and output a first SLM feature of the sample synthesized speech data and a second SLM feature of the sample adversarial speech data;

[0027] The first SLM feature and the second SLM feature are input into the discrimination head, and a second discrimination result is output.

[0028] In some embodiments, the loss function corresponding to the second discriminator is:

[0029]

[0030] Among them, DSLM represents the SLM discriminator, C represents the discriminant head in the SLM discriminator, W represents the SLM model in the SLM discriminator, and L slm represents the loss of SLM, represents the minimization of the loss of the generator G, represents the maximization of the loss of the SLM discriminator D, E x and E t represents the least squares loss function calculation, D SLM (x) represents the probability that the real waveform is identified as the real waveform by the SLM discriminator D, D SLM (G(t)) represents the probability that the fake waveform generated by the text data t input generator G is judged as a real waveform by the SLM discriminator D, D SLM (m) represents the optimized discriminator, m represents the real waveform or the fake waveform, represents the data distribution of the real waveform, It indicates the distribution of false waveforms. represents the density function of the real waveform, Density function representing the pseudo waveform.

[0031] In some embodiments, the method further comprises:

[0032] Obtaining the emotion type corresponding to the training sample;

[0033] Determining, according to the emotion type, a first weight corresponding to the first discriminator and a second weight corresponding to the second discriminator;

[0034] The method of performing adversarial training on the initial speech synthesis model according to the first discrimination result and the second discrimination result to obtain a trained speech synthesis model includes:

[0035] According to the first discrimination result, the second discrimination result, the first weight and the second weight, adversarial training is performed on the initial speech synthesis model to obtain a trained speech synthesis model.

[0036] In some embodiments, obtaining the emotion type corresponding to the training sample includes:

[0037] Based on the emotion clustering model, clustering the training samples to obtain a training sample set corresponding to at least one emotion type;

[0038] The emotion type corresponding to the training sample set is determined as the emotion type of the training samples included in the training sample set.

[0039] A second aspect of an embodiment of the present application provides a speech synthesis device based on a reference speech, the device comprising:

[0040] A first acquisition module is configured to acquire target text data and target reference speech data of a target speaker;

[0041] The first output module is configured to input the target text data and the target reference speech data into a pre-trained speech synthesis model, and output the target synthesized speech data corresponding to the target speaker, wherein the target synthesized speech data is consistent with the content of the target text data, and the pre-trained speech synthesis model is obtained by adversarially training an initial speech synthesis model based on a speech language model SLM discriminator, and the SLM discriminator is used to discriminate the SLM features of the sample synthesized speech data and the SLM features of the sample adversarial speech data of the initial speech synthesis model, and the SLM features are generated by the SLM discriminator according to the sample synthesized speech data and the sample adversarial speech data, respectively.

[0042] A third aspect of an embodiment of the present application provides an electronic device, comprising: a processor; and a memory for storing a program; wherein the program comprises instructions, which, when executed by the processor, cause the processor to execute the reference speech-based speech synthesis method of the first aspect described above.

[0043] A fourth aspect of an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the reference speech-based speech synthesis method of the first aspect is implemented.

[0044] A fifth aspect of an embodiment of the present application provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer executes the reference speech-based speech synthesis method of the first aspect mentioned above.

[0045] A sixth aspect of an embodiment of the present application provides a computer program, which, when executed by a processor, can implement the speech synthesis method based on reference speech of the first aspect mentioned above.

[0046] In the method provided in the embodiment of the present application, the electronic device obtains the target text data and the target reference speech data of the target speaker. Then, the electronic device inputs the target text data and the target reference speech data into the pre-trained speech synthesis model, and outputs the target synthesized speech data corresponding to the target speaker. Among them, the content of the target synthesized speech data is consistent with that of the target text data, and the pre-trained speech synthesis model is obtained by adversarial training of the initial speech synthesis model based on the SLM discriminator. The SLM discriminator is used to discriminate the SLM features of the sample synthesized speech data of the initial speech synthesis model and the SLM features of the sample adversarial speech data. The SLM features are generated by the SLM discriminator according to the sample synthesized speech data and the sample adversarial speech data, respectively. In this way, during the training of the speech synthesis model, the speech synthesis model is adversarially trained by the SLM discriminator. In the subsequent use of the speech synthesis model, it is only necessary to obtain the reference speech data of the target speaker with a shorter duration, and it is not necessary to obtain the speech data of the target speaker with a specific duration, so that the synthesized speech data finally output by the speech synthesis model can be guaranteed to be consistent with the sound characteristics of the target speaker. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0048] Figure 1 A flowchart of a speech synthesis method based on reference speech provided in an embodiment of the present application;

[0049] Figure 2 A schematic diagram of the structure of a speech synthesis model provided in an embodiment of the present application;

[0050] Figure 3 A schematic diagram of the structure of a variance adapter provided in an embodiment of the present application;

[0051] Figure 4 A schematic diagram of the structure of a predictor provided in an embodiment of the present application;

[0052] Figure 5 A flowchart of a speech synthesis method based on reference speech provided in an embodiment of the present application;

[0053] Figure 6 A schematic diagram of the structure of a text style encoder provided in an embodiment of the present application;

[0054] Figure 7A flowchart of a method for training a speech synthesis model provided in an embodiment of the present application;

[0055] Figure 8 A flowchart of a method for training a speech synthesis model provided in an embodiment of the present application;

[0056] Fig. 9 A schematic diagram of the structure of a speech synthesis device based on reference speech provided in an embodiment of the present application;

[0057] Fig.10 A schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0058] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below in conjunction with the accompanying drawings.

[0059] It should be understood that the "multiple" mentioned in this application refers to two or more. In the description of this application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in order to facilitate the clear description of the technical solution of this application, the words "first" and "second" are used to distinguish between the same items or similar items with basically the same functions and effects. Those skilled in the art can understand that the words "first" and "second" do not limit the quantity and execution order, and the words "first" and "second" do not limit them to be different.

[0060] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0061] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0062] In recent years, with the advancement of artificial intelligence technology, speech synthesis technology has also developed rapidly. Speech synthesis technology can convert text data into natural, fluent and expressive synthetic speech data.

[0063] At present, in the process of speech synthesis based on a pre-trained speech synthesis model, it is also necessary to further optimize the speech synthesis model based on speech data of a specific length of time (usually 30 minutes to 60 minutes) of the target speaker, so as to ensure that the speech synthesis model can learn the voice characteristics of the target speaker, and then ensure that the speech synthesis model ultimately outputs synthesized speech data close to the target speaker.

[0064] However, if the speech data of a specific length of the target speaker cannot be obtained, it will make it impossible to optimize the training of the pre-trained speech synthesis model, which will cause the synthesized speech data ultimately output by the speech synthesis model to deviate significantly from the sound characteristics of the target speaker.

[0065] In order to solve the above problems, the embodiment of the present application provides a speech synthesis method based on reference speech. The method can be applied to any speech synthesis scenario. The method of the embodiment of the present application is described below. Figure 1 A flowchart of a speech synthesis method based on reference speech is provided in an embodiment of the present application, such as Figure 1 As shown, the specific processing process is as follows.

[0066] Step 101, obtaining target text data and target reference speech data of a target speaker.

[0067] In one embodiment, in order to reduce the deviation between the synthesized speech data finally output by the speech synthesis model and the sound characteristics of the target speaker when the speech data of a specific duration (usually 30 minutes to 60 minutes) of the target speaker cannot be obtained. When the electronic device converts the target text data into the synthesized speech data corresponding to the target speaker through the speech synthesis model, the electronic device can also obtain the target reference speech data of the target speaker. Among them, compared with the speech data of the specific duration of the target speaker required by the traditional speech synthesis model, the speech duration of the target reference speech data of the target speaker in the embodiment of the present application is shorter (for example, 3 seconds to 5 seconds). The target speaker can be a real person or a virtual person, which is not limited in the embodiment of the present application.

[0068] Step 102, input the target text data and the target reference speech data into the pre-trained speech synthesis model, and output the target synthesized speech data corresponding to the target speaker. The target synthesized speech data is consistent with the target text data, and the pre-trained speech synthesis model is obtained by adversarially training the initial speech synthesis model based on the SLM discriminator, and the SLM discriminator is used to discriminate the SLM features of the sample synthesized speech data of the initial speech synthesis model and the SLM features of the sample adversarial speech data, and the SLM features are generated by the SLM discriminator according to the sample synthesized speech data and the sample adversarial speech data, respectively.

[0069] In one embodiment, after the electronic device obtains the target text data and the target reference voice data, it can further input the target text data and the target reference voice data into a pre-trained speech synthesis model. Among them, the pre-trained speech synthesis model is obtained by adversarial training based on a speech language model. Correspondingly, the pre-trained speech synthesis model is obtained by adversarial training of the initial speech synthesis model based on an SLM (speech language model) discriminator. The SLM discriminator is used to discriminate the SLM features of the sample synthesized speech data of the initial speech synthesis model and the SLM features of the sample adversarial speech data. The SLM features are generated by the SLM discriminator according to the sample synthesized speech data and the sample adversarial speech data, respectively. Since the speech synthesis model converts the target text data into the synthesized speech data corresponding to the target speaker. Therefore, the content of the target synthesized speech data is consistent with that of the target text data.

[0070] It should be noted that the structure of the speech synthesis model, the process of the speech synthesis model outputting the target synthesized speech data corresponding to the target speaker, and the adversarial training process of the speech synthesis model will be introduced in detail in the embodiments of the present application, which will not be repeated here.

[0071] In the embodiment of the present application, the traditional discriminator usually performs adversarial training based on the Mel spectrum features of sample synthesized speech data and sample adversarial speech data. The Mel spectrum feature is obtained after Fourier transform of the speech data, and does not need to be generated by a model. It can only characterize the waveform characteristics of the speech data. The SLM feature is generated by the SLM model in the SLM discriminator, which can imitate human perception to evaluate speech data, and the content it represents is richer. Therefore, in the process of adversarial training of the speech synthesis model, using the SLM feature as the basis for discrimination can make the effect of the synthetic speech data generated by the speech synthesis model after adversarial training more ideal.

[0072] In some embodiments, Figure 2 A schematic diagram of the structure of a speech synthesis model provided in an embodiment of the present application is shown in FIG. Figure 2 As shown, the speech synthesis model includes a phoneme encoder, a text encoder, a text style encoder, an audio style encoder, a variance adapter and a text decoder.

[0073] A phoneme encoder is used to extract phoneme features from input text data. Among them, text data is a written form of language content, which can reflect the direct meaning, grammatical structure, contextual association and other information of the language. The phoneme features are the phoneme features of the actual pronunciation formed after further conversion of the text data, which can reflect the pronunciation characteristics and acoustic performance of the language. The phoneme encoder can convert each word in the text data into its corresponding phoneme features based on the text data and the phoneme dictionary, thereby obtaining the phoneme features corresponding to the text data. Among them, the phoneme dictionary can be a preset dictionary library. For example, the phoneme dictionary can be a resource library constructed by itself according to the target language, or the phoneme dictionary can also be an existing phoneme dictionary library, which is not limited in the embodiments of the present application.

[0074] A text encoder is used to convert phoneme features into text features. The text features can represent the semantic content of the text data. The semantic content includes the meaning of words, the structure of sentences, and the meaning of context. In the embodiment of the present application, a feedforward transformer block is used as the basic structure of the text encoder, and the text encoder includes a one-dimensional self-attention layer and a one-dimensional convolutional layer.

[0075] The text style encoder is used to extract text style features from the input phoneme features. The text style features can represent the style features of the text data, such as intonation, stress, rhythm, and emotion. The text style encoder mainly includes a diffusion denoising model. During the training process, the diffusion denoising model gradually transforms the original data into a Gaussian distribution through a diffusion process, so as to learn the relationship between the original data and the noisy data. During the inference process, the diffusion denoising model restores the original data from the Gaussian distribution through a reverse process.

[0076] Specifically, during the training process of the diffusion noise reduction model, the corresponding speech data set (the speech data set is a general speech data set, not the speech data set of the target speaker) is first selected to generate the training phoneme features, and then Gaussian noise is gradually added to the phoneme features in an iterative manner until the phoneme features are converted into a pure Gaussian noise vector. Assume that the training word-level phoneme feature is y 0 , the number of iterations is t, and the pure Gaussian noise vector after adding noise is y t , then y 1 =k 1 *y 0 ,y 2 =k 2 *y 1 ,y 3 =k 3 *y 2 ……y t =k t *y t -1. Among them, k represents the variance, k t represents the variance value corresponding to different iteration times t, k t It can be calculated in advance based on the standard Gaussian distribution, that is, for any t, a certain k can be directly calculated t Used for model training.

[0077] On this basis, if we randomly select a t, we can determine the corresponding k through the above calculation process. t With y t . 0 and t as the training input of the diffusion denoising model, y t As the training output of the diffusion denoising model, the diffusion denoising model can be trained to learn the relationship between speech data and noisy speech data. At the same time, in the process of training the diffusion denoising model, since the loss function changes greatly in the early stage of training, the random selection of t can effectively avoid the diffusion denoising model from over-concentrating on the information of a certain time period in the early stage of training, thereby ignoring the information of other time periods. Based on this, the random distribution of t should be as uniform as possible.

[0078] After completing the training based on sufficient speech data, the training parameters of the diffusion denoising model are reversely processed, so that the diffusion denoising model can restore the original data from the Gaussian distribution through the reverse process to obtain the corresponding style features. Specifically, the phoneme features corresponding to the aforementioned text data are input into the diffusion denoising model, and the diffusion denoising model outputs the text style features of the text data. Through the diffusion denoising model, the text style features represented by the phoneme features can be learned, thereby making the speech synthesis effect of the speech synthesis model more ideal during training and use.

[0079] The audio style encoder is used to extract audio style features from the mel spectrum of the input reference audio. The audio style features can represent the acoustic features of the target speaker. The audio style encoder can be composed of four residual blocks and an average pooling layer.

[0080] A variance adapter is used to add variance information to text features. The variance information may include at least one or more of duration, pitch, and energy. The variance adapter can provide enough information to predict variant speech for the one-to-many mapping problem in speech synthesis models. During the variance adapter training process, the true values ​​of time, pitch, and energy extracted from real speech are added as input to the text features to predict the target synthesized speech. At the same time, independent variance predictions are made for duration, pitch, and energy to generate the target synthesized speech during inference.

[0081] Figure 3 This is a schematic diagram of the structure of a variance adapter provided in an embodiment of the present application. Figure 3 As shown, the variance adapter includes at least one or more of a time predictor, a pitch predictor, and an energy predictor. In the variance adapter, the duration predictor, the pitch predictor, and the energy predictor have similar model structures. With text features as input, the variance of each phoneme (duration) or frame (pitch and energy) is predicted using mean squared error (MSE) loss. Figure 4 A schematic diagram of the structure of a predictor provided in an embodiment of the present application. Taking the time predictor as an example, Figure 4 As shown, the time predictor includes a 2-layer 1D convolutional network (Cove1D) with ReLU activation and an additional linear layer (LinearLayer). Each convolutional network includes a normalization layer (LN) and a Dropout layer. The linear layer is used to project the hidden state into the output sequence. For the duration predictor, the output sequence is the length of each phoneme in the logarithmic domain. For the pitch predictor, the output sequence is the frame-level fundamental frequency sequence, and for the energy predictor, the output sequence is the energy sequence of each mel-spectrogram frame. All predictors share the same model structure, but do not share model parameters.

[0082] The text features, text style features and audio style features output by the above text encoder, text style encoder and audio style encoder are aggregated and input into the variance adapter. Accordingly, the variance adapter adds different variance information such as duration, pitch and energy to the text features.

[0083] The text decoder is used to convert the text features after adding variance into synthetic speech output.

[0084] In some embodiments, based on the structure of the speech synthesis model introduced above, such as Figure 5 As shown, the electronic device inputs the target text data and the target reference speech data into the pre-trained speech synthesis model, and the processing process of outputting the target synthesized speech data corresponding to the target speaker is as follows:

[0085] Step 501, input the target text data into the phoneme encoder, and output the phoneme features of the target text data.

[0086] In one embodiment, after the electronic device obtains the target text data, the target text data can be input into a phoneme encoder. Accordingly, the phoneme encoder can output the phoneme features of the target text data. The phoneme features can represent the pronunciation features and acoustic performance of the target text data.

[0087] Step 502: Input the phoneme features into a text encoder and a text style encoder respectively, and output the first text features and the first text style features of the target text data respectively.

[0088] In one embodiment, after the electronic device obtains the phoneme features of the target text data, it can merge the phoneme features and the phoneme position codes to obtain a merged result. Then, the electronic device can input the merged result into a text encoder. Accordingly, the text encoder can output a first text feature of the target text data. Among them, the first text feature can characterize the semantic content of the target text data. The semantic content can include the meaning of words, the structure of sentences, and the meaning of context, etc. At the same time, the electronic device can also input the phoneme features into a text style encoder. Accordingly, the text style encoder can output a first text style feature of the target text data. Among them, the first text style feature can characterize the style features of the target text data such as intonation, stress, rhythm, emotion, etc.

[0089] Step 503: input the target reference speech data into an audio style encoder, and output the audio style features of the target reference speech data.

[0090] In one embodiment, after the electronic device obtains the target reference speech data of the target speaker, the target reference speech data can be further input into the audio style encoder. Accordingly, the audio style encoder can output the audio style features of the target reference speech data. The audio style features can characterize the acoustic features of the target speaker. Optionally, after the electronic device obtains the target reference speech data, the Mel spectrum of the target reference speech data can be determined, and then the Mel spectrum of the target reference speech data can be input into the audio style encoder.

[0091] Step 504: Input the first text feature, the first text style feature and the audio style feature into the variance adapter, and output the second text feature of the target text data. The second text feature is the text feature after adding the target style feature to the first text feature, and the target style feature is the style feature predicted by the variance adapter based on the input first text style feature and the audio style feature.

[0092] In one embodiment, after the electronic device obtains the first text feature, the first text style feature and the audio style feature, it can merge the first text feature, the first text style feature and the audio style feature to obtain a merged result. Then, the electronic device can input the merged result into the variance adapter. Correspondingly, the variance adapter can obtain the target style feature through the predictor prediction based on the input first text style feature and audio style feature. Among them, the target style feature can characterize the acoustic style features of the target text data and the target speaker. The predictor can include at least one or more of a time predictor, a pitch predictor and an energy predictor. The target style feature can include at least one or more of a time feature, a pitch feature and an energy feature. Then, the variance adapter can add the target style feature to the first text feature and output the second text feature of the target text data.

[0093] Step 505: input the second text feature into a text decoder, and output target synthesized speech data corresponding to the target speaker.

[0094] In one embodiment, after obtaining the second text feature, the electronic device may further merge the second text feature and the phoneme position code to obtain a merged result. Then, the electronic device may input the merged result into a text decoder. Accordingly, the text decoder may output the target synthesized speech data corresponding to the target speaker.

[0095] In some embodiments, Figure 6 This is a schematic diagram of the structure of a text style encoder provided in an embodiment of the present application. Figure 6 As shown, the text style encoder includes an encoding module, a noise module, a diffusion noise reduction module and a decoding module. The encoding module is used to compress the input phoneme features. The encoding module includes a 3-layer convolutional network and a bidirectional long short-term memory (LSTM) layer. The noise module is used to output random Gaussian noise. The diffusion noise reduction module is used to extract text style features from the input phoneme features. The decoding module is used to decompress the text style features. The decoding module includes a fully connected layer.

[0096] Correspondingly, the electronic device inputs the phoneme feature into the text style encoder, and the processing process of outputting the first text style feature of the target text data is as follows:

[0097] Step 1: Input the phoneme features into the encoding module and output the reduced-dimensional phoneme features.

[0098] In one implementation, after the electronic device obtains the phoneme features of the target text data, the phoneme features may be input into the encoding module. Accordingly, the encoding module may perform dimensionality reduction (compression) processing on the phoneme features and output the reduced-dimensional phoneme features.

[0099] Step 2: Input the reduced-dimensional phoneme features and noise data into the diffusion noise reduction module, and output the second text style features of the target text data, wherein the noise data is generated by the noise module.

[0100] In one embodiment, after the electronic device obtains the phoneme features after dimensionality reduction, it can further merge the noise data generated by the noise module with the phoneme features after dimensionality reduction to obtain a merged result. Wherein, the noise data can be random Gaussian noise data, which is not limited in the embodiment of the present application. Then, the electronic device can input the merged result into the diffusion noise reduction module. Accordingly, the diffusion noise reduction module can output the second text style feature of the target text data.

[0101] Step three: input the second text style feature into the decoding module, and output the first text style feature of the target text data.

[0102] In one embodiment, after the electronic device obtains the second text style feature of the target text data, the electronic device can further input the second text style feature into the decoding module. Correspondingly, the decoding module can decompress the second text style feature and output the decompressed second text style feature, that is, the first text style feature of the target text data.

[0103] In the embodiment of the present application, an encoding module and a decoding module are added to the text style encoder to perform feature compression and decompression processing, thereby greatly improving the rate at which the text style encoder outputs text style features.

[0104] In some embodiments, Figure 7 A flowchart of a method for training a speech synthesis model provided in an embodiment of the present application.

[0105] like Figure 7 As shown, the electronic device further performs the following steps:

[0106] Step 701: Obtain at least one training sample, wherein the training sample includes sample text data, sample reference voice data, and sample adversarial voice data.

[0107] In one embodiment, when an electronic device needs to perform adversarial training on an initial speech synthesis model, the electronic device can obtain at least one training sample. The training sample includes sample text data, sample reference voice data, and sample adversarial voice data. The content of the sample text data and the sample reference voice data is consistent; the sample reference voice data and the sample adversarial voice data belong to the same speaker, and the sample reference voice data and the sample adversarial voice data can be the same or different, which is not limited in the embodiment of the present application.

[0108] Step 702: input the sample text data and the sample reference speech data into an initial speech synthesis model, and output the sample synthesized speech data corresponding to the sample text data.

[0109] In one embodiment, after the electronic device obtains at least one training sample, it can further input the sample text data and the sample reference speech data into the initial speech synthesis model. Accordingly, the initial speech synthesis model can output the sample synthesized speech data corresponding to the sample text data.

[0110] Step 703: input the sample synthetic speech data and the sample adversarial speech data into a first discriminator, and output a first discrimination result.

[0111] In one embodiment, reference Figure 2 As shown, the embodiment of the present application uses a first discriminator to distinguish the waveforms of the sample synthesized speech data and the sample adversarial speech data to perform the first adversarial training on the initial speech synthesis model. Therefore, after the electronic device obtains the sample synthesized speech data, the sample synthesized speech data and the sample adversarial speech data can be input into the first discriminator. Accordingly, the first discriminator can output a first discrimination result. Among them, the first discrimination result includes the probability that the sample synthesized speech data is real speech data and the probability that the sample adversarial speech data is real speech data.

[0112] Step 704: Input the sample synthesized speech data and the sample adversarial speech data into a second discriminator, and output a second discrimination result. The second discriminator is an SLM discriminator.

[0113] In one embodiment, reference Figure 2As shown, the embodiment of the present application discriminates the sample synthesized speech data and the sample adversarial speech data on the SLM features through the second discriminator (SLM discriminator) to perform a second adversarial training on the initial speech synthesis model. Therefore, after the electronic device obtains the sample synthesized speech data, the sample synthesized speech data and the sample adversarial speech data can be input into the second discriminator. Accordingly, the second discriminator can output a second discrimination result. Among them, the second discrimination result includes the probability that the SLM features of the sample synthesized speech data are real speech data and the probability that the SLM features of the sample adversarial speech data are real speech data.

[0114] Step 705: Perform adversarial training on the initial speech synthesis model according to the first discrimination result and the second discrimination result to obtain a trained speech synthesis model.

[0115] In one embodiment, after the electronic device obtains the first discrimination result and the second discrimination result, it can perform adversarial training on the initial speech synthesis model according to the first discrimination result and the second discrimination result to obtain a trained speech synthesis model. Among them, the first discrimination of the first discriminator and the second discrimination of the SLM discriminator are independent of each other. That is, the inputs of the first discriminator and the SLM discriminator are both sample synthesized speech data output by the initial speech synthesis model, and the first discriminator and the SLM discriminator perform adversarial training on the initial speech synthesis model in their respective discriminations.

[0116] In some embodiments, Figure 8 A flowchart of a method for training a speech synthesis model provided in an embodiment of the present application. Figure 2 As shown, the second discriminator includes an SLM model and a discriminant head. In the embodiment of the present application, the SLM discriminator is used to discriminate the SLM features of the sample synthesized speech data and the sample adversarial speech data, so as to perform an adversarial training on the initial speech synthesis model. Among them, the SLM model can select the HuBERT large model, or other types of models, which are not limited in the embodiment of the present application; the HuBERT large model is trained by 10,000 hours of WenetSpeech (a multi-domain Chinese speech recognition data set) Chinese data, and uniquely transfers knowledge from the SLM encoder to the generation task through adversarial training. Since the number of parameters of HuBERT is much larger than the number of parameters of the speech synthesis model, in order to avoid its function being too powerful, the parameters of the HuBERT model (represented by W) need to be fixed during the actual training process. The discriminant head can select a convolutional neural network, or other types of networks, which are not limited in the embodiment of the present application. The SLM features can imitate human perception to evaluate the quality of synthesized speech.

[0117] In an embodiment of the present application, a conventional discriminator (i.e., a first discriminator) is usually trained adversarially based on the Mel-spectrogram features of sample synthesized speech data and sample adversarial speech data. The Mel-spectrogram features are obtained after Fourier transform of speech data, and do not need to be generated by a model. They can only characterize the waveform characteristics of speech data. The SLM features are generated by the SLM model in the SLM discriminator, which can imitate human perception to evaluate speech data, and the content represented is richer. Therefore, in the process of adversarial training of the speech synthesis model, using the SLM features as the basis for discrimination can make the effect of the synthesized speech data generated by the speech synthesis model after adversarial training more ideal.

[0118] like Figure 8 As shown, the electronic device inputs the sample synthesized speech data and the sample adversarial speech data into the second discriminator, and the processing process of outputting the second discrimination result is as follows:

[0119] Step 801, input sample synthetic speech data and sample adversarial speech data into the SLM model, and output the first SLM features of the sample synthetic speech data and the second SLM features of the sample adversarial speech data.

[0120] In one embodiment, after the electronic device obtains the sample synthesized speech data, the sample synthesized speech data and the sample adversarial speech data can be input into the SLM model of the SLM discriminator. Accordingly, the SLM model can output the first SLM feature of the sample synthesized speech data and the second SLM feature of the sample adversarial speech data.

[0121] Step 802, input the first SLM feature and the second SLM feature into the discrimination head, and output a second discrimination result.

[0122] In one embodiment, after the electronic device obtains the first SLM feature and the second SLM feature, the first SLM feature and the second SLM feature can be further input into the discrimination head of the SLM discriminator. Accordingly, the discrimination head can output a second discrimination result. The second discrimination result includes the probability that the first SLM feature of the sample synthetic speech data is the real speech data and the probability that the SLM feature of the sample adversarial speech data is the real speech data.

[0123] For the first discriminator, the specific adversarial training process is: the electronic device can determine the generator (i.e., speech synthesis model) loss and the discriminator (i.e., first discriminator) loss based on the first discrimination result and the first loss function of the first discriminator, and perform adversarial training on the generator and the discriminator based on the generator loss and the discriminator loss. The loss function for the first discriminator is as follows:

[0124] The first is the least-squares loss function used for adversarial training, which is formulated as follows:

[0125] L adv (D)=E (y,z) [(D(y)-1) 2 +(D(G(z))) 2 ]

[0126] L adv (G) = E z [(D(G(z))-1) 2 ]

[0127] Among them, L adv (D) represents the loss of the discriminator D, E (y,z) represents the least squares loss function calculation, D(y) represents the probability that the real waveform y is judged as the real waveform by the discriminator D, G(z) represents the false waveform output by the text z input generator G, and D(G(z)) represents the probability that the false waveform is judged as the real waveform by the discriminator D; L adv (G) represents the loss of generator G, E z Represents the least squares loss function calculation.

[0128] The second is the feature matching loss specifically applied to the generator G. The feature matching loss can be regarded as a reconstruction loss, which is used to constrain the output of the intermediate layer of the discriminator. The formula is as follows:

[0129]

[0130] Among them, L fm (G) represents the feature matching loss of generator G, E (y,z) represents the least squares loss function calculation, T represents the number of layers of the discriminator D, and D l Represents the output feature map of the l-th layer discriminator, N l represents the number of feature maps in layer l, D l (y) represents the probability that the real waveform y is judged as the real waveform by the l-th layer discriminator, G(z) represents the fake waveform output by the text z input generator G, D l (G(z)) represents the probability that a fake waveform is identified as a real waveform by the l-th layer discriminator.

[0131] For the SLM discriminator, the specific adversarial training process is: the electronic device can determine the generator (i.e., speech synthesis model) loss and the discriminator (i.e., second discriminator) loss based on the second discrimination result and the second loss function of the second discriminator, and perform adversarial training on the generator and the discriminator based on the generator loss and the discriminator loss. SLM =C°W (C represents the discriminant head in the SLM discriminator, and W represents the SLM model in the SLM discriminator) is as follows:

[0132]

[0133] Among them, L slm represents the loss of SLM, represents the minimization of the loss of the generator G, represents the maximization of the loss of the SLM discriminator D, E x and E t represents the least squares loss function calculation, D SLM (x) represents the probability that the real waveform is identified as the real waveform by the SLM discriminator D, D SLM (G(t)) represents the probability that the fake waveform generated by the text input generator G is judged as a real waveform by the SLM discriminator D, D SLM (m) represents the optimized discriminator, m can be a real waveform or a fake waveform, represents the data distribution of the real waveform, It indicates the distribution of false waveforms. represents the density function of the real waveform, represents the density function of the pseudo waveform, if Then the generator G is optimal, which means that when converged, the data distribution of the real waveform and the data distribution of the fake waveform can match in the SLM feature space, which can effectively imitate human perception and thus achieve human-like speech synthesis, that is, the generated speech effect is closer to the real human speech, and it is difficult to distinguish between the true and the false.

[0134] In some embodiments, the electronic device may also perform adversarial training with different emphases based on the emotion type of the training sample, and the electronic device may further perform the following steps:

[0135] Step 1: Obtain the emotion type corresponding to the training sample.

[0136] In one implementation, the electronic device may obtain the emotion type corresponding to the training sample, wherein the emotion type may include a neutral emotion type and a strong emotion type, or may include other types, which are not limited in the present application embodiment.

[0137] Step 2: Determine the first weight corresponding to the first discriminator and the second weight corresponding to the second discriminator according to the emotion type.

[0138] In one embodiment, the embodiment of the present application uses a first discriminator and an SLM discriminator to perform comprehensive adversarial training. Therefore, for training samples of different emotion types, the first discriminator and the SLM discriminator have different emphases, that is, during the training process, different weights are assigned to the first discriminator and the SLM discriminator for different emotion types. For example, for training samples of the emotion-bland type, the weight of the first discriminator is higher, and the weight of the SLM discriminator is lower. For training samples of the emotion-strong type, the weight of the SLM discriminator is higher, and the weight of the first discriminator is lower. After the electronic device obtains the emotion type corresponding to the training sample, it can determine the first weight corresponding to the first discriminator and the second weight corresponding to the second discriminator according to the emotion type.

[0139] Correspondingly, the electronic device performs adversarial training on the initial speech synthesis model according to the first discrimination result and the second discrimination result, and the processing process of obtaining the trained speech synthesis model is: according to the first discrimination result, the second discrimination result, the first weight and the second weight, the initial speech synthesis model is performed adversarial training to obtain the trained speech synthesis model.

[0140] In one implementation, the electronic device determines a first weight corresponding to the first discriminator and a second weight corresponding to the second discriminator, and can further perform adversarial training on the initial speech synthesis model based on the first discrimination result, the second discrimination result, the first weight, and the second weight to obtain a trained speech synthesis model. For example, the electronic device determines that the parameter adjustment amount of the speech synthesis model is △T based on the first discrimination result. 1 , the electronic device can determine the parameter adjustment amount of the final speech synthesis model as a*△T based on the first weight a 1 Similarly, the electronic device determines the parameter adjustment amount of the speech synthesis model as △T based on the second discrimination result. 2 , the electronic device can determine the parameter adjustment amount of the final speech synthesis model as b*△T based on the second weight b. 2 .

[0141] In some embodiments, the process of the electronic device obtaining the emotion type corresponding to the training sample is as follows:

[0142] Step 1: Cluster the training samples based on the sentiment clustering model to obtain a training sample set corresponding to at least one sentiment type.

[0143] In one embodiment, the electronic device can cluster the training samples based on the emotion clustering model to obtain a training sample set corresponding to at least one emotion type. Among them, the emotion clustering model can select the BERT model (Bidirectional Encoder Representations from Transformers, a pre-trained language model based on the Transformer architecture), or other types of models, which are not limited in the embodiments of the present application. The emotion clustering model can cluster the training samples based on the intensity of emotion expressed by the sample text data in the training samples and / or the intensity of emotion expressed by the sample reference voice data to obtain a training sample set corresponding to each emotion type.

[0144] Step 2: Determine the emotion type corresponding to the training sample set as the emotion type of the training samples contained in the training sample set.

[0145] In one implementation, after obtaining the training sample set corresponding to each emotion type, the electronic device may determine the emotion type corresponding to the training sample set as the emotion type of the training samples included in the training sample set.

[0146] In the speech synthesis method based on reference speech provided in the embodiment of the present application, the electronic device obtains target text data and target reference speech data of the target speaker. Then, the electronic device inputs the target text data and the target reference speech data into the pre-trained speech synthesis model, and outputs the target synthesized speech data corresponding to the target speaker. Among them, the target synthesized speech data is consistent with the content of the target text data, and the pre-trained speech synthesis model is obtained by adversarial training of the initial speech synthesis model based on the SLM discriminator, and the SLM discriminator is used to discriminate the SLM features of the sample synthesized speech data of the initial speech synthesis model and the SLM features of the sample adversarial speech data. In this way, during the training of the speech synthesis model, the speech synthesis model is adversarially trained by the SLM discriminator. In the subsequent use of the speech synthesis model, it is only necessary to obtain the reference speech data of the target speaker with a shorter duration, and it is not necessary to obtain the speech data of the target speaker with a specific duration, which can ensure that the synthesized speech data finally output by the speech synthesis model is consistent with the sound characteristics of the target speaker.

[0147] Fig. 9 The structure diagram of a speech synthesis device based on reference speech provided in an embodiment of the present application is shown in FIG. Fig. 9 As shown, the speech synthesis device 900 based on reference speech includes: a first acquisition module 910 and a first output module 920. Among them:

[0148] A first acquisition module 910, configured to acquire target text data and target reference speech data of a target speaker;

[0149] The first output module 920 is used to input the target text data and the target reference speech data into a pre-trained speech synthesis model, and output the target synthesized speech data corresponding to the target speaker. The target synthesized speech data is consistent with the content of the target text data. The pre-trained speech synthesis model is obtained by adversarially training the initial speech synthesis model based on the speech language model SLM discriminator. The SLM discriminator is used to discriminate the SLM features of the sample synthesized speech data of the initial speech synthesis model and the SLM features of the sample adversarial speech data. The SLM features are generated by the SLM discriminator according to the sample synthesized speech data and the sample adversarial speech data, respectively.

[0150] In some embodiments, the speech synthesis model includes a phoneme encoder, a text encoder, a text style encoder, an audio style encoder, a variance adapter, and a text decoder; the first output module 920 is specifically used to:

[0151] Input the target text data into the phoneme encoder and output the phoneme features of the target text data;

[0152] Inputting the phoneme features into a text encoder and a text style encoder respectively, and outputting a first text feature and a first text style feature of the target text data respectively;

[0153] Inputting the target reference speech data into an audio style encoder and outputting the audio style features of the target reference speech data;

[0154] Input the first text feature, the first text style feature and the audio style feature into the variance adapter, and output the second text feature of the target text data; wherein the second text feature is the text feature after adding the target style feature to the first text feature, and the target style feature is the style feature predicted by the variance adapter based on the input first text style feature and the audio style feature;

[0155] The second text feature is input into the text decoder, and the target synthesized speech data corresponding to the target speaker is output.

[0156] In some embodiments, the text style encoder includes an encoding module, a noise module, a diffusion noise reduction module and a decoding module; the first output module 920 is specifically used to:

[0157] Input the phoneme features into the encoding module and output the phoneme features after dimensionality reduction;

[0158] Inputting the reduced-dimensional phoneme features and noise data into the diffusion noise reduction module, and outputting the second text style features of the target text data; the noise data is generated by the noise module;

[0159] The second text style feature is input into the decoding module, and the first text style feature of the target text data is output.

[0160] In some embodiments, the apparatus further comprises:

[0161] A second acquisition module, used to acquire at least one training sample, the training sample including sample text data, sample reference voice data and sample adversarial voice data;

[0162] A second output module, used to input the sample text data and the sample reference speech data into the initial speech synthesis model, and output the sample synthesized speech data corresponding to the sample text data;

[0163] A third output module, used for inputting the sample synthesized speech data and the sample adversarial speech data into the first discriminator, and outputting a first discrimination result;

[0164] A fourth output module, used for inputting the sample synthesized speech data and the sample adversarial speech data into a second discriminator, and outputting a second discrimination result, wherein the second discriminator is an SLM discriminator;

[0165] The training module is used to perform adversarial training on the initial speech synthesis model according to the first discrimination result and the second discrimination result to obtain a trained speech synthesis model.

[0166] In some embodiments, the second discriminator includes an SLM model and a discriminant head; and a fourth output module, specifically configured to:

[0167] Input the sample synthetic speech data and the sample adversarial speech data into the SLM model, and output the first SLM feature of the sample synthetic speech data and the second SLM feature of the sample adversarial speech data;

[0168] The first SLM feature and the second SLM feature are input into the discrimination head, and a second discrimination result is output.

[0169] In some embodiments, the loss function corresponding to the second discriminator is:

[0170]

[0171] Among them, D SLM represents the SLM discriminator, C represents the discriminant head in the SLM discriminator, W represents the SLM model in the SLM discriminator, and L slm represents the loss of SLM, represents the minimization of the loss of the generator G, represents the maximization of the loss of the SLM discriminator D, E x and E t represents the least squares loss function calculation, D SLM(x) represents the probability that the real waveform is identified as the real waveform by the SLM discriminator D, D SLM (G(t)) represents the probability that the fake waveform generated by the text data t input generator G is judged as a real waveform by the SLM discriminator D, D SLM (m) represents the optimized discriminator, m represents the real waveform or the fake waveform, represents the data distribution of the real waveform, It indicates the distribution of false waveforms. represents the density function of the real waveform, Density function representing the pseudo waveform.

[0172] In some embodiments, the apparatus further comprises:

[0173] The third acquisition module is used to obtain the emotion type corresponding to the training sample;

[0174] A determination module, used to determine a first weight corresponding to the first discriminator and a second weight corresponding to the second discriminator according to the emotion type;

[0175] According to the first discrimination result and the second discrimination result, the initial speech synthesis model is subjected to adversarial training to obtain a trained speech synthesis model, including:

[0176] According to the first discrimination result, the second discrimination result, the first weight and the second weight, adversarial training is performed on the initial speech synthesis model to obtain a trained speech synthesis model.

[0177] In some embodiments, the third acquisition module is specifically configured to:

[0178] Based on the sentiment clustering model, the training samples are clustered to obtain a training sample set corresponding to at least one sentiment type;

[0179] The emotion type corresponding to the training sample set is determined as the emotion type of the training samples contained in the training sample set.

[0180] Fig.10 A schematic diagram of an electronic device provided in an embodiment of the present application. In some embodiments, the electronic device includes one or more processors and a memory. The memory is configured to store one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the speech synthesis method based on the reference speech in the above-mentioned embodiment.

[0181] like Fig.10As shown, the electronic device 1000 includes: a processor (processor) 1001 and a memory 1002. Exemplarily, the electronic device 1000 may also include: a communication interface (Communications Interface) 1003 and a communication bus 1004.

[0182] The processor 1001, the memory 1002 and the communication interface 1003 communicate with each other via the communication bus 1004. The communication interface 1003 is used to communicate with other devices such as client or other server network elements.

[0183] In some embodiments, the processor 1001 is used to execute the program 1005, and specifically, the relevant steps in the above-mentioned speech synthesis method embodiment based on reference speech can be executed. Specifically, the program 1005 can include program code, and the program code includes computer executable instructions.

[0184] Exemplarily, the processor 1001 may be a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application. The electronic device 1000 may include one or more processors, which may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.

[0185] In some embodiments, the memory 1002 is used to store the program 1005. The memory 1002 may include a high-speed RAM memory, and may also include a non-volatile memory (NVM), such as at least one disk memory.

[0186] Program 1005 can be specifically called by processor 1001 to enable electronic device 1000 to perform operations of a speech synthesis method based on reference speech.

[0187] An embodiment of the present application provides a computer-readable storage medium, which stores at least one executable instruction. When the executable instruction is executed on the electronic device 1000, the electronic device 1000 executes the speech synthesis method based on the reference speech in the above embodiment.

[0188] The executable instructions can be specifically used to enable the electronic device 1000 to perform operations of a speech synthesis method based on a reference speech.

[0189] For example, the computer readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc (CD-ROM), a magnetic tape, a floppy disk, an optical data storage device, and the like.

[0190] In some embodiments, an embodiment of the present application provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer executes the reference speech-based speech synthesis method described in any of the above embodiments.

[0191] In some embodiments, the embodiments of the present application further provide a computer program, which, when executed by a processor, can implement the speech synthesis method based on reference speech described in any of the above embodiments.

[0192] The beneficial effects that can be achieved by the speech synthesis device based on reference speech, electronic device, computer-readable storage medium, computer program product and computer program provided in the embodiments of the present application can refer to the beneficial effects of the speech synthesis method based on reference speech provided above, and will not be repeated here.

[0193] It should be noted that, in the application, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "including a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0194] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0195] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, which can be embodied in any computer-readable medium for use by an instruction execution system, apparatus or device (such as a computer-based system, a system including a processor or other system that can fetch instructions from an instruction execution system, apparatus or device and execute instructions), or used in combination with these instruction execution systems, apparatuses or devices.

[0196] For the purposes of this specification, a "computer-readable medium" can be any apparatus that can contain, store, communicate, propagate, or transport the program for use by an instruction execution system, apparatus, or device or in conjunction with such instruction execution system, apparatus, or device.

[0197] More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection having one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM).

[0198] In addition, the computer readable medium can even be a paper or other suitable medium on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other medium, and then editing, interpreting or processing in other suitable ways as necessary, and then storing it in the computer memory. It should be understood that various parts of the present application can be implemented in hardware, software, firmware or a combination thereof.

[0199] In the above embodiments, multiple steps or methods may be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it may be implemented by any one of the following technologies known in the art or a combination thereof: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0200] The above-described embodiments of the present application do not constitute a limitation on the protection scope of the present application.

Claims

1. A speech synthesis method based on reference speech, characterized in that: The method comprises: Obtaining target text data and target reference speech data of a target speaker; The target text data and the target reference speech data are input into a pre-trained speech synthesis model, and the target synthesized speech data corresponding to the target speaker is output, wherein the target synthesized speech data is consistent with the content of the target text data, and the pre-trained speech synthesis model is obtained by adversarially training an initial speech synthesis model based on a speech language model SLM discriminator, wherein the SLM discriminator is used to discriminate the SLM features of the sample synthesized speech data and the SLM features of the sample adversarial speech data of the initial speech synthesis model, and the SLM features are generated by the SLM discriminator according to the sample synthesized speech data and the sample adversarial speech data, respectively.

2. The method according to claim 1, characterized in that: The speech synthesis model includes a phoneme encoder, a text encoder, a text style encoder, an audio style encoder, a variance adapter and a text decoder; the target text data and the target reference speech data are input into the pre-trained speech synthesis model, and the target synthesized speech data corresponding to the target speaker is output, including: Inputting the target text data into the phoneme encoder and outputting the phoneme features of the target text data; Inputting the phoneme features into the text encoder and the text style encoder respectively, and outputting the first text feature and the first text style feature of the target text data respectively; Inputting the target reference speech data into the audio style encoder, and outputting the audio style features of the target reference speech data; Input the first text feature, the first text style feature and the audio style feature into the variance adapter, and output a second text feature of the target text data; wherein the second text feature is a text feature after adding a target style feature to the first text feature, and the target style feature is a style feature predicted by the variance adapter based on the input first text style feature and the audio style feature; The second text feature is input into the text decoder, and the target synthesized speech data corresponding to the target speaker is output.

3. The method according to claim 2, characterized in that The text style encoder comprises an encoding module, a noise module, a diffusion noise reduction module and a decoding module; the inputting the phoneme feature into the text style encoder and outputting the first text style feature of the target text data comprises: Inputting the phoneme features into the encoding module, and outputting the phoneme features after dimensionality reduction; Inputting the reduced-dimensional phoneme features and noise data into the diffusion noise reduction module, and outputting the second text style features of the target text data; the noise data is generated by the noise module; The second text style feature is input into the decoding module, and the first text style feature of the target text data is output.

4. The method according to claim 1, characterized in that: The method further comprises: Acquire at least one training sample, wherein the training sample includes sample text data, sample reference voice data, and sample adversarial voice data; Inputting the sample text data and the sample reference speech data into the initial speech synthesis model, and outputting the sample synthesized speech data corresponding to the sample text data; Inputting the sample synthesized speech data and the sample adversarial speech data into a first discriminator, and outputting a first discrimination result; Inputting the sample synthesized speech data and the sample adversarial speech data into a second discriminator, and outputting a second discrimination result, wherein the second discriminator is an SLM discriminator; According to the first discrimination result and the second discrimination result, adversarial training is performed on the initial speech synthesis model to obtain a trained speech synthesis model.

5. The method according to claim 4, characterized in that The second discriminator includes an SLM model and a discriminant head; The step of inputting the sample synthesized speech data and the sample adversarial speech data into a second discriminator and outputting a second discrimination result comprises: Input the sample synthesized speech data and the sample adversarial speech data into the SLM model, and output a first SLM feature of the sample synthesized speech data and a second SLM feature of the sample adversarial speech data; The first SLM feature and the second SLM feature are input into the discrimination head, and a second discrimination result is output.

6. The method according to claim 4, characterized in that The loss function corresponding to the second discriminator is: Among them, D SLM represents the SLM discriminator, C represents the discriminant head in the SLM discriminator, W represents the SLM model in the SLM discriminator, and L slm represents the loss of SLM, represents the minimization of the loss of the generator G, represents the maximization of the loss of the SLM discriminator D, E x and E t represents the least squares loss function calculation, D SLM (x) represents the probability that the real waveform is identified as the real waveform by the SLM discriminator D, D SLM (G(t)) represents the probability that the fake waveform generated by the text data t input generator G is judged as a real waveform by the SLM discriminator D, D SLM (m) represents the optimized discriminator, m represents the real waveform or the fake waveform, represents the data distribution of the real waveform, It indicates the distribution of false waveforms. represents the density function of the real waveform, Density function representing the pseudo waveform.

7. The method according to claim 6, characterized in that The method further comprises: Obtaining the emotion type corresponding to the training sample; Determining, according to the emotion type, a first weight corresponding to the first discriminator and a second weight corresponding to the second discriminator; The method of performing adversarial training on the initial speech synthesis model according to the first discrimination result and the second discrimination result to obtain a trained speech synthesis model includes: According to the first discrimination result, the second discrimination result, the first weight and the second weight, adversarial training is performed on the initial speech synthesis model to obtain a trained speech synthesis model.

8. The method according to claim 7, characterized in that The obtaining of the emotion type corresponding to the training sample comprises: Based on the emotion clustering model, clustering the training samples to obtain a training sample set corresponding to at least one emotion type; The emotion type corresponding to the training sample set is determined as the emotion type of the training samples included in the training sample set.

9. A speech synthesis device based on reference speech, characterized in that: The device comprises: A first acquisition module is configured to acquire target text data and target reference speech data of a target speaker; The first output module is configured to input the target text data and the target reference speech data into a pre-trained speech synthesis model, and output the target synthesized speech data corresponding to the target speaker, wherein the target synthesized speech data is consistent with the content of the target text data, and the pre-trained speech synthesis model is obtained by adversarially training an initial speech synthesis model based on a speech language model SLM discriminator, and the SLM discriminator is used to discriminate the SLM features of the sample synthesized speech data and the SLM features of the sample adversarial speech data of the initial speech synthesis model, and the SLM features are generated by the SLM discriminator according to the sample synthesized speech data and the sample adversarial speech data, respectively.

10. An electronic device, comprising: processor; and a memory storing a program; wherein the program comprises instructions, which, when executed by the processor, cause the processor to perform the method according to any one of claims 1-8.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method steps of any one of claims 1 to 8 are implemented.

Images