A speech synthesis method, apparatus, and electronic device based on reference speech.

Abstract

This application discloses a speech synthesis method, apparatus, and electronic device based on reference speech. The method includes: acquiring target text data and target reference speech data of a target speaker; inputting the target text data and target reference speech data into a pre-trained speech synthesis model, and outputting target synthesized speech data corresponding to the target speaker. The pre-trained speech synthesis model is obtained by adversarial training of an initial speech synthesis model based on an SLM discriminator. The SLM discriminator is used to distinguish between the SLM features of sample synthesized speech data and sample adversarial speech data of the initial speech synthesis model. The SLM features are generated by the SLM discriminator based on the sample synthesized speech data and sample adversarial speech data, respectively. Using this application, even when it is impossible to obtain speech data of a specific duration from the target speaker, it can still ensure that the synthesized speech data finally output by the speech synthesis model is consistent with the voice features of the target speaker.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computers, and in particular to a speech synthesis method and device based on reference speech and electronic equipment. BACKGROUND

[0002] In recent years, with the progress of artificial intelligence technology, speech synthesis technology (TTS) has also developed rapidly. Speech synthesis technology can convert text data into natural, fluent and expressive synthesized speech data.

[0003] Currently, in the process of speech synthesis based on a pre-trained speech synthesis model, further optimization training of the speech synthesis model is needed based on speech data of a specific duration (usually 30 minutes to 60 minutes) of a target speaker, so as to ensure that the speech synthesis model can learn the voice characteristics of the target speaker, and thus ensure that the synthesized speech data output by the speech synthesis model is close to the target speaker.

[0004] However, in the case where the speech data of a specific duration of the target speaker cannot be obtained, the pre-trained speech synthesis model cannot be optimized and trained, and thus the synthesized speech data output by the speech synthesis model deviates greatly from the voice characteristics of the target speaker. SUMMARY

[0005] To solve the above problems, the embodiments of the present application provide a speech synthesis method and device based on reference speech and electronic equipment, which can reduce the deviation of the synthesized speech data output by the speech synthesis model from the voice characteristics of the target speaker in the case where the speech data of a specific duration of the target speaker cannot be obtained. Specifically, the embodiments of the present application disclose the following technical solutions:

[0006] The first aspect of the embodiments of the present application provides a speech synthesis method based on reference speech, comprising:

[0007] obtaining target text data and target reference speech data of a target speaker;

[0008] inputting the target text data and the target reference speech data into a pre-trained speech synthesis model, and outputting target synthesized speech data corresponding to the target speaker, the target synthesized speech data being consistent with the content of the target text data, the pre-trained speech synthesis model being obtained by performing adversarial training on an initial speech synthesis model based on a speech language model (SLM) discriminator, the SLM discriminator being configured to discriminate between SLM features of sample synthesized speech data of the initial speech synthesis model and SLM features of sample adversarial speech data, the SLM features being respectively generated by the SLM discriminator based on the sample synthesized speech data and the sample adversarial speech data.

[0009] In some embodiments, the speech synthesis model comprises a phoneme encoder, a text encoder, a text style encoder, an audio style encoder, a variance adapter, and a text decoder; and the inputting the target text data and the target reference speech data into a pre-trained speech synthesis model, and outputting target synthesized speech data corresponding to the target speaker, comprises:

[0010] inputting the target text data into the phoneme encoder, and outputting phoneme features of the target text data;

[0011] inputting the phoneme features into the text encoder and the text style encoder respectively, and outputting first text features and first text style features of the target text data respectively;

[0012] inputting the target reference speech data into the audio style encoder, and outputting audio style features of the target reference speech data;

[0013] inputting the first text features, the first text style features, and the audio style features into the variance adapter, and outputting second text features of the target text data; wherein the second text features are text features obtained by adding target style features to the first text features, the target style features being style features predicted by the variance adapter based on the input first text style features and the audio style features;

[0014] inputting the second text features into the text decoder, and outputting the target synthesized speech data corresponding to the target speaker.

[0015] In some embodiments, the text style encoder comprises an encoding module, a noise module, a diffusion noise reduction module, and a decoding module; and the inputting the phoneme features into the text style encoder, and outputting the first text style features of the target text data, comprises:

[0016] inputting the phoneme features into the encoding module, and outputting reduced-dimension phoneme features;

[0017] inputting the reduced-dimension phoneme features and noise data into the diffusion denoising module, and outputting second text style features of the target text data; the noise data is generated by the noise module;

[0018] inputting the second text style features into the decoding module, and outputting first text style features of the target text data.

[0019] In some embodiments, the method further comprises:

[0020] obtaining at least one training sample, the training sample comprising sample text data, sample reference speech data, and sample adversarial speech data;

[0021] inputting the sample text data and the sample reference speech data into the initial speech synthesis model, and outputting sample synthesized speech data corresponding to the sample text data;

[0022] inputting the sample synthesized speech data and the sample adversarial speech data into a first discriminator, and outputting a first discrimination result;

[0023] inputting the sample synthesized speech data and the sample adversarial speech data into a second discriminator, and outputting a second discrimination result, the second discriminator being an SLM discriminator;

[0024] performing adversarial training on the initial speech synthesis model according to the first discrimination result and the second discrimination result, to obtain a trained speech synthesis model.

[0025] In some embodiments, the second discriminator comprises an SLM model and a discrimination head; and the inputting the sample synthesized speech data and the sample adversarial speech data into the second discriminator, and outputting the second discrimination result, comprises:

[0026] inputting the sample synthesized speech data and the sample adversarial speech data into the SLM model, and outputting first SLM features of the sample synthesized speech data and second SLM features of the sample adversarial speech data;

[0027] inputting the first SLM features and the second SLM features into the discrimination head, and outputting the second discrimination result.

[0028] In some embodiments, the loss function corresponding to the second discriminator is:

[0029]

[0030] wherein DSLM denotes an SLM discriminator, C denotes a discriminant head in the SLM discriminator, W denotes an SLM model in the SLM discriminator, L slm denotes a loss of the SLM, denotes loss minimization of the generator G, denotes loss maximization of the SLM discriminator D, E x and E t denotes a least square loss function calculation, D SLM (x) denotes a probability that a real waveform is discriminated as a real waveform by the SLM discriminator D, D SLM (G(t)) denotes a probability that a fake waveform generated by inputting text data t into the generator G is discriminated as a real waveform by the SLM discriminator D, D SLM (m) denotes an optimized discriminator, m denotes a real waveform or a fake waveform, denotes a data distribution of a real waveform, denotes a distribution that is a fake waveform, denotes a density function of a real waveform, denotes a density function of a fake waveform.

[0031] In some embodiments, the method further comprises:

[0032] obtaining a sentiment type corresponding to the training sample;

[0033] According to the sentiment type, determining a first weight corresponding to the first discriminator and a second weight corresponding to the second discriminator;

[0034] According to the first discrimination result and the second discrimination result, the initial speech synthesis model is adversarial training to obtain a trained speech synthesis model, comprising:

[0035] According to the first discrimination result, the second discrimination result, the first weight and the second weight, the initial speech synthesis model is adversarial training to obtain a trained speech synthesis model.

[0036] In some embodiments, the obtaining a sentiment type corresponding to the training sample comprises:

[0037] Based on the sentiment clustering model, the training samples are clustered to obtain at least one training sample set corresponding to a sentiment type;

[0038] The sentiment type corresponding to the training sample set is determined as the sentiment type of the training sample contained in the training sample set.

[0039] The second aspect of the embodiment of the application provides a speech synthesis device based on a reference speech, the device comprises:

[0040] a first obtaining module configured to obtain target text data and target reference speech data of a target speaker;

[0041] a first output module configured to input the target text data and the target reference speech data into a pre-trained speech synthesis model, and output target synthesized speech data corresponding to the target speaker, the target synthesized speech data being consistent with the content of the target text data, the pre-trained speech synthesis model being obtained by performing adversarial training on an initial speech synthesis model based on a speech language model (SLM) discriminator, the SLM discriminator being configured to discriminate between SLM features of sample synthesized speech data of the initial speech synthesis model and SLM features of sample adversarial speech data, the SLM features being generated by the SLM discriminator based on the sample synthesized speech data and the sample adversarial speech data, respectively.

[0042] The third aspect of the embodiments of the present application provides an electronic device, comprising: a processor; and a memory storing a program; wherein the program includes instructions that, when executed by the processor, cause the processor to perform the speech synthesis method based on reference speech of the first aspect.

[0043] The fourth aspect of the embodiments of the present application provides a computer-readable storage medium having a computer program stored thereon, the computer program being executed by a processor to implement the speech synthesis method based on reference speech of the first aspect.

[0044] The fifth aspect of the embodiments of the present application provides a computer program product, the computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions that, when executed by a computer, cause the computer to perform the speech synthesis method based on reference speech of the first aspect.

[0045] The sixth aspect of the embodiments of the present application provides a computer program, which, when executed by a processor, can implement the speech synthesis method based on reference speech of the first aspect.

[0046] In the method provided by the embodiment of the application, the electronic device obtains target text data and target reference speech data of a target speaker. Then, the electronic device inputs the target text data and the target reference speech data into a pre-trained speech synthesis model, and outputs target synthesized speech data corresponding to the target speaker. The target synthesized speech data is consistent with the content of the target text data, and the pre-trained speech synthesis model is obtained by performing adversarial training on an initial speech synthesis model based on an SLM discriminator. The SLM discriminator is used to distinguish SLM features of sample synthesized speech data of the initial speech synthesis model and SLM features of sample adversarial speech data. The SLM features are respectively generated by the SLM discriminator according to the sample synthesized speech data and the sample adversarial speech data. In this way, in the speech synthesis model training process, the speech synthesis model is subjected to adversarial training by the SLM discriminator. In the subsequent use process of the speech synthesis model, only the reference speech data with a short time length of the target speaker needs to be obtained, and the speech data with a specific time length of the target speaker does not need to be obtained, so that the synthesized speech data finally output by the speech synthesis model can be ensured to be consistent with the voice features of the target speaker. BRIEF DESCRIPTION OF DRAWINGS

[0047] In order to more clearly illustrate the technical solutions of the embodiments of the application, the drawings needed in the embodiments will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0048] Figure 1 A flowchart of a speech synthesis method based on reference speech provided by the embodiment of the application;

[0049] Figure 2 A structural schematic diagram of a speech synthesis model provided by the embodiment of the application;

[0050] Figure 3 A structural schematic diagram of a variance adapter provided by the embodiment of the application;

[0051] Figure 4 A structural schematic diagram of a predictor provided by the embodiment of the application;

[0052] Figure 5 A flowchart of a speech synthesis method based on reference speech provided by the embodiment of the application;

[0053] Figure 6 A structural schematic diagram of a text style encoder provided by the embodiment of the application;

[0054] Figure 7A flowchart of a method for training a speech synthesis model is provided for embodiments of the present application.

[0055] Figure 8 A flowchart of a method for training a speech synthesis model is provided for embodiments of the present application.

[0056] Figure 9 A structural schematic diagram of a speech synthesis device based on a reference speech is provided for embodiments of the present application.

[0057] Figure 10 A schematic diagram of an electronic device is provided for embodiments of the present application. DETAILED DESCRIPTION

[0058] In order to make the objectives, technical solutions and advantages of the present application clearer, the embodiments of the present application will be further described in detail below with reference to the drawings.

[0059] It should be understood that the "multiple" mentioned in the present application refers to two or more than two. In the description of the present application, unless otherwise specified, " / " represents the meaning of or, for example, A / B can represent A or B; "and / or" in the present application is only a description of the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B, which can represent the three cases of A alone, A and B together, and B alone. In addition, in order to clearly describe the technical solutions of the present application, the same items or similar items with basically the same functions and roles are distinguished by using "first", "second", etc. The skilled in the art can understand that "first", "second", etc. do not limit the quantity and execution order, and "first", "second", etc. also do not necessarily mean different.

[0060] In the present application, the reference to "one embodiment" or "some embodiments" means that the specific features, structures or characteristics described in connection with the embodiment are included in one or more embodiments of the present application. Therefore, the statements "in one embodiment", "in some embodiments", "in other some embodiments", "in another some embodiments" and the like appearing in different places in the specification are not necessarily all referring to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized. The terms "include", "contain", "have" and their variants mean "include but not limited to", unless otherwise specifically emphasized.

[0061] It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant regulations and standards of relevant countries and regions, and provide corresponding operation portal for user to choose authorization or refusal.

[0062] In recent years, with the progress of artificial intelligence technology, speech synthesis technology has also developed rapidly. Speech synthesis technology can convert text data into natural, fluent and expressive synthesized speech data.

[0063] Currently, in the process of speech synthesis based on a pre-trained speech synthesis model, further optimization training of the speech synthesis model is still needed based on speech data of a specific duration (usually 30 minutes to 60 minutes) of a target speaker, so as to ensure that the speech synthesis model can learn the voice characteristics of the target speaker, and then ensure that the synthesized speech data output by the speech synthesis model is close to the target speaker.

[0064] However, in the case where the speech data of a specific duration of the target speaker cannot be obtained, the pre-trained speech synthesis model cannot be optimized and trained, and thus the synthesized speech data output by the speech synthesis model deviates greatly from the voice characteristics of the target speaker.

[0065] In order to solve the above problems, the embodiment of the present application provides a speech synthesis method based on reference speech. The method can be applied to any speech synthesis scene. The method of the embodiment of the present application is described below. Figure 1 The flow chart of the speech synthesis method based on reference speech provided by the embodiment of the present application is shown in Figure 1 The specific processing process is as follows.

[0066] Step 101, obtaining target text data and target reference speech data of a target speaker.

[0067] In an embodiment, in order to reduce the deviation of the synthesized speech data output by the speech synthesis model from the voice characteristics of the target speaker in the case that the speech data of the target speaker for a specific duration (usually 30 minutes to 60 minutes) cannot be obtained. When the electronic device converts the target text data into the synthesized speech data corresponding to the target speaker by the speech synthesis model, the electronic device can also obtain the target reference speech data of the target speaker. Among them, compared with the specific duration of the speech data of the target speaker required by the traditional speech synthesis model, the voice duration of the target reference speech data of the target speaker in the embodiment of the application is shorter (such as 3 seconds to 5 seconds). The target speaker can be a real person or a virtual person, which is not limited in the embodiment of the application.

[0068] Step 102, input the target text data and the target reference speech data into the pre-trained speech synthesis model, and output the target synthesized speech data corresponding to the target speaker. Among them, the content of the target synthesized speech data is consistent with the target text data, and the pre-trained speech synthesis model is obtained by performing adversarial training on the initial speech synthesis model based on the SLM discriminator. The SLM discriminator is used to discriminate the SLM features of the sample synthesized speech data of the initial speech synthesis model and the SLM features of the sample adversarial speech data, and the SLM features are respectively generated by the SLM discriminator according to the sample synthesized speech data and the sample adversarial speech data.

[0069] In an embodiment, after the electronic device obtains the target text data and the target reference speech data, the electronic device can further input the target text data and the target reference speech data into the pre-trained speech synthesis model. Among them, the pre-trained speech synthesis model is obtained by performing adversarial training based on the speech language model. Correspondingly, the pre-trained speech synthesis model is obtained by performing adversarial training on the initial speech synthesis model based on the SLM (speech language model) discriminator. The SLM discriminator is used to discriminate the SLM features of the sample synthesized speech data of the initial speech synthesis model and the SLM features of the sample adversarial speech data, and the SLM features are respectively generated by the SLM discriminator according to the sample synthesized speech data and the sample adversarial speech data. Since the speech synthesis model converts the target text data into the synthesized speech data corresponding to the target speaker. Therefore, the content of the target synthesized speech data is consistent with the target text data.

[0070] It should be noted that the structure of the speech synthesis model, the process of the speech synthesis model outputting the target synthesized speech data corresponding to the target speaker, and the adversarial training process of the speech synthesis model will be described in detail in the subsequent embodiments of the application, which will not be described here.

[0071] In the embodiments of the present application, the traditional discriminator usually performs adversarial training based on the Mel-spectrogram features of the sample synthesized speech data and the sample adversarial speech data. The Mel-spectrogram features are obtained after Fourier transformation based on the speech data, do not need to be generated by a model, and can only represent the waveform features of the speech data. The SLM features are generated by the SLM model in the SLM discriminator, can evaluate the speech data by imitating human perception, and represent more abundant content. Therefore, in the process of adversarial training of the speech synthesis model, taking the SLM features as the basis for discrimination can make the synthesized speech data generated by the speech synthesis model after adversarial training more ideal.

[0072] In some embodiments, Figure 2 A structural schematic diagram of a speech synthesis model provided in the embodiments of the present application is shown in FIG. 1. Figure 2 As shown in FIG. 1, the speech synthesis model includes a phoneme encoder, a text encoder, a text style encoder, an audio style encoder, a variance adapter, and a text decoder.

[0073] The phoneme encoder is configured to extract phoneme features from input text data. The text data is a written form of language content, which can represent direct meaning, grammatical structure, and context association of the language. The phoneme features are actual pronunciation phoneme features converted from the text data, which can represent pronunciation features and acoustic performance of the language. The phoneme encoder can convert each word in the text data into its corresponding phoneme features based on the text data and a phoneme dictionary, thereby obtaining phoneme features corresponding to the text data. The phoneme dictionary can be a preset dictionary library. For example, the phoneme dictionary can be a resource library constructed according to a target language, or the phoneme dictionary can be an existing phoneme dictionary library, which is not limited in the embodiments of the present application.

[0074] The text encoder is configured to convert the phoneme features into text features. The text features can represent semantic content of the text data. The semantic content includes meaning of words, structure of sentences, and meaning of contexts, etc. A feed-forward transformer block is used as a basic structure of the text encoder in the embodiments of the present application, and the text encoder includes a one-dimensional self-attention layer and a one-dimensional convolution layer.

[0075] The text style encoder is configured to extract text style features from the input phoneme features. The text style features can represent style features such as intonation, stress, prosody, and emotion of the text data. The text style encoder mainly includes a diffusion denoising model. In the training process, the diffusion denoising model gradually transforms the original data into a Gaussian distribution through a diffusion process to learn the relationship between the original data and the noisy data. In the inference process, the diffusion denoising model restores the original data from the Gaussian distribution through a reverse process.

[0076] Specifically, in the training process of the diffusion denoising model, first, a corresponding speech data set (which is a general speech data set, not a speech data set of the target speaker) is selected to generate phoneme features for training, and then the phoneme features are gradually added with Gaussian noise in an iterative manner until the phoneme features are converted into a pure Gaussian noise vector. Assuming that the training word-level phoneme feature is y0, the iteration number is t, and the pure Gaussian noise vector after adding noise is y t , then y1=k1*y0, y2=k2*y1, y3=k3*y2…y t t t -1. Wherein k represents variance, k t represents the variance value corresponding to different iteration numbers t, and k t can be calculated in advance based on the standard Gaussian distribution, that is, for any t, a certain k t can be directly calculated.

[0077] On this basis, a t is randomly selected, and the corresponding k t and y t can be determined through the above calculation process. y0 and t are used as the training input of the diffusion denoising model, and y t is used as the training output of the diffusion denoising model, so as to train the diffusion denoising model, that is, to make the diffusion denoising model learn the relationship between the speech data and the noisy speech data. At the same time, during the training of the diffusion denoising model, the loss function changes greatly in the early stage of training, so the random selection of t can effectively avoid the diffusion denoising model from focusing too much on the information of a certain time period in the early stage of training, and ignoring the information of other time periods. Based on this, the random distribution of t should be as uniform as possible.

[0078] After sufficient speech data is used to complete the training, the parameters of the diffusion denoising model are processed in reverse, so that the diffusion denoising model can recover the original data from the Gaussian distribution through the reverse process and obtain the corresponding style features. Specifically, the phoneme features corresponding to the text data are input into the diffusion denoising model, and the diffusion denoising model outputs the text style features of the text data. Through the diffusion denoising model, the text style features represented by the phoneme features can be learned, and the speech synthesis effect of the speech synthesis model in the training and use process is more ideal.

[0079] The audio style encoder is used to extract audio style features from the input reference audio mel spectrum. The audio style features can represent the acoustic characteristics of the target speaker. The audio style encoder can be composed of four residual blocks and an average pooling layer.

[0080] ​​A variance adapter is used to add variance information to text features. This variance information can include at least one or more of duration, pitch, and energy. The variance adapter can provide sufficient information for predicting variant speech in a one-to-many mapping problem in a speech synthesis model. During variance adapter training, the ground truth values ​​of time, pitch, and energy extracted from real speech are added to the text features as input to predict the target synthesized speech. Simultaneously, independent variance predictions are performed for duration, pitch, and energy to generate the target synthesized speech during inference.

[0081] Figure 3 This is a schematic diagram of a variance adapter provided in an embodiment of this application. Figure 3 As shown, the variance adapter includes at least one or more of a time predictor, a pitch predictor, and an energy predictor. Within the variance adapter, the duration predictor, pitch predictor, and energy predictor have similar model structures. Using text features as input, the variance of each phoneme (duration) or frame (pitch and energy) is predicted using mean squared error (MSE) loss. Figure 4 This is a schematic diagram of a predictor provided in an embodiment of this application. Taking a time predictor as an example, as... Figure 4 As shown, the temporal predictor consists of a two-layer one-dimensional convolutional network (Cove1D) with ReLU activation and an additional linear layer. Each convolutional network includes a normalization layer (LN) and a dropout layer. The linear layer is used to project the hidden state into the output sequence. For the duration predictor, the output sequence is the length of each phoneme in the logarithmic domain. For the pitch predictor, the output sequence is the frame-level fundamental frequency sequence, and for the energy predictor, the output sequence is the energy sequence for each Mel spectrogram frame. All predictors share the same model structure but do not share model parameters.

[0082] The text features, text style features, and audio style features output by the text encoder, text style encoder, and audio style encoder are aggregated and then input into the variance adapter. Accordingly, the variance adapter adds different variance information, such as duration, pitch, and energy, to the text features.

[0083] A text decoder is used to convert text features with added variance into synthesized speech output.

[0084] In some embodiments, based on the structure of the speech synthesis model described above, such as Figure 5 As shown, the electronic device inputs the target text data and target reference speech data into a pre-trained speech synthesis model, and outputs the target synthesized speech data corresponding to the target speaker. The processing procedure is as follows:

[0085] Step 501, input the target text data into the phoneme encoder, and output the phoneme features of the target text data.

[0086] In an implementation, after the electronic device obtains the target text data, the electronic device can input the target text data into the phoneme encoder. Correspondingly, the phoneme encoder can output the phoneme features of the target text data. The phoneme features can represent the pronunciation features and acoustic performance of the target text data.

[0087] Step 502, input the phoneme features into the text encoder and the text style encoder respectively, and output the first text features and the first text style features of the target text data.

[0088] In an implementation, after the electronic device obtains the phoneme features of the target text data, the electronic device can combine the phoneme features and the phoneme position encoding to obtain a combination result. Then, the electronic device can input the combination result into the text encoder. Correspondingly, the text encoder can output the first text features of the target text data. The first text features can represent the semantic content of the target text data, which can include the meaning of a word, the structure of a sentence, and the meaning of a context, etc. Meanwhile, the electronic device can also input the phoneme features into the text style encoder. Correspondingly, the text style encoder can output the first text style features of the target text data. The first text style features can represent the style features of the target text data, such as intonation, emphasis, prosody, emotion, etc.

[0089] Step 503, input the target reference speech data into the audio style encoder, and output the audio style features of the target reference speech data.

[0090] In an implementation, after the electronic device obtains the target reference speech data of the target speaker, the electronic device can further input the target reference speech data into the audio style encoder. Correspondingly, the audio style encoder can output the audio style features of the target reference speech data. The audio style features can represent the acoustic features of the target speaker. Optionally, after the electronic device obtains the target reference speech data, the electronic device can determine the mel spectrum of the target reference speech data, and input the mel spectrum of the target reference speech data into the audio style encoder.

[0091] Step 504, input the first text features, the first text style features, and the audio style features into the variance adapter, and output the second text features of the target text data. The second text features are the text features obtained by adding the target style features to the first text features, and the target style features are the style features predicted by the variance adapter based on the input first text style features and the audio style features.

[0092] In one implementation, after obtaining a first text feature, a first text style feature, and an audio style feature, the electronic device can merge the first text feature, the first text style feature, and the audio style feature to obtain a merged result. Then, the electronic device can input the merged result into a variance adapter. Accordingly, the variance adapter can predict a target style feature based on the input first text style feature and audio style feature using a predictor. This target style feature can characterize the target text data and the acoustic style features of the target speaker. The predictor can include at least one or more of a time predictor, a pitch predictor, and an energy predictor. The target style feature can also include at least one or more of a time feature, a pitch feature, and an energy feature. Then, the variance adapter can add the target style feature to the first text feature, outputting a second text feature of the target text data.

[0093] Step 505: Input the second text feature into the text decoder and output the target synthesized speech data corresponding to the target speaker.

[0094] In one implementation, after obtaining the second text feature, the electronic device can further merge the second text feature and the phoneme position code to obtain a merged result. The electronic device can then input the merged result into a text decoder. Correspondingly, the text decoder can output the target synthesized speech data corresponding to the target speaker.

[0095] In some embodiments, Figure 6 This is a schematic diagram of the structure of a text style encoder provided in an embodiment of this application. Figure 6 As shown, the text style encoder includes an encoding module, a noise module, a diffusion noise reduction module, and a decoding module. The encoding module compresses the input phoneme features. This module consists of three convolutional layers and one bidirectional long short-term memory (LSTM) layer. The noise module outputs random Gaussian noise. The diffusion noise reduction module extracts text style features from the input phoneme features. The decoding module decompresses the text style features. This module consists of one fully connected layer.

[0096] Accordingly, the electronic device inputs phoneme features into the text style encoder, and the processing of the first text style feature of the output target text data is as follows:

[0097] Step 1: Input the phoneme features into the encoding module and output the dimensionality-reduced phoneme features.

[0098] In an embodiment, after obtaining the phoneme features of the target text data, the electronic device can input the phoneme features into the encoding module. Correspondingly, the encoding module can perform dimension reduction (compression) processing on the phoneme features and output the dimension-reduced phoneme features.

[0099] In step two, the dimension-reduced phoneme features and noise data are input into the diffusion denoising module to output the second text style features of the target text data. The noise data is generated by the noise module.

[0100] In an embodiment, after obtaining the dimension-reduced phoneme features, the electronic device can further merge the noise data generated by the noise module with the dimension-reduced phoneme features to obtain a merging result. The noise data can be random Gaussian noise data, which is not limited in the embodiments of the present application. Then, the electronic device can input the merging result into the diffusion denoising module. Correspondingly, the diffusion denoising module can output the second text style features of the target text data.

[0101] In step three, the second text style features are input into the decoding module to output the first text style features of the target text data.

[0102] In an embodiment, after obtaining the second text style features of the target text data, the electronic device can further input the second text style features into the decoding module. Correspondingly, the decoding module can perform decompression processing on the second text style features and output the decompressed second text style features, which are the first text style features of the target text data.

[0103] In the embodiments of the present application, the encoding module and the decoding module are added to the text style encoder to perform compression processing and decompression processing on the features, thereby greatly improving the rate of the text style encoder outputting the text style features.

[0104] In some embodiments, Figure 7 A flowchart of a training method of a speech synthesis model according to an embodiment of the present application is shown in FIG. 7.

[0105] As shown in FIG. 7, the electronic device further performs the following steps: Figure 7

[0106] In step 701, at least one training sample is obtained. The training sample includes sample text data, sample reference speech data, and sample adversarial speech data.

[0107] ​In one implementation, when an electronic device needs to perform adversarial training on an initial speech synthesis model, the electronic device can acquire at least one training sample. The training sample includes sample text data, sample reference speech data, and sample adversarial speech data. The sample text data and sample reference speech data have the same content; the sample reference speech data and sample adversarial speech data belong to the same speaker, and the sample reference speech data and sample adversarial speech data may be the same or different; this application embodiment does not limit this.

[0108] Step 702: Input the sample text data and sample reference speech data into the initial speech synthesis model, and output the sample synthesized speech data corresponding to the sample text data.

[0109] In one implementation, after acquiring at least one training sample, the electronic device can further input the sample text data and sample reference speech data into an initial speech synthesis model. Correspondingly, the initial speech synthesis model can output the sample synthesized speech data corresponding to the sample text data.

[0110] Step 703: Input the sample synthesized speech data and sample adversarial speech data into the first discriminator and output the first discrimination result.

[0111] In one implementation, reference Figure 2 As shown, in this embodiment of the application, a first discriminator performs waveform discrimination on the sample synthesized speech data and sample adversarial speech data to conduct the first adversarial training on the initial speech synthesis model. Therefore, after obtaining the sample synthesized speech data, the electronic device can input the sample synthesized speech data and sample adversarial speech data into the first discriminator. Correspondingly, the first discriminator can output a first discrimination result. The first discrimination result includes the probability that the sample synthesized speech data is real speech data and the probability that the sample adversarial speech data is real speech data.

[0112] Step 704: Input the sample synthesized speech data and sample adversarial speech data into the second discriminator and output the second discrimination result. The second discriminator is an SLM discriminator.

[0113] In one implementation, reference Figure 2As shown, the second discriminator (SLM discriminator) is used to discriminate the SLM features of the sample synthesized speech data and the sample adversarial speech data, so as to perform second adversarial training on the initial speech synthesis model. Therefore, after obtaining the sample synthesized speech data, the electronic device can input the sample synthesized speech data and the sample adversarial speech data into the second discriminator. Correspondingly, the second discriminator can output a second discrimination result. The second discrimination result includes the probability that the SLM feature of the sample synthesized speech data is real speech data and the probability that the SLM feature of the sample adversarial speech data is real speech data.

[0114] At step 705, the initial speech synthesis model is subjected to adversarial training according to the first discrimination result and the second discrimination result, to obtain a trained speech synthesis model.

[0115] In an implementation, after obtaining the first discrimination result and the second discrimination result, the electronic device can perform adversarial training on the initial speech synthesis model according to the first discrimination result and the second discrimination result, to obtain a trained speech synthesis model. The first discrimination of the first discriminator and the second discrimination of the SLM discriminator are independent of each other. That is, the inputs of the first discriminator and the SLM discriminator are both the sample synthesized speech data output by the initial speech synthesis model, and the first discriminator and the SLM discriminator perform adversarial training on the initial speech synthesis model in their respective discriminations.

[0116] In some embodiments, Figure 8 A flowchart of a speech synthesis model training method according to an embodiment of the present application is shown in FIG. 6. As shown in FIG. 6, the speech synthesis model training method includes the following steps. Figure 2 As shown, the second discriminator includes an SLM model and a discrimination head. The SLM discriminator is used to discriminate the SLM features of the sample synthesized speech data and the sample adversarial speech data, so as to perform first adversarial training on the initial speech synthesis model. The SLM model can be a HuBERT large model or other types of models, which are not limited in the present application. The HuBERT large model is trained by one million hours of WenetSpeech (a multi-domain Chinese speech recognition dataset) Chinese data, and knowledge is uniquely transferred from the SLM encoder to the generation task through adversarial training. Since the number of parameters of HuBERT is much larger than that of the speech synthesis model, in order to avoid its function being too powerful, the parameters (denoted as W) of the HuBERT model need to be fixed in the actual training process. The discrimination head can be a convolutional neural network or other types of networks, which are not limited in the present application. The SLM feature can mimic human perception to evaluate the quality of synthesized speech.

[0117] In the embodiment of the present application, the traditional discriminator (i.e., the first discriminator) generally performs adversarial training based on the mel-spectral features of the sample synthesized speech data and the sample adversarial speech data. The mel-spectral features are obtained after Fourier transformation based on the speech data, do not need to be generated by a model, and can only represent the waveform features of the speech data. The SLM features are generated by the SLM model in the SLM discriminator, can evaluate the speech data by simulating human perception, and represent more abundant content. Therefore, in the process of adversarial training of the speech synthesis model, taking the SLM features as the basis for discrimination can make the synthesized speech data generated by the speech synthesis model after adversarial training more ideal.

[0118] As shown in FIG. 8, the electronic device inputs the sample synthesized speech data and the sample adversarial speech data into the second discriminator, and the processing process of outputting the second discrimination result is as follows: Figure 8

[0119] Step 801, input the sample synthesized speech data and the sample adversarial speech data into the SLM model, and output the first SLM feature of the sample synthesized speech data and the second SLM feature of the sample adversarial speech data.

[0120] In an implementation manner, after obtaining the sample synthesized speech data, the electronic device can input the sample synthesized speech data and the sample adversarial speech data into the SLM model of the SLM discriminator. Correspondingly, the SLM model can output the first SLM feature of the sample synthesized speech data and the second SLM feature of the sample adversarial speech data.

[0121] Step 802, input the first SLM feature and the second SLM feature into the discrimination head, and output the second discrimination result.

[0122] In an implementation manner, after obtaining the first SLM feature and the second SLM feature, the electronic device can further input the first SLM feature and the second SLM feature into the discrimination head of the SLM discriminator. Correspondingly, the discrimination head can output the second discrimination result. The second discrimination result includes the probability that the first SLM feature of the sample synthesized speech data is real speech data and the probability that the SLM feature of the sample adversarial speech data is real speech data.

[0123] For the first discriminator, the specific adversarial training process is as follows: based on the first discrimination result and the first loss function of the first discriminator, the electronic device can determine the generator (i.e., the speech synthesis model) loss and the discriminator (i.e., the first discriminator) loss, and perform adversarial training on the generator and the discriminator based on the generator loss and the discriminator loss. The loss function of the first discriminator is as follows:

[0124] ​The first one is a least-squares loss function for adversarial training, which is formulated as follows:

[0125] L adv (D) = E (y,z) [(D(y) - 1) 2 + (D(G(z))) 2 ]

[0126] L adv (G) = E z [(D(G(z)) - 1) 2 ]

[0127] where L adv (D) denotes the loss of the discriminator D, E (y,z) denotes the least-squares loss function calculation, D(y) denotes the probability that the real waveform y is identified as a real waveform by the discriminator D, G(z) denotes the fake waveform output by the generator G input with the text z, and D(G(z)) denotes the probability that the fake waveform is identified as a real waveform by the discriminator D; L adv (G) denotes the loss of the generator G, and E z denotes the least-squares loss function calculation.

[0128] The second one is a feature-matching loss specially applied to the generator G, which can be regarded as a reconstruction loss and is used to constrain the output of the intermediate layers of the discriminator, and is formulated as follows:

[0129]

[0130] where L fm (G) denotes the feature-matching loss of the generator G, E (y,z) denotes the least-squares loss function calculation, T denotes the number of layers of the discriminator D, D l denotes the output feature map of the l-th layer of the discriminator, N l denotes the number of feature maps of the l-th layer, D l (y) denotes the probability that the real waveform y is identified as a real waveform by the l-th layer of the discriminator, G(z) denotes the fake waveform output by the generator G input with the text z, and D l (G(z)) denotes the probability that the fake waveform is identified as a real waveform by the l-th layer of the discriminator.

[0131] For the SLM discriminator, the specific adversarial training process is as follows: the electronic device can determine the generator (i.e., the speech synthesis model) loss and the discriminator (i.e., the second discriminator) loss based on the second discrimination result and the second loss function of the second discriminator, and perform adversarial training on the generator and the discriminator based on the generator loss and the discriminator loss. Wherein, for the SLM discriminator D SLM The loss function of the SLM discriminator D

[0132]

[0133] Wherein, L slm represents the loss of the SLM, represents the loss minimization of the generator G, represents the loss maximization of the SLM discriminator D, E x and E t represent the least square loss function calculation, D SLM (x) represents the probability that the real waveform is discriminated as a real waveform by the SLM discriminator D, D SLM (G(t)) represents the probability that the false waveform generated by inputting the text t into the generator G is discriminated as a real waveform by the SLM discriminator D, D SLM (m) represents the optimization of the discriminator, m can be a real waveform or a false waveform, represents the data distribution of the real waveform, represents the distribution of the false waveform, represents the density function of the real waveform, represents the density function of the false waveform, if the generator G is optimal, which means that when converging, the data distribution of the real waveform and the data distribution of the false waveform can match in the SLM feature space, can effectively imitate human perception, and thus realize the synthesis of human-like speech, that is, the generated speech is more close to the real human speech, and it is difficult to distinguish between true and false.

[0134] In some embodiments, the electronic device can also perform different focused adversarial training based on the emotional type of the training sample, and the electronic device further performs the following steps:

[0135] Step one, obtaining the emotional type corresponding to the training sample.

[0136] In an implementation, the electronic device can obtain the emotional type corresponding to the training sample. Wherein, the emotional type can include an emotional flat type and an emotional strong type, or other types, which are not limited by the embodiments of the present application.

[0137] Step two, according to the emotion type, determine the first weight corresponding to the first discriminator and the second weight corresponding to the second discriminator.

[0138] In an implementation, the embodiment of the present application adopts the first discriminator and the SLM discriminator for comprehensive adversarial training, so that the first discriminator and the SLM discriminator focus on different emotion types, that is, in the training process, the first discriminator and the SLM discriminator are given different weights for different emotion types. For example, for the training sample of the emotion type of flat, the weight of the first discriminator is higher, and the weight of the SLM discriminator is lower. For the training sample of the emotion type of strong, the weight of the SLM discriminator is higher, and the weight of the first discriminator is lower. After the electronic device obtains the emotion type corresponding to the training sample, the first weight corresponding to the first discriminator and the second weight corresponding to the second discriminator can be determined according to the emotion type.

[0139] Correspondingly, the electronic device performs adversarial training on the initial speech synthesis model according to the first discrimination result and the second discrimination result to obtain the trained speech synthesis model. The processing process of the trained speech synthesis model is as follows: performing adversarial training on the initial speech synthesis model according to the first discrimination result, the second discrimination result, the first weight and the second weight to obtain the trained speech synthesis model.

[0140] In an implementation, the electronic device determines the first weight corresponding to the first discriminator and the second weight corresponding to the second discriminator, and can further perform adversarial training on the initial speech synthesis model according to the first discrimination result, the second discrimination result, the first weight and the second weight to obtain the trained speech synthesis model. For example, the electronic device determines that the parameter adjustment amount of the speech synthesis model based on the first discrimination result is △T1, and then the electronic device can determine the final parameter adjustment amount of the speech synthesis model based on the first weight a as a*△T1. Similarly, the electronic device determines that the parameter adjustment amount of the speech synthesis model based on the second discrimination result is △T2, and then the electronic device can determine the final parameter adjustment amount of the speech synthesis model based on the second weight b as b*△T2.

[0141] In some embodiments, the electronic device obtains the emotion type corresponding to the training sample in the following processing process:

[0142] Step one, based on the emotion clustering model, clustering the training sample to obtain at least one training sample set corresponding to the emotion type.

[0143] In an implementation, the electronic device can cluster the training samples based on an emotion clustering model to obtain a training sample set corresponding to each emotion type. The emotion clustering model can be a BERT model (Bidirectional Encoder Representations from Transformers, a pre-training language model based on a Transformer architecture), or other types of models, which are not limited by the embodiments of the present application. The emotion clustering model can cluster the training samples based on the intensity of emotion expressed by the sample text data and / or the intensity of emotion expressed by the sample reference speech data to obtain a training sample set corresponding to each emotion type.

[0144] In step two, the emotion type corresponding to the training sample set is determined as the emotion type of the training samples included in the training sample set.

[0145] In an implementation, after the electronic device obtains a training sample set corresponding to each emotion type, the electronic device can determine the emotion type corresponding to the training sample set as the emotion type of the training samples included in the training sample set.

[0146] In the speech synthesis method based on reference speech provided by the embodiments of the present application, the electronic device obtains target text data and target reference speech data of a target speaker. Then, the electronic device inputs the target text data and the target reference speech data into a pre-trained speech synthesis model to output target synthesis speech data corresponding to the target speaker. The target synthesis speech data is consistent with the content of the target text data, and the pre-trained speech synthesis model is obtained by performing adversarial training on an initial speech synthesis model based on an SLM discriminator. The SLM discriminator is used to discriminate between the SLM features of the sample synthesis speech data of the initial speech synthesis model and the SLM features of the sample adversarial speech data. In this way, during the training of the speech synthesis model, the speech synthesis model is adversarially trained by the SLM discriminator. Subsequently, during the use of the speech synthesis model, only the reference speech data of the target speaker with a short duration needs to be obtained, and the specific duration of the speech data of the target speaker does not need to be obtained, which can ensure that the synthesis speech data finally output by the speech synthesis model is consistent with the voice features of the target speaker.

[0147] Figure 9 A structural schematic diagram of a speech synthesis device based on reference speech provided by the embodiments of the present application is shown in FIG. 9. As shown in FIG. 9, the speech synthesis device based on reference speech 900 includes a first obtaining module 910 and a first output module 920. Wherein: Figure 9 The first obtaining module 910 is configured to obtain target text data and target reference speech data of a target speaker.

[0148] The first output module 920 is configured to input the target text data and the target reference speech data into a pre-trained speech synthesis model to output target synthesis speech data corresponding to the target speaker.

[0149] The first output module 920 is configured to input the target text data and the target reference speech data into a pre-trained speech synthesis model, and output target synthesized speech data corresponding to the target speaker, the target synthesized speech data being consistent with the content of the target text data, the pre-trained speech synthesis model being obtained by performing adversarial training on an initial speech synthesis model based on a speech language model (SLM) discriminator, and the SLM discriminator being configured to distinguish SLM features of sample synthesized speech data of the initial speech synthesis model and SLM features of sample adversarial speech data.

[0150] In some embodiments, the speech synthesis model comprises a phoneme encoder, a text encoder, a text style encoder, an audio style encoder, a variance adapter, and a text decoder; and the first output module 920 is specifically configured to:

[0151] input the target text data into the phoneme encoder, and output phoneme features of the target text data;

[0152] input the phoneme features into the text encoder and the text style encoder respectively, and output first text features and first text style features of the target text data respectively;

[0153] input the target reference speech data into the audio style encoder, and output audio style features of the target reference speech data;

[0154] input the first text features, the first text style features, and the audio style features into the variance adapter, and output second text features of the target text data; the second text features are text features obtained by adding target style features to the first text features, and the target style features are style features predicted by the variance adapter based on the input first text style features and the audio style features;

[0155] input the second text features into the text decoder, and output the target synthesized speech data corresponding to the target speaker.

[0156] In some embodiments, the text style encoder comprises an encoding module, a noise module, a diffusion denoising module, and a decoding module; and the first output module 920 is specifically configured to:

[0157] input the phoneme features into the encoding module, and output reduced-dimension phoneme features;

[0158] input the reduced-dimension phoneme features and noise data into the diffusion denoising module, and output second text style features of the target text data; the noise data is generated by the noise module;

[0159] The second text style feature is input into the decoding module, and the first text style feature of the target text data is output.

[0160] In some embodiments, the apparatus further includes:

[0161] The second obtaining module is configured to obtain at least one training sample, the training sample including sample text data, sample reference speech data, and sample adversarial speech data.

[0162] The second output module is configured to input the sample text data and the sample reference speech data into an initial speech synthesis model, and output sample synthesized speech data corresponding to the sample text data.

[0163] The third output module is configured to input the sample synthesized speech data and the sample adversarial speech data into the first discriminator, and output a first discrimination result.

[0164] The fourth output module is configured to input the sample synthesized speech data and the sample adversarial speech data into the second discriminator, and output a second discrimination result, the second discriminator being an SLM discriminator.

[0165] The training module is configured to perform adversarial training on the initial speech synthesis model according to the first discrimination result and the second discrimination result, to obtain a trained speech synthesis model.

[0166] In some embodiments, the second discriminator includes an SLM model and a discrimination head; and the fourth output module is specifically configured to:

[0167] input the sample synthesized speech data and the sample adversarial speech data into the SLM model, and output a first SLM feature of the sample synthesized speech data and a second SLM feature of the sample adversarial speech data;

[0168] input the first SLM feature and the second SLM feature into the discrimination head, and output the second discrimination result.

[0169] In some embodiments, a loss function corresponding to the second discriminator is:

[0170]

[0171] wherein D SLM represents the SLM discriminator, C represents the discrimination head in the SLM discriminator, W represents the SLM model in the SLM discriminator, L slm represents the loss of the SLM, represents the loss minimization of the generator G, represents the loss maximization of the SLM discriminator D, E x and E t represent the least square loss function calculation, D SLM(x) represents a probability that a real waveform is discriminated as a real waveform by the SLM discriminator D, D SLM (G(t)) represents a probability that a fake waveform generated by the text data t input generator G is discriminated as a real waveform by the SLM discriminator D, D SLM (m) represents an optimization discriminator, m represents a real waveform or a fake waveform, represents a data distribution of a real waveform, represents a distribution that is a fake waveform, represents a density function of a real waveform, represents a density function of a fake waveform.

[0172] In some embodiments, the apparatus further includes:

[0173] a third obtaining module, configured to obtain a sentiment type corresponding to the training sample;

[0174] a determining module, configured to determine, according to the sentiment type, a first weight corresponding to the first discriminator and a second weight corresponding to the second discriminator;

[0175] performing adversarial training on the initial speech synthesis model according to the first discrimination result and the second discrimination result to obtain a trained speech synthesis model, including:

[0176] performing adversarial training on the initial speech synthesis model according to the first discrimination result, the second discrimination result, the first weight and the second weight to obtain the trained speech synthesis model.

[0177] In some embodiments, the third obtaining module is specifically configured to:

[0178] perform clustering on the training samples based on a sentiment clustering model to obtain a training sample set corresponding to at least one sentiment type;

[0179] determine, as a sentiment type of a training sample included in the training sample set, a sentiment type corresponding to the training sample set.

[0180] Figure 10 A schematic diagram of an electronic device provided by an embodiment of the present application. In some embodiments, the electronic device includes one or more processors and a memory. The memory is configured to store one or more programs. Wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the speech synthesis method based on reference speech in the above-mentioned embodiments.

[0181] As Figure 10As shown, the electronic device 1000 includes a processor 1001 and a memory 1002. The electronic device 1000 may, for example, also include a communications interface 1003 and a communications bus 1004.

[0182] The processor 1001, the memory 1002, and the communications interface 1003 communicate with each other via the communications bus 1004. The communications interface 1003 is configured to communicate with network elements such as clients or other servers.

[0183] In some embodiments, the processor 1001 is configured to execute the program 1005, and specifically can execute the steps of the above-described voice synthesis method based on a reference voice. Specifically, the program 1005 can include program code including computer-executable instructions.

[0184] The processor 1001 may, for example, be a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement one or more embodiments of the present application. The electronic device 1000 can include one or more processors, which can be the same type of processor, such as one or more CPUs, or different types of processors, such as one or more CPUs and one or more ASICs.

[0185] In some embodiments, the memory 1002 is configured to store the program 1005. The memory 1002 can include a high-speed RAM memory, and can also include a non-volatile memory (NVM), such as at least one disk memory.

[0186] The program 1005 may, for example, be invoked by the processor 1001 to cause the electronic device 1000 to perform the operations of the voice synthesis method based on a reference voice.

[0187] The present embodiments provide a computer-readable storage medium storing at least one executable instruction, which, when executed on the electronic device 1000, causes the electronic device 1000 to perform the voice synthesis method based on a reference voice in the above-described embodiments.

[0188] The executable instruction may, for example, be configured to cause the electronic device 1000 to perform the voice synthesis method based on a reference voice.

[0189] For example, the computer readable storage medium can be a Read-Only Memory (ROM), a Random Access Memory (RAM), a Compact Disc Read-Only Memory (CD-ROM), a magnetic tape, a floppy disk, and an optical data storage device, etc.

[0190] In some embodiments, the embodiments of the present application provide a computer program product, which comprises a computer program stored on a non-transitory computer readable storage medium, the computer program comprising program instructions which, when executed by a computer, cause the computer to perform the speech synthesis method based on reference speech according to any of the embodiments described above.

[0191] In some embodiments, the embodiments of the present application also provide a computer program which, when executed by a processor, can implement the speech synthesis method based on reference speech according to any of the embodiments described above.

[0192] The speech synthesis device based on reference speech, the electronic device, the computer readable storage medium, the computer program product and the computer program provided by the embodiments of the present application can achieve the beneficial effects of the speech synthesis method based on reference speech provided above, which will not be described here again.

[0193] It should be noted that, in the application, the relationship terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between the entities or operations. Moreover, the terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. Without more limitations, the element defined by the statement "including a" does not exclude the presence of another identical element in the process, method, article or device including the element.

[0194] Each of the embodiments in the specification is described in a relevant manner, and the same or similar parts between each of the embodiments can be referred to each other. Each of the embodiments focuses on the difference from other embodiments. Especially, for the device embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the part of the method embodiments.

[0195] The logic and / or steps represented in the flow diagrams or otherwise described herein, for example, can be considered as a sequence of executable instructions, and can be embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor-containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions, or a combination thereof.

[0196] For purposes of this specification, a "computer-readable medium" can be any apparatus that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device.

[0197] More specific examples (a non-exhaustive list) of the computer-readable medium include the following: an electrical connection (electronic) having one or more wires, a portable computer diskette (magnetic), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, and a portable compact disc read-only memory (CDROM).

[0198] Additionally, the computer-readable medium can even be paper or another suitable medium upon which the program is printed, as the program can be electronically captured, for example, by optically scanning the paper or other suitable medium, then electronically capturing the program, and then storing it in computer memory. It should be understood that parts of this application can be implemented in hardware, software, firmware, or combinations thereof.

[0199] In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, and as in another embodiment, any of the following technologies known in the art, or a combination thereof, can be used: discrete logic circuitry having logic gates for implementing logic functions upon data signals, application specific integrated circuits having appropriate combinational logic gates, programmable gate arrays (PGA), field programmable gate arrays (FPGA), and so forth.

[0200] The embodiments of the application described above do not represent all possible implementations of the application.

Claims

1. A speech synthesis method based on a reference speech, characterized by, The method comprises: obtaining target text data and target reference speech data of a target speaker; inputting the target text data and the target reference speech data into a pre-trained speech synthesis model to output target synthesized speech data corresponding to the target speaker, the target synthesized speech data being consistent with the content of the target text data, the pre-trained speech synthesis model being obtained by performing adversarial training on an initial speech synthesis model based on a speech language model (SLM) discriminator, the SLM discriminator being configured to distinguish SLM features of sample synthesized speech data of the initial speech synthesis model from SLM features of sample adversarial speech data, the SLM features being generated by the SLM discriminator based on the sample synthesized speech data and the sample adversarial speech data respectively; the speech synthesis model comprises a phoneme encoder, a text encoder, a text style encoder, an audio style encoder, a variance adapter, and a text decoder; the inputting the target text data and the target reference speech data into the pre-trained speech synthesis model to output the target synthesized speech data corresponding to the target speaker comprises: inputting the target text data into the phoneme encoder to output phoneme features of the target text data; inputting the phoneme features into the text encoder and the text style encoder respectively to output first text features and first text style features of the target text data; inputting the target reference speech data into the audio style encoder to output audio style features of the target reference speech data; inputting the first text features, the first text style features, and the audio style features into the variance adapter to output second text features of the target text data; wherein the second text features are text features obtained by adding target style features to the first text features, the target style features being style features predicted by the variance adapter based on the input first text style features and the audio style features; inputting the second text features into the text decoder to output the target synthesized speech data corresponding to the target speaker.

2. The method of claim 1, wherein, The text style encoder comprises an encoding module, a noise module, a diffusion noise reduction module, and a decoding module; the inputting the phoneme features into the text style encoder to output the first text style features of the target text data comprises: inputting the phoneme features into the encoding module to output reduced-dimension phoneme features; inputting the reduced-dimension phoneme features and noise data into the diffusion noise reduction module to output second text style features of the target text data; the noise data is generated by the noise module; inputting the second text style features into the decoding module to output the first text style features of the target text data.

3. The method of claim 1, wherein, The method further comprises: obtaining at least one training sample, the training sample comprising sample text data, sample reference speech data, and sample adversarial speech data; input the sample text data and the sample reference speech data into the initial speech synthesis model, and output sample synthesized speech data corresponding to the sample text data; input the sample synthesized speech data and the sample adversarial speech data into a first discriminator, and output a first discrimination result; input the sample synthesized speech data and the sample adversarial speech data into a second discriminator, and output a second discrimination result, the second discriminator being an SLM discriminator; perform adversarial training on the initial speech synthesis model according to the first discrimination result and the second discrimination result, to obtain a trained speech synthesis model.

4. The method of claim 3, wherein, the second discriminator comprises an SLM model and a discrimination head; the inputting of the sample synthesized speech data and the sample adversarial speech data into the second discriminator to output the second discrimination result comprises: input the sample synthesized speech data and the sample adversarial speech data into the SLM model, and output first SLM features of the sample synthesized speech data and second SLM features of the sample adversarial speech data; input the first SLM features and the second SLM features into the discrimination head, and output the second discrimination result.

5. The method of claim 3, wherein, the loss function corresponding to the second discriminator is: ； ； wherein, denotes an SLM discriminator, , denotes a discriminative head in the SLM discriminator, denotes an SLM model in the SLM discriminator, denotes a loss of the SLM, denotes a loss minimization of the generator G, denotes a loss maximization of the SLM discriminator D, and denotes a least square loss function computation, denotes a probability that a real waveform is discriminated as a real waveform by the SLM discriminator D, denotes a probability that a fake waveform generated by the generator G inputting the text data t is discriminated as a real waveform by the SLM discriminator D, denotes an optimized discriminator, m denotes a real waveform or a fake waveform, denotes a data distribution of a real waveform, denotes a distribution that is a fake waveform, denotes a density function of a real waveform, denotes a density function of a fake waveform.

6. The method of claim 5, wherein, the method further comprises: obtain a sentiment type corresponding to the training sample; determine a first weight corresponding to the first discriminator and a second weight corresponding to the second discriminator according to the sentiment type; the adversarial training on the initial speech synthesis model according to the first discrimination result and the second discrimination result to obtain the trained speech synthesis model comprises: perform adversarial training on the initial speech synthesis model according to the first discrimination result, the second discrimination result, the first weight, and the second weight, to obtain the trained speech synthesis model.

7. The method of claim 6, wherein, the obtaining of the sentiment type corresponding to the training sample comprises: cluster the training sample based on a sentiment clustering model, to obtain a training sample set corresponding to at least one sentiment type; determine the sentiment type corresponding to the training sample set as the sentiment type of the training sample included in the training sample set.

8. A speech synthesis apparatus based on a reference speech, characterized by, the apparatus comprises: a first obtaining module configured to obtain target text data and target reference speech data of a target speaker; a first output module configured to input the target text data and the target reference speech data into a pre-trained speech synthesis model, and output target synthesized speech data corresponding to the target speaker, the target synthesized speech data being consistent with the content of the target text data, the pre-trained speech synthesis model being obtained by performing adversarial training on an initial speech synthesis model based on a speech language model (SLM) discriminator, the SLM discriminator being configured to discriminate SLM features of sample synthesized speech data of the initial speech synthesis model and SLM features of sample adversarial speech data, the SLM features being respectively generated by the SLM discriminator according to the sample synthesized speech data and the sample adversarial speech data; The speech synthesis model comprises a phoneme encoder, a text encoder, a text style encoder, an audio style encoder, a variance adapter, and a text decoder; the first output module is specifically configured to: input the target text data into the phoneme encoder to output phoneme features of the target text data; input the phoneme features into the text encoder and the text style encoder respectively to output first text features and first text style features of the target text data respectively; input the target reference speech data into the audio style encoder to output audio style features of the target reference speech data; input the first text features, the first text style features, and the audio style features into the variance adapter to output second text features of the target text data; wherein the second text features are text features obtained by adding target style features to the first text features, and the target style features are style features predicted by the variance adapter based on the input first text style features and the audio style features; and input the second text features into the text decoder to output target synthesized speech data corresponding to the target speaker.

9. An electronic device comprising: a processor; and a memory storing a program; wherein the program includes instructions that, when executed by the processor, cause the processor to perform the method according to any one of claims 1-7.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program, when executed by a processor, implements the method steps of any one of claims 1 to 7. The computer program, when executed by a processor, implements the method steps of any one of claims 1 to 7.

Images