Speech synthesis methods, devices, electronic devices and storage media

Abstract

This invention provides a speech synthesis method, apparatus, electronic device, and storage medium. The method includes: acquiring text to be synthesized; performing acoustic modeling based on the text to obtain speech features corresponding to the text, wherein the speech features and sample speech features reside in the same feature space, and the sample speech features are obtained by feature extraction from sample speech using an end-to-end speech synthesis model or a self-supervised speech feature extraction model; and reconstructing waveforms based on the speech features to obtain synthesized speech corresponding to the text. The method provided by this invention uses speech features residing in the same feature space as sample speech features extracted based on an end-to-end speech synthesis model or a self-supervised speech feature extraction model as a bridge between the acoustic model and the vocoder to achieve speech synthesis. This allows for the reference of richer acoustic information during the speech synthesis process, thereby improving the speech quality of the synthesized speech.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a speech synthesis method, apparatus, electronic device, and storage medium. Background Technology

[0002] Speech synthesis, also known as text-to-speech (TTS), aims to convert input text into fluent and natural output speech, and is a key technology for realizing human-computer voice interaction.

[0003] Speech synthesis systems typically consist of two modules: an acoustic model and a vocoder module. The acoustic model maps the input text to acoustic features, while the vocoder reconstructs the speech waveform from these acoustic features. As the bridge between the acoustic model and the vocoder, the quality of the acoustic features largely determines the upper limit of the synthesized speech's sound quality.

[0004] Most mainstream speech synthesis systems currently use mel-spectrograms as acoustic features. Mel-spectrogram features are extracted through steps such as pre-emphasis, framing, windowing, and Fourier transform. These pre-defined features are fundamentally limited. Compared to the original audio, mel-spectrogram features lose some acoustic information, thus limiting the improvement of synthesized speech quality. Summary of the Invention

[0005] This invention provides a speech synthesis method, apparatus, electronic device, and storage medium to address the shortcomings of existing technologies that use Mel spectrum features for speech synthesis, which limit the improvement of synthesized speech quality.

[0006] This invention provides a speech synthesis method, comprising:

[0007] Obtain the text to be synthesized;

[0008] Acoustic modeling is performed based on the text to obtain the speech features corresponding to the text. The speech features are in the same feature space as the sample speech features. The sample speech features are obtained by an end-to-end speech synthesis model or a self-supervised speech feature extraction model to extract features from the sample speech.

[0009] Based on the speech features, waveform reconstruction is performed to obtain the synthesized speech corresponding to the text.

[0010] According to a speech synthesis method provided by the present invention, the step of performing acoustic modeling based on the text to obtain speech features corresponding to the text includes:

[0011] Based on the acoustic model, the text is acoustically modeled to obtain the speech features corresponding to the text;

[0012] The acoustic model is trained based on sample text and the sample speech features of the corresponding sample speech.

[0013] According to a speech synthesis method provided by the present invention, the training steps of the acoustic model include:

[0014] Based on the initial acoustic model, acoustic modeling is performed on the sample text to obtain the predicted speech features and predicted phoneme durations corresponding to the sample text.

[0015] Based on the predicted speech features and the sample speech features of the sample speech, as well as the predicted phoneme duration and the phoneme duration label of the sample speech, the initial acoustic model is iterated to obtain the acoustic model.

[0016] According to a speech synthesis method provided by the present invention, the step of reconstructing the waveform based on the speech features to obtain the synthesized speech corresponding to the text includes:

[0017] Based on the vocoder, the waveform of the speech features is reconstructed to obtain the synthesized speech corresponding to the text.

[0018] The vocoder is trained based on sample speech and the sample speech features of the sample speech.

[0019] According to a speech synthesis method provided by the present invention, the training steps of the vocoder include:

[0020] Based on the initial vocoder, the waveform of the sample speech features is reconstructed to obtain the predicted speech;

[0021] Based on the initial discriminator, the authenticity of the predicted speech and the sample speech is determined, and the authenticity determination results of the predicted speech and the sample speech are obtained.

[0022] Based on the authenticity determination results of the predicted speech and the sample speech, the parameters of the initial vocoder and the initial discriminator are iterated, and the parameters of the initial vocoder are iterated based on the difference between the predicted speech and the sample speech.

[0023] According to a speech synthesis method provided by the present invention, the step of determining the difference between the predicted speech and the sample speech includes:

[0024] Based on the acoustic features of the predicted speech and the acoustic features of the sample speech, and / or the discriminative features of the predicted speech and the discriminative features of the sample speech, the difference between the predicted speech and the sample speech is determined;

[0025] The discriminative features are those extracted by the initial discriminator when determining the authenticity of speech.

[0026] According to a speech synthesis method provided by the present invention, the step of obtaining the sample speech features includes:

[0027] The sample speech is input into the self-supervised speech feature extraction model, and the features output by any hidden layer in the self-supervised speech feature extraction model are used as the sample speech features.

[0028] Alternatively, the sample speech can be input into the end-to-end speech synthesis model to obtain the first hidden layer feature output by the encoder of the end-to-end speech synthesis model as the sample speech feature, or the second hidden layer feature used as input to the decoder in the end-to-end speech synthesis model can be obtained as the sample speech feature.

[0029] The present invention also provides a speech synthesis device, comprising:

[0030] The acquisition unit is used to acquire the text to be synthesized;

[0031] The modeling unit is used to perform acoustic modeling based on the text to obtain the speech features corresponding to the text. The speech features are in the same feature space as the sample speech features. The sample speech features are obtained by an end-to-end speech synthesis model or a self-supervised speech feature extraction model to extract features from the sample speech.

[0032] A synthesis unit is used to reconstruct the waveform based on the speech features to obtain the synthesized speech corresponding to the text.

[0033] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement any of the above-described speech synthesis methods.

[0034] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the speech synthesis method as described above.

[0035] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements any of the above-described speech synthesis methods.

[0036] The speech synthesis method, apparatus, electronic device, and storage medium provided by this invention use speech features that are in the same feature space as sample speech features extracted based on an end-to-end speech synthesis model or a self-supervised speech feature extraction model as a bridge between the acoustic model and the vocoder to achieve speech synthesis. This allows richer acoustic information to be referenced during the speech synthesis process, thereby improving the speech quality of the synthesized speech. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0038] Figure 1 This is a flowchart illustrating the speech synthesis method provided by the present invention;

[0039] Figure 2 This is a flowchart of the operation of the self-supervised speech feature extraction model provided by the present invention;

[0040] Figure 3 This is a flowchart of the operation of the end-to-end speech synthesis model provided by the present invention;

[0041] Figure 4 This is a flowchart of the training process for the acoustic model provided by the present invention;

[0042] Figure 5 This is a flowchart of the training process of the vocoder provided by the present invention;

[0043] Figure 6 This is a schematic diagram of the speech synthesis device provided by the present invention;

[0044] Figure 7 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0045] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0046] In related technologies, speech synthesis is typically based on Mel-spectral features. Specifically, during training, text and Mel-spectral features are used as inputs and outputs for the acoustic model, while the Mel-spectral features and speech waveform are used as inputs and outputs for the vocoder, allowing for separate training of the acoustic model and the vocoder. In the testing phase, test text is input into the acoustic model to obtain predicted Mel-spectral features, and then the vocoder is used to obtain the final speech waveform from the predicted Mel-spectral features.

[0047] However, the loss of some acoustic information in Mel-spectral features limits the quality of synthesized speech. To overcome the bottleneck caused by acoustic features, research has shifted towards end-to-end speech synthesis, achieving relatively good results. However, end-to-end speech synthesis models generally have a large number of parameters, posing significant challenges in practical applications.

[0048] To address the above problems, this invention provides a speech synthesis method to improve the quality of synthesized speech. Figure 1 This is a flowchart illustrating the speech synthesis method provided by the present invention, as shown below. Figure 1 As shown, the method includes:

[0049] Step 110: Obtain the text to be synthesized.

[0050] Specifically, the text to be synthesized is the text used for speech synthesis. The text to be synthesized can be directly input by the user, or it can be obtained by capturing images through image acquisition devices such as scanners, mobile phones, and cameras and performing OCR (Optical Character Recognition) on the images, or it can be obtained by crawling the Internet. This embodiment of the invention does not specifically limit the types of text.

[0051] Step 120: Perform acoustic modeling based on the text to obtain the speech features corresponding to the text. The speech features are in the same feature space as the sample speech features. The sample speech features are obtained by extracting features from the sample speech using an end-to-end speech synthesis model or a self-supervised speech feature extraction model.

[0052] Step 130: Based on the speech features, perform waveform reconstruction to obtain the synthesized speech corresponding to the text.

[0053] Specifically, step 120 involves performing acoustic modeling based on the text to obtain the corresponding speech features, and step 130 involves waveform reconstruction based on the speech features. This corresponds to the related technologies that use acoustic models to extract Mel-spectral features as acoustic features and then use a vocoder to reconstruct the waveforms from the Mel-spectral features. However, unlike the related technologies, in this embodiment of the invention, Mel-spectral features are no longer used as a bridge between the acoustic model and the vocoder; instead, speech features are used to replace Mel-spectral features as the new acoustic features.

[0054] The speech features here are no longer the lossy features extracted through steps such as pre-emphasis, framing, windowing, and Fourier transform, but rather lossless features that reside in the same feature space as the sample speech features extracted by the end-to-end speech synthesis model or the self-supervised speech feature extraction model. Here, the sample speech refers to the pre-collected speech data.

[0055] It is understandable that end-to-end speech synthesis models also involve extracting speech features from text and reconstructing waveforms based on these features. However, due to the nature of end-to-end models, the process of extracting speech features within the model is not subject to human interference and is entirely learned and implemented by the model itself. Therefore, the speech features extracted by the end-to-end speech synthesis model for sample speech are high-level hidden layer representations, containing richer acoustic information compared to Mel-spectral features. Here, the end-to-end speech synthesis model extracts features from sample speech in either by inverse encoding the sample speech to obtain sample speech features, or by extracting speech features from the sample text corresponding to the sample speech. This embodiment of the invention does not specifically limit this approach.

[0056] Self-supervised speech feature extraction models are pre-trained models obtained through self-supervised learning, such as the wav2vec 2.0 model and the HuBERT model. During the self-supervised learning process, the input is heavily masked before training using a contrastive learning loss function. The resulting speech feature extraction format is also unaffected by human intervention and is entirely learned by the model itself. Therefore, the sample speech features extracted by the self-supervised speech feature extraction model are high-level hidden layer representations, containing richer acoustic information than Mel-spectral features. Here, the self-supervised speech feature extraction model extracts features from sample speech; that is, the sample speech is directly input into the self-supervised speech feature extraction model, and features extracted from any hidden layer during the feature extraction process are used as the sample speech features.

[0057] Considering that sample speech features extracted by an end-to-end speech synthesis model or by a self-supervised speech feature extraction model can contain richer acoustic information than Mel-spectral features, this embodiment of the invention restricts text-based speech features, i.e., speech features used for waveform reconstruction, to be in the same feature space as sample speech features extracted by the end-to-end speech synthesis model or by the self-supervised speech feature extraction model.

[0058] It is understandable that features within the same feature space share commonalities in information representation. That is, the speech features extracted from the text during acoustic modeling in this embodiment of the invention can contain richer acoustic information than Mel spectrum features, and the synthesized speech obtained based on this can guarantee its speech quality.

[0059] The method provided in this invention uses speech features that are in the same feature space as the sample speech features extracted based on an end-to-end speech synthesis model or a self-supervised speech feature extraction model as a bridge between the acoustic model and the vocoder to achieve speech synthesis. This allows richer acoustic information to be referenced during the speech synthesis process, thereby improving the speech quality of the synthesized speech.

[0060] Based on the above embodiments, step 120 includes:

[0061] Based on the acoustic model, the text is acoustically modeled to obtain the speech features corresponding to the text;

[0062] The acoustic model is trained based on sample text and the sample speech features of the corresponding sample speech. The sample speech features are obtained by extracting features from the sample speech based on the end-to-end speech synthesis model or the self-supervised speech feature extraction model.

[0063] Specifically, in order to acquire speech features, an acoustic model needs to be pre-trained so that the acoustic model can learn the mapping relationship between text and speech features.

[0064] Here, the training of the acoustic model can be supervised learning, specifically using sample text as training data and the sample speech features corresponding to the sample text as training labels. To ensure the effectiveness of the training labels, the sample text can be input into an end-to-end speech synthesis model to obtain the speech features obtained by the end-to-end speech synthesis model in acoustic modeling of the sample text as the sample speech features. Alternatively, the sample speech can be input into a self-supervised speech feature extraction model to obtain the speech features obtained by the self-supervised speech feature extraction model in feature extraction of the sample speech as the sample speech features.

[0065] It is understandable that the sample speech features obtained based on the end-to-end speech synthesis model or the self-supervised speech feature extraction model must be in the feature space of the features extracted by the end-to-end speech synthesis model or the self-supervised speech feature extraction model. Using the sample speech features obtained in this way as labels for supervised learning of the acoustic model can also ensure that the speech features obtained by the acoustic model obtained by supervised learning when used for acoustic modeling are in the same feature space as the above sample speech features. This allows richer acoustic information to be referenced in the speech synthesis process, thereby improving the speech quality of synthesized speech.

[0066] Based on any of the above embodiments, the training steps of the acoustic model used in step 120 may include:

[0067] Based on the initial acoustic model, acoustic modeling is performed on the sample text to obtain the predicted speech features and predicted phoneme durations corresponding to the sample text.

[0068] Based on the predicted speech features and the sample speech features of the sample speech, as well as the predicted phoneme duration and the phoneme duration label of the sample speech, the initial acoustic model is iterated to obtain the acoustic model.

[0069] Specifically, at the beginning of training the acoustic model, an initial acoustic model can be obtained. This initial acoustic model can be the acoustic model after parameter initialization. Using the initial acoustic model, acoustic modeling of the sample text can be performed. Specifically, during the acoustic modeling process, the phoneme sequence of the sample text can be encoded, and the duration of each phoneme in the phoneme sequence can be predicted, thus obtaining the predicted phoneme duration. Subsequently, the predicted speech features corresponding to the sample text can be generated based on the predicted phoneme durations and the encoded features of the phoneme sequence. In other words, through the initial acoustic model, the predicted speech features and predicted phoneme durations of the sample text can be obtained.

[0070] After obtaining the predicted speech features and predicted phoneme durations of the sample text, the predicted speech features can be compared with the sample speech features used as labels. In addition, the predicted phoneme durations can be compared with the phoneme duration labels of the sample speech. This forms a loss function to iterate the parameters of the initial acoustic model in order to obtain an acoustic model that can be used for acoustic modeling.

[0071] For example, by comparing the predicted speech features with the sample speech features, the acoustic model loss can be obtained.

[0072]

[0073] In the formula, T represents the number of speech frames, and y t Let be the predicted speech features at time t. Let be the sample speech features at time t.

[0074] By comparing the predicted phoneme duration with the phoneme duration labels of the sample speech, the duration model loss can be obtained.

[0075]

[0076] In the formula, N is the number of phonemes, d is the predicted phoneme duration, and d * For phoneme duration tags.

[0077] Therefore, the loss function used to iterate the initial acoustic model parameters can be determined.

[0078]

[0079] In the formula, λ dur Preset weights.

[0080] Furthermore, the calculation method for the acoustic model loss can differ depending on the specific speech features of the sample. For example, for speech features determined by a self-supervised speech feature extraction model, such as those determined by wav2vec2.0, the mean squared error (MSE) or mean absolute error (MAE) can be used to calculate the acoustic model loss. Conversely, for speech features determined by an end-to-end speech synthesis model, such as those determined by a conditional variational autoencoder with adversarial learning (VITS) model, the Kullback-Leibler (KL) divergence distance can be used to calculate the acoustic model loss.

[0081] The method provided in this invention combines speech features and phoneme duration to train an acoustic model, which helps to improve the acoustic modeling effect of the acoustic model.

[0082] Based on any of the above embodiments, step 130 includes:

[0083] Based on the vocoder, the waveform of the speech features is reconstructed to obtain the synthesized speech corresponding to the text.

[0084] The vocoder is trained based on sample speech and sample speech features of the sample speech. The sample speech features are obtained by extracting features from the sample speech based on the end-to-end speech synthesis model or the self-supervised speech feature extraction model.

[0085] Specifically, in order to achieve waveform reconstruction based on speech features, a vocoder needs to be pre-trained so that it can learn the mapping relationship between speech features and speech waveforms.

[0086] Here, the training of the vocoder can be supervised learning, specifically using the sample speech features as training data and the sample speech itself as training labels. To ensure the effectiveness of the sample speech features used as training data, the sample speech can be input into an end-to-end speech synthesis model to extract its speech features in reverse, or the sample speech can be input into a self-supervised speech feature extraction model to obtain the speech features extracted by the self-supervised speech feature extraction model.

[0087] It is understandable that the sample speech features obtained based on the end-to-end speech synthesis model or the self-supervised speech feature extraction model must be located in the feature space of the features extracted by the end-to-end speech synthesis model or the self-supervised speech feature extraction model. Using the sample speech features obtained in this way as training data for supervised learning of the vocoder can also ensure that the vocoder obtained by supervised learning can achieve waveform reconstruction of features that are in the same feature space as the above-mentioned sample speech features. This allows richer acoustic information to be referenced in the speech synthesis process, thereby improving the speech quality of the synthesized speech.

[0088] Based on any of the above embodiments, the training steps of the vocoder used in step 130 may include:

[0089] Based on the initial vocoder, the waveform of the sample speech features is reconstructed to obtain the predicted speech;

[0090] Based on the initial discriminator, the authenticity of the predicted speech and the sample speech is determined, and the authenticity determination results of the predicted speech and the sample speech are obtained.

[0091] Based on the authenticity determination results of the predicted speech and the sample speech, the parameters of the initial vocoder and the initial discriminator are iterated, and the parameters of the initial vocoder are iterated based on the difference between the predicted speech and the sample speech.

[0092] Specifically, vocoder training can be achieved through adversarial training. In adversarial training, the initial vocoder model can be treated as a generator, and the initial discriminator as a discriminator. The initial vocoder, acting as a generator, reconstructs the waveform of the input sample speech features, outputting the reconstructed speech, i.e., the predicted speech. The initial discriminator, acting as a discriminator, distinguishes between the real and predicted speech generated by the initial generator. During this process, the initial vocoder and the initial discriminator engage in a game of adversarial competition. The initial vocoder aims to output predicted speech that is as similar as possible to the sample speech, making it difficult for the initial discriminator to distinguish between the sample and predicted speech. The initial discriminator aims to ensure that its output of the true / false judgment result matches the actual input speech, achieving a more accurate and reliable discrimination effect.

[0093] Based on this, after obtaining the true / false discrimination result of the initial discriminator output, adversarial training can be performed on the initial vocoder and the initial discriminator based on the true / false discrimination result. In addition, the parameters of the initial vocoder can be iterated based on the difference between the sample speech and the predicted speech output by the initial vocoder, thereby realizing the training of the vocoder.

[0094] The method provided in this invention trains a vocoder through adversarial training, thereby enabling the synthesized speech output by the trained vocoder to be more realistic.

[0095] Based on any of the above embodiments, the step of determining the difference between the predicted speech and the sample speech used to iterate the initial vocoder parameters during vocoder training includes:

[0096] Based on the acoustic features of the predicted speech and the acoustic features of the sample speech, and / or the discriminative features of the predicted speech and the discriminative features of the sample speech, the difference between the predicted speech and the sample speech is determined;

[0097] The discriminative features are those extracted by the initial discriminator when determining the authenticity of speech.

[0098] Specifically, the difference between the predicted speech and the sample speech can be measured by the difference between their acoustic features and / or the difference between the features extracted when the two are used to distinguish between true and false in the initial discriminator.

[0099] Furthermore, the acoustic features of the predicted speech and the sample speech can be represented using Mel-spectral features or other types of acoustic features. Measuring the distance between the predicted speech and the sample speech along the acoustic feature dimension reflects the difference between them.

[0100] Furthermore, the discriminant features of the predicted speech and the discriminant features of the sample speech can be obtained by inputting the predicted speech and the sample speech into the initial discriminator, respectively. The initial discriminator then extracts the discriminant features of the predicted speech and the sample speech, and judges the authenticity of the predicted speech and the sample speech based on these features. In this process, the distance between the discriminant features extracted by the initial discriminator can also reflect the difference between the predicted speech and the sample speech.

[0101] Based on this, the training of the vocoder and discriminator can be achieved using the following loss function:

[0102]

[0103] Where G represents the generator and G represents the discriminator. This is the generator loss, which corresponds to the loss of the initial vocoder. Including combating losses Mel spectrum loss and feature map loss Adversarial loss primarily trains the model to deceive the discriminator, making it believe the synthesized speech is real; Mel-spectral loss measures the distance between the sample speech and the predicted speech along the Mel-spectral dimension, reflecting the difference between the acoustic features of the predicted speech and the sample speech; Feature map loss calculates the distance between the discriminative features extracted by the discriminator for the sample speech and the predicted speech, respectively, reflecting the difference between the discriminative features of the predicted speech and the sample speech.

[0104] The discriminator loss is mainly trained by classifying sample speech as true and predicted speech as false.

[0105] Based on any of the above embodiments, the steps for obtaining the sample speech features required for training the acoustic model in step 120 and the vocoder in step 130 may include:

[0106] The sample speech is input into the self-supervised speech feature extraction model, and the features output by any hidden layer in the self-supervised speech feature extraction model are used as the sample speech features.

[0107] Alternatively, the sample speech can be input into the end-to-end speech synthesis model to obtain the first hidden layer feature output by the encoder of the end-to-end speech synthesis model as the sample speech feature, or the second hidden layer feature used as input to the decoder in the end-to-end speech synthesis model can be obtained as the sample speech feature.

[0108] Specifically, for the case of obtaining sample speech features based on a self-supervised speech feature extraction model, the sample speech is input into the self-supervised speech feature extraction model. The self-supervised speech feature extraction model can apply the cascaded hidden layers it contains to extract features from the sample speech layer by layer. During the feature extraction process layer by layer, the features output by any hidden layer can be used as the sample speech features.

[0109] For example, Figure 2 This is a flowchart of the self-supervised speech feature extraction model provided by the present invention, as follows: Figure 2 As shown, a self-supervised speech feature extraction model can include a CNN encoder, a quantization layer, and a Transformer module. It can be understood that the CNN encoder, quantization layer, and Transformer module can all be regarded as hidden layers in the self-supervised speech feature extraction model. Correspondingly, the shallow representation Z output by the CNN encoder, the quantization representation Q output by the quantization layer, and the context representation C output by the Transformer module can all be used as sample speech features.

[0110] For the case of obtaining sample speech features based on end-to-end speech synthesis models, considering that end-to-end speech synthesis models can usually be divided into two parts: encoder and decoder, when determining sample speech features, the hidden layer features obtained by encoding the phoneme sequence of the sample text based on the encoder can be used as the sample speech features, which is denoted as the first hidden layer features; or the hidden layer features obtained by inverse decoding the sample speech based on the decoder can be denoted as the second hidden layer features, which is used as the sample speech features.

[0111] For example, Figure 3 This is a flowchart of the end-to-end speech synthesis model provided by the present invention, as follows: Figure 3 As shown, by inputting the phoneme sequence of the sample text into a prior encoder and predicting the duration based on the output of the prior encoder, and then performing Gaussian upsampling combined with the duration, the hidden layer representation z' sampled from the data distribution can be obtained, denoted here as the first hidden layer feature. Alternatively, the hidden layer representation z' obtained by encoding the linear spectrum of the sample speech using a posterior encoder can be decoded, denoted here as the second hidden layer feature. Either the first or second hidden layer feature can be used as the sample speech feature.

[0112] Based on any of the above embodiments, the acoustic model and vocoder used for speech synthesis can be trained using the following steps:

[0113] First, high-quality audio databases are collected. The text from these high-quality audio databases undergoes pre-processing to obtain the phoneme sequence for each sample text. Then, based on an end-to-end speech synthesis model or a self-supervised speech feature extraction model, sample speech features are extracted from the sample speech. Furthermore, the duration of each phoneme in the sample text can be obtained by forced alignment using a speech recognition model.

[0114] Figure 4 This is a flowchart of the training process for the acoustic model provided by this invention, as follows: Figure 4 As shown, the phoneme sequence of the sample text can be input into the initial acoustic model. The encoder of the initial acoustic model encodes the input phoneme sequence, and the encoded features are input into the duration prediction module for duration prediction. The predicted phoneme duration is compared with the actual duration to obtain the duration loss. Additionally, the encoded features are input into the variance prediction module for variance prediction. The predicted variance is fused with the actual duration, and then combined with the encoded features for Gaussian upsampling. The upsampled features are then fused with the positional encoding obtained from duration prediction and input into the decoder to obtain the predicted speech features output by the decoder. By comparing the differences between the predicted speech features and the sample speech features, the acoustic loss can be obtained. Combining the duration loss and acoustic loss, the parameters of the initial acoustic model can be iterated to obtain the final acoustic model.

[0115] Figure 5 This is a flowchart of the training process for the vocoder provided by the present invention, as follows: Figure 5 As shown, the sample speech features extracted by the end-to-end speech synthesis model or the self-supervised speech feature extraction model can be input into the initial vocoder to obtain the predicted speech output by the initial vocoder. By comparing the predicted speech and the sample speech, the loss function is determined to iterate the parameters of the initial vocoder, thereby obtaining the vocoder.

[0116] Based on any of the above embodiments Figure 6 This is a schematic diagram of the speech synthesis device provided by the present invention, as shown below. Figure 6 As shown, the device includes:

[0117] Acquisition unit 610 is used to acquire the text to be synthesized;

[0118] Modeling unit 620 is used to perform acoustic modeling based on the text to obtain speech features corresponding to the text. The speech features are in the same feature space as the sample speech features. The sample speech features are obtained by an end-to-end speech synthesis model or a self-supervised speech feature extraction model to extract features from the sample speech.

[0119] The synthesis unit 630 is used to reconstruct the waveform based on the speech features to obtain the synthesized speech corresponding to the text.

[0120] The apparatus provided in this invention uses speech features that are in the same feature space as the sample speech features extracted based on an end-to-end speech synthesis model or a self-supervised speech feature extraction model as a bridge between the acoustic model and the vocoder to achieve speech synthesis. This allows richer acoustic information to be referenced during the speech synthesis process, thereby improving the speech quality of the synthesized speech.

[0121] Based on any of the above embodiments, the modeling unit is used for:

[0122] Based on the acoustic model, the text is acoustically modeled to obtain the speech features corresponding to the text;

[0123] The acoustic model is trained based on sample text and the sample speech features of the corresponding sample speech.

[0124] Based on any of the above embodiments, the device further includes an acoustic model training unit, used for:

[0125] Based on the initial acoustic model, acoustic modeling is performed on the sample text to obtain the predicted speech features and predicted phoneme durations corresponding to the sample text.

[0126] Based on the predicted speech features and the sample speech features of the sample speech, as well as the predicted phoneme duration and the phoneme duration label of the sample speech, the initial acoustic model is iterated to obtain the acoustic model.

[0127] Based on any of the above embodiments, the synthesis unit is used for:

[0128] Based on the vocoder, the waveform of the speech features is reconstructed to obtain the synthesized speech corresponding to the text.

[0129] The vocoder is trained based on sample speech and the sample speech features of the sample speech.

[0130] Based on any of the above embodiments, the device further includes a vocoder training unit, used for:

[0131] Based on the initial vocoder, the waveform of the sample speech features is reconstructed to obtain the predicted speech;

[0132] Based on the initial discriminator, the authenticity of the predicted speech and the sample speech is determined, and the authenticity determination results of the predicted speech and the sample speech are obtained.

[0133] Based on the authenticity determination results of the predicted speech and the sample speech, the parameters of the initial vocoder and the initial discriminator are iterated, and the parameters of the initial vocoder are iterated based on the difference between the predicted speech and the sample speech.

[0134] Based on any of the above embodiments, the vocoder training unit is further configured to:

[0135] Based on the acoustic features of the predicted speech and the acoustic features of the sample speech, and / or the discriminative features of the predicted speech and the discriminative features of the sample speech, the difference between the predicted speech and the sample speech is determined;

[0136] The discriminative features are those extracted by the initial discriminator when determining the authenticity of speech.

[0137] Based on any of the above embodiments, the device further includes a sample acquisition unit, used for:

[0138] The sample speech is input into the self-supervised speech feature extraction model, and the features output by any hidden layer in the self-supervised speech feature extraction model are used as the sample speech features.

[0139] Alternatively, the sample speech can be input into the end-to-end speech synthesis model to obtain the first hidden layer feature output by the encoder of the end-to-end speech synthesis model as the sample speech feature, or the second hidden layer feature used as input to the decoder in the end-to-end speech synthesis model can be obtained as the sample speech feature.

[0140] Figure 7 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 7 As shown, the electronic device may include a processor 710, a communications interface 720, a memory 730, and a communication bus 740. The processor 710, communications interface 720, and memory 730 communicate with each other via the communication bus 740. The processor 710 can call logical instructions in the memory 730 to execute a speech synthesis method. This method includes: acquiring text to be synthesized; performing acoustic modeling based on the text to obtain speech features corresponding to the text, wherein the speech features and sample speech features are in the same feature space, and the sample speech features are obtained by feature extraction from the sample speech using an end-to-end speech synthesis model or a self-supervised speech feature extraction model; and reconstructing the waveform based on the speech features to obtain synthesized speech corresponding to the text.

[0141] Furthermore, the logical instructions in the aforementioned memory 730 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0142] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the speech synthesis method provided by the above methods. The method includes: acquiring text to be synthesized; performing acoustic modeling based on the text to obtain speech features corresponding to the text, wherein the speech features and sample speech features are in the same feature space, and the sample speech features are obtained by feature extraction of sample speech by an end-to-end speech synthesis model or a self-supervised speech feature extraction model; and performing waveform reconstruction based on the speech features to obtain synthesized speech corresponding to the text.

[0143] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the speech synthesis method provided by the above methods. The method includes: acquiring text to be synthesized; performing acoustic modeling based on the text to obtain speech features corresponding to the text, wherein the speech features and sample speech features are in the same feature space, and the sample speech features are obtained by feature extraction of sample speech by an end-to-end speech synthesis model or a self-supervised speech feature extraction model; and reconstructing the waveform based on the speech features to obtain synthesized speech corresponding to the text.

[0144] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0145] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0146] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A speech synthesis method, characterized in that, include: Obtain the text to be synthesized; Based on the acoustic model, the text is acoustically modeled to obtain the speech features corresponding to the text. The acoustic model is trained based on the sample text and the sample speech features of the corresponding sample speech. The sample speech features are the labels for the acoustic model to perform supervised learning. The speech features and the sample speech features are in the same feature space. The sample speech features are obtained by extracting features from the sample speech using an end-to-end speech synthesis model or a self-supervised speech feature extraction model. Based on the speech features, waveform reconstruction is performed to obtain the synthesized speech corresponding to the text.

2. The speech synthesis method according to claim 1, characterized in that, The training steps for the acoustic model include: Based on the initial acoustic model, acoustic modeling is performed on the sample text to obtain the predicted speech features and predicted phoneme durations corresponding to the sample text. Based on the predicted speech features and the sample speech features of the sample speech, as well as the predicted phoneme duration and the phoneme duration label of the sample speech, the initial acoustic model is iterated to obtain the acoustic model.

3. The speech synthesis method according to claim 1, characterized in that, The step of reconstructing the waveform based on the speech features to obtain the synthesized speech corresponding to the text includes: Based on the vocoder, the waveform of the speech features is reconstructed to obtain the synthesized speech corresponding to the text. The vocoder is trained based on sample speech and the sample speech features of the sample speech.

4. The speech synthesis method according to claim 3, characterized in that, The training steps for the vocoder include: Based on the initial vocoder, the waveform of the sample speech features is reconstructed to obtain the predicted speech; Based on the initial discriminator, the authenticity of the predicted speech and the sample speech is determined, and the authenticity determination results of the predicted speech and the sample speech are obtained. Based on the authenticity determination results of the predicted speech and the sample speech, the parameters of the initial vocoder and the initial discriminator are iterated, and the parameters of the initial vocoder are iterated based on the difference between the predicted speech and the sample speech.

5. The speech synthesis method according to claim 4, characterized in that, The steps for determining the difference between the predicted speech and the sample speech include: Based on the acoustic features of the predicted speech and the acoustic features of the sample speech, and / or the discriminative features of the predicted speech and the discriminative features of the sample speech, the difference between the predicted speech and the sample speech is determined; The discriminative features are those extracted by the initial discriminator when determining the authenticity of speech.

6. The speech synthesis method according to any one of claims 1 to 5, characterized in that, The steps for obtaining the sample speech features include: The sample speech is input into the self-supervised speech feature extraction model, and the features output by any hidden layer in the self-supervised speech feature extraction model are used as the sample speech features. Alternatively, the sample speech can be input into the end-to-end speech synthesis model to obtain the first hidden layer feature output by the encoder of the end-to-end speech synthesis model as the sample speech feature, or the second hidden layer feature used as input to the decoder in the end-to-end speech synthesis model can be obtained as the sample speech feature.

7. A speech synthesis device, characterized in that, include: The acquisition unit is used to acquire the text to be synthesized; The modeling unit is used to perform acoustic modeling on the text based on an acoustic model to obtain the speech features corresponding to the text. The acoustic model is trained based on sample text and the sample speech features of the sample speech corresponding to the sample text. The sample speech features are labels for the acoustic model to perform supervised learning. The speech features and the sample speech features are in the same feature space. The sample speech features are obtained by extracting features from the sample speech using an end-to-end speech synthesis model or a self-supervised speech feature extraction model. A synthesis unit is used to reconstruct the waveform based on the speech features to obtain the synthesized speech corresponding to the text.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the speech synthesis method as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the speech synthesis method as described in any one of claims 1 to 6.

Images