Speech synthesis method, model training method and related device

Abstract

The invention discloses a speech synthesis method, a model training method and a related device. The method comprises the following steps: determining a hidden variable corresponding to text information; determining fundamental frequency information corresponding to the text information based on the hidden variable; and based on the hidden variables and the fundamental frequency information, generating synthetic speech corresponding to the text information. Therefore, in the process of generating the voice from the text, the fundamental frequency trend can be controlled from the phoneme level of the text information by introducing the fundamental frequency information, and meanwhile, the problem of pitch breakage when the decoder synthesizes the voice is solved, so that the tone quality of the synthesized voice is improved.

Description

Technical Field

[0001] The present application relates to but is not limited to the field of computer technology, and in particular to a speech synthesis method, a model training method and related devices. Background Art

[0002] Speech synthesis is an important technology in the field of artificial intelligence. Existing speech synthesis technologies include end-to-end speech synthesis models and non-end-to-end speech synthesis models. Among them, the advantage of the end-to-end speech synthesis model is that the model training efficiency is high, and the advantage of the non-end-to-end speech synthesis model is that the composition of each module in the model is flexible. However, compared with the end-to-end speech synthesis model, the sound quality of the non-end-to-end speech synthesis model is slightly worse.

[0003] The Variational Inference with adversarial learning for end-to-end Text-to-Speech (VITS) model is the current mainstream end-to-end speech synthesis model. The speech information synthesized by this model and most vocoder models based on Generative Adversarial Network (GAN) has the problem of interruption or mutation of pitch continuity, which makes the synthesized speech sound unnatural or discontinuous.

[0004] Therefore, how to solve the pitch gap problem existing in the current speech synthesis model has become an urgent problem that needs to be solved. Summary of the invention

[0005] In view of this, the present application at least provides a speech synthesis method, a model training method and related devices.

[0006] The technical solution of this application is implemented as follows:

[0007] On the one hand, the present application provides a speech synthesis method, comprising:

[0008] Determine the latent variables corresponding to the text information;

[0009] Based on the latent variable, determining fundamental frequency information corresponding to the text information;

[0010] Based on the latent variable and the fundamental frequency information, a synthesized speech corresponding to the text information is generated.

[0011] In some embodiments, determining the hidden variable corresponding to the text information includes:

[0012] Determining a phoneme duration corresponding to at least one phoneme in the text information;

[0013] Based on the phoneme duration, upsampling the text vector representation corresponding to the at least one phoneme to obtain at least one frame vector representation corresponding to the at least one phoneme;

[0014] The latent variable is determined based on the at least one frame vector representation.

[0015] In some embodiments, generating the synthesized speech corresponding to the text information based on the latent variable and the fundamental frequency information includes:

[0016] Based on the baseband information and the noise information, generating baseband embedding information;

[0017] The synthesized speech is generated based on the fundamental frequency embedding information and the latent variable.

[0018] In some embodiments, generating baseband embedding information based on the baseband information and noise information includes:

[0019] Performing a sine transform on the baseband information to obtain a sine wave representation;

[0020] Baseband embedding information is generated based on the sinusoidal wave representation and the noise information.

[0021] On the other hand, the present application also provides a speech synthesis model training method, comprising:

[0022] Determine sample data and an initial model; the sample data includes sample text and sample speech corresponding to the sample text;

[0023] Using the initial model, generating a synthesized speech corresponding to the sample text; wherein the initial model includes an initial fundamental frequency predictor and an initial decoder; the initial fundamental frequency predictor is used to predict the predicted fundamental frequency information corresponding to the sample data; the initial decoder is used to generate the corresponding synthesized speech based on the fundamental frequency information and latent variables corresponding to the sample data;

[0024] Based on the loss between the sample speech and the synthesized speech, the initial model is updated to obtain a trained speech synthesis model.

[0025] In some embodiments, the initial model includes an initial a priori encoder and an initial a posteriori encoder;

[0026] Before using the initial model to generate the synthesized speech corresponding to the sample speech, the method further includes:

[0027] Using the initial a priori encoder, generating a priori latent variables corresponding to the sample text;

[0028] Based on the sample speech, using the initial posterior encoder, generating a posterior latent variable corresponding to the sample text;

[0029] Based on the loss between the priori latent variable and the a posteriori latent variable, the initial a priori encoder and the initial a posteriori encoder are updated until a preset convergence condition is met.

[0030] In some embodiments, the sample text has a corresponding sample fundamental frequency; the method further includes:

[0031] Based on the a posteriori latent variables, using the initial fundamental frequency predictor, generating a predicted fundamental frequency;

[0032] Based on the loss between the predicted fundamental frequency and the sample fundamental frequency, the initial fundamental frequency predictor is updated until a preset convergence condition is met.

[0033] In some embodiments, the initial a priori encoder includes an initial duration predictor and an initial mapping module; the sample text includes at least one phoneme; each of the phonemes has a sample duration;

[0034] The step of using the initial a priori encoder to generate a priori latent variables corresponding to the sample text includes:

[0035] Using the initial duration predictor, generating a predicted duration of at least one phoneme in the sample text;

[0036] Based on the at least one phoneme and the predicted duration, the prior latent variable is generated using the initial mapping module; or, based on the at least one phoneme and the sample duration, the prior latent variable is generated using the initial mapping module.

[0037] In some embodiments, the method further comprises:

[0038] Based on the loss between the predicted duration and the sample duration, the initial duration predictor is updated until a preset convergence condition is met.

[0039] In some embodiments, the generating of synthesized speech corresponding to the sample text using the initial model includes:

[0040] The synthesized speech is generated based on the sample fundamental frequency and the a posteriori latent variable using the initial decoder.

[0041] On the other hand, the present application also provides a speech synthesis device, the device comprising:

[0042] A first determination module is used to determine the latent variables corresponding to the text information;

[0043] A second determination module, used to determine the voice fundamental frequency information corresponding to the text information based on the latent variable;

[0044] The first generating module is used to generate the speech information corresponding to the text information based on the latent variable and the speech fundamental frequency information.

[0045] On the other hand, the present application also provides a speech synthesis model training device, comprising:

[0046] The third determination module is used to determine sample data and an initial model; the sample data has corresponding sample speech; the initial model includes an initial fundamental frequency predictor and an initial decoder; the initial fundamental frequency predictor is used to predict the speech fundamental frequency information corresponding to the text information, so that the initial decoder generates the corresponding speech information based on the speech fundamental frequency information;

[0047] A second generation module is used to generate a synthetic speech corresponding to the sample text using the initial model;

[0048] The first updating module is used to update the initial model based on the loss between the sample text corresponding to the sample speech and the synthesized speech to obtain a trained speech synthesis model.

[0049] On the other hand, the present application provides a computer device, including a memory and a processor, wherein the memory stores a computer program executable on the processor, and the processor implements part or all of the steps in the above method when executing the program.

[0050] On the other hand, the present application provides a computer-readable storage medium having a computer program stored thereon, which implements part or all of the steps in the above method when executed by a processor.

[0051] On the other hand, the present application provides a computer program, comprising computer-readable codes. When the computer-readable codes are executed in a computer device, a processor in the computer device executes some or all of the steps for implementing the above method.

[0052] On the other hand, the present application provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and when the computer program is read and executed by a computer, it implements part or all of the steps in the above method.

[0053] In the speech synthesis method provided by the present application, first, the hidden variables corresponding to the text information are determined; then, the fundamental frequency information corresponding to the text information is determined based on the hidden variables; finally, the synthesized speech corresponding to the text information is generated based on the hidden variables and the fundamental frequency information. In this way, in the process of generating speech from text, by introducing the fundamental frequency information, the fundamental frequency trend can be controlled from the phoneme level of the text information, and the pitch break problem when the decoder synthesizes speech is solved, thereby improving the sound quality of the synthesized speech.

[0054] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and are not intended to limit the technical solutions of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and are used together with the specification to illustrate the technical solution of the present application.

[0056] Figure 1 A schematic diagram of the implementation flow of a speech synthesis method provided in this application;

[0057] Figure 2 A schematic diagram of an implementation flow of an embodiment of a speech synthesis method provided by the present application;

[0058] Figure 3 A schematic diagram of the implementation flow of a speech synthesis model training method provided in this application;

[0059] Figure 4 A block diagram of an implementation of an embodiment of training an initial model using the speech synthesis model training method provided by the present application;

[0060] Figure 5 A block diagram of an implementation of an embodiment of training an initial decoder in an initial model according to the speech synthesis model training method provided by the present application;

[0061] Figure 6 A schematic diagram of the structure of a speech synthesis device provided in this application;

[0062] Figure 7 A schematic diagram of the structure of a speech synthesis model training device provided in this application;

[0063] Figure 8 A hardware entity diagram of a computer device provided for this application. DETAILED DESCRIPTION

[0064] In order to make the purpose, technical solutions and advantages of the present application clearer, the technical solutions of the present application are further elaborated in detail below in conjunction with the drawings and embodiments. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0065] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0066] The terms "first / second / third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It is understandable that "first / second / third" can be interchanged with a specific order or sequence where permitted so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0067] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing this application and are not intended to limit this application.

[0068] The present application provides a speech synthesis method, which can be executed by a processor of a computer device, wherein the computer device may refer to a device with data processing capabilities such as a server, a laptop computer, a tablet computer, and a desktop computer.

[0069] Figure 1 A flowchart of a speech synthesis method provided in this application is shown in FIG. Figure 1 As shown, the method includes the following steps S101 to S103:

[0070] Step S101, determining the hidden variables corresponding to the text information.

[0071] Here, the text information refers to the text information for which synthesized speech is to be generated.

[0072] In some embodiments, the text information may be text information input by a user in an application for implementing the speech synthesis method.

[0073] In some embodiments, the text information may also be text information actively determined based on the user's interaction information in other applications. For example, when the user uses an instant messaging application to interact with an interactive object, the user's interaction content is actively identified, and the text information that the user expects to generate voice information is determined from the interaction content.

[0074] Here, determining the latent variable corresponding to the text information refers to encoding the text information to obtain the representation of the text information in the latent space.

[0075] In some embodiments, at least one phoneme may be determined from the text information, and the at least one phoneme may be mapped to a latent variable using a prior encoder.

[0076] Here, a phoneme is the smallest unit or the smallest speech segment that makes up a syllable, and is the smallest sound unit in a language that can distinguish meanings. For example, in the case where the synthesized speech is in Chinese, the phoneme may be the smallest speech segment corresponding to Chinese pinyin. For example, for the text information "中国" (China), the corresponding phonemes may include: "zh", "ong1", "g", and "uo2".

[0077] In this way, by analyzing the phonemes in the text information and precisely controlling and combining the phonemes, a synthesized speech with specific speech characteristics and styles can be generated.

[0078] In some embodiments, each phoneme corresponding to the text information may be mapped to a corresponding phoneme identity (ID) according to a preset mapping relationship, and the latent variable corresponding to the text information may be determined based on at least one phoneme ID using a prior encoder.

[0079] Here, the phoneme ID is a number or code that uniquely identifies a phoneme. In a speech synthesis system, phoneme data can be efficiently managed and processed through the phoneme ID.

[0080] Step S102, based on the latent variable, determine the fundamental frequency information corresponding to the text information.

[0081] Here, the fundamental frequency (F0) information refers to the periodic vibration frequency of the lowest frequency in the speech waveform, which determines the pitch of the sound, that is, the pitch information in the speech.

[0082] In this way, based on the latent variable, using a preset fundamental frequency predictor, the fundamental frequency information corresponding to the latent variable is generated, that is, the fundamental frequency information corresponding to the text information, thereby introducing the fundamental frequency information into the speech synthesis process.

[0083] Step S103, based on the latent variable and the fundamental frequency information, generate the synthesized speech corresponding to the text information.

[0084] Here, the decoder, also known as a vocoder, is a neural network suitable for decoding a latent variable into corresponding speech information.

[0085] Based on the latent variable and the fundamental frequency information corresponding to the text information, using a decoder, the synthesized speech corresponding to the text information is generated.

[0086] Since the baseband information is introduced into the speech synthesis method provided in the present application, the decoder adopted by the method provided in the present application also introduces a network structure for processing the baseband information to realize the processing of the baseband information.

[0087] In some embodiments, the decoder provided by the present application is composed of an upsampling module, a downsampling module, a global information extraction module and a HiFiGan vocoder. Among them, the HiFiGan vocoder is a vocoder based on Generative Adversial Networks (GAN), which can convert the spectrum generated by the acoustic model into high-quality voice information.

[0088] In the speech synthesis method provided by the present application, first, the hidden variables corresponding to the text information are determined; then, the fundamental frequency information corresponding to the text information is determined based on the hidden variables; finally, the synthesized speech corresponding to the text information is generated based on the hidden variables and the fundamental frequency information. In this way, in the process of generating speech from text, by introducing the fundamental frequency information, the fundamental frequency trend can be controlled from the phoneme level of the text information, and the pitch break problem when the decoder synthesizes speech is solved, thereby improving the sound quality of the synthesized speech.

[0089] In some embodiments, the determining of the hidden variable corresponding to the text information, that is, the above step S101, can be implemented as the following steps S1011 to S1013:

[0090] Step S1011, determining a phoneme duration corresponding to at least one phoneme in the text information.

[0091] Here, at least one phoneme in the text information is determined, and the phoneme duration corresponding to each phoneme is determined, so that the duration information of the phoneme level in the synthesized speech is controlled based on the phoneme duration.

[0092] In some embodiments, a duration predictor may be used to determine the phoneme duration corresponding to each phoneme in the text information.

[0093] Step S1012: up-sample the text vector representation corresponding to the at least one phoneme based on the phoneme duration to obtain at least one frame vector representation corresponding to the at least one phoneme.

[0094] Here, at least one phoneme is upsampled based on the phoneme duration, that is, based on the phoneme duration of at least one phoneme, the text vector representation corresponding to the at least one phoneme is upsampled to obtain a vector representation at the frame level.

[0095] In some embodiments, first, based on the phoneme duration of at least one phoneme, the number of frames corresponding to each phoneme is determined; then, for each phoneme, the text vector representation corresponding to the phoneme is expanded to the same number as the corresponding number of frames, thereby obtaining at least one frame vector representation.

[0096] In some embodiments, for each phoneme, the corresponding text vector representation is split according to the number of corresponding frames, thereby obtaining at least one corresponding frame vector representation.

[0097] Step S1013: determining the latent variable based on the at least one frame vector representation.

[0098] Here, a neural network is used to map at least one frame vector representation into latent variables.

[0099] In some embodiments, a multi-layer one-dimensional convolutional neural network (CNN) may be used to map at least one frame vector representation into latent variables.

[0100] In some embodiments, the projection module and flow structure in the VITS model can be used to map at least one frame vector representation into latent variables.

[0101] Here, the Projection module is used to convert at least one frame vector representation into a set of covariance matrices and mean sequences, and the Flow structure is used to convert the covariance matrix and mean sequence into latent variables.

[0102] In the above embodiment, by determining the phoneme duration corresponding to at least one phoneme in the text information, and converting the text vector representation of at least one phoneme into at least one frame vector representation based on the phoneme duration, and determining the latent variable corresponding to the text information based on the at least one frame vector representation, the phoneme-level duration control and audio trend control are achieved in the speech synthesis process, thereby substantially improving the effect of improving the sound quality of the synthesized speech.

[0103] In some embodiments, the generating of the synthesized speech corresponding to the text information based on the latent variable and the fundamental frequency information, that is, the above step S103, can be implemented as the following steps S1031 to S1032:

[0104] Step S1031: Generate baseband embedding information based on the baseband information and noise information.

[0105] In some embodiments, the noise information refers to random noise information.

[0106] In some embodiments, the noise information may be Gaussian noise, salt and pepper noise, Poisson noise, or the like.

[0107] Here, noise information is added to the baseband information, thereby obtaining embedded information corresponding to the baseband information.

[0108] In some embodiments, the above step S1031 can be implemented as the following steps S1033 to S1034:

[0109] Step S1033, performing a sine transform on the baseband information to obtain a sine wave representation;

[0110] Step S1034: Generate baseband embedding information based on the sine wave representation and the noise information.

[0111] Here, a sine transform is performed on the baseband information, thereby obtaining a sine wave representation corresponding to the baseband information.

[0112] In some embodiments, voice and UVoice (VUV) information is determined based on the baseband information. For example, when the baseband information is higher than a preset baseband threshold, it is determined to be voice information; when the baseband information is lower than the preset baseband threshold, it is determined to be silence information.

[0113] Thus, in some embodiments, a sine transform is performed on the baseband information and the determined VUV information to obtain a sine wave representation corresponding to the baseband information and the VUV information.

[0114] In some embodiments, first, noise information is added to the sinusoidal wave representation corresponding to the baseband information and the VUV information to obtain a sinusoidal wave representation containing noise; then, the sinusoidal wave representation containing noise is upsampled, and CNN is used to learn global information to obtain global features; then, the global features are downsampled to obtain the baseband embedding information.

[0115] Step S1032: Generate the synthesized speech based on the fundamental frequency embedding information and the latent variable.

[0116] Here, the baseband embedding information and the latent variable are added and sliced ​​to obtain multiple sub-data; the multiple sub-data are input into the decoder to perform speech prediction, and the predicted synthesized speech is output.

[0117] In the above embodiment, when latent variables are used to predict the synthesized speech corresponding to the text information, fundamental frequency information is explicitly introduced as a control condition for the speech synthesis process, so that the decoder can be guided to synthesize speech with more accurate fundamental frequency information, thereby improving the sound quality of the synthesized speech.

[0118] Next, combine Figure 2 , the implementation process of an embodiment of the speech synthesis method provided by the present application is described. Figure 2As shown, this embodiment includes the following steps S201 to S208:

[0119] Step S201, determining a plurality of phonemes corresponding to the text information; then, executing step S202;

[0120] Step S202, determining multiple phoneme IDs corresponding to multiple phonemes; then, executing step S203;

[0121] Step S203, generating text vector representations corresponding to multiple phoneme IDs; then, executing step S204;

[0122] Step S204, predicting the phoneme durations corresponding to the multiple phoneme IDs; then, executing step S205;

[0123] Step S205, based on the phoneme duration, up-sample the text vector representation of the phoneme ID to vector representations at multiple frame levels; then, execute step S206;

[0124] Step S206, mapping the vector representations of multiple frame levels to the latent variable z; then, executing step S207;

[0125] Step S207, predicting the fundamental frequency information corresponding to the text information based on the latent variable z; then, executing step S208;

[0126] Step S208: Generate synthesized speech based on the latent variable z and the fundamental frequency information.

[0127] It can be seen from the above embodiments that in the speech synthesis method provided by the present application, by generating fundamental frequency information corresponding to text information, and using the fundamental frequency information as a control condition for generating synthesized speech, it is possible to use more accurate fundamental frequency information to guide the generation process of synthesized speech, which can effectively solve the problem of pitch discontinuity in synthesized speech and thus improve the sound quality of synthesized speech.

[0128] On the other hand, the present application also provides a speech synthesis model training method, which can be executed by a processor of a computer device, wherein the computer device may refer to a device with data processing capabilities such as a server, a laptop, a tablet computer or a desktop computer.

[0129] Figure 3 A flowchart of a speech synthesis model training method provided in this application is shown in FIG. Figure 3 As shown, the method includes the following steps S301 to S303:

[0130] Step S301, determining sample data and an initial model; the sample data includes sample text and sample speech corresponding to the sample text.

[0131] Here, the sample data refers to the data used to train and fine-tune the initial model, and the sample data includes sample text and sample speech corresponding to the sample text.

[0132] For example, approximately 400 hours of multi-speaker data may be prepared as sample data, and 20 to 500 sentences of the data may be used as fine-tuning data, and the rest of the data may be used as training data.

[0133] In some embodiments, denoising processing may be performed on the sample data to reduce the signal-to-noise ratio of the sample data and improve data quality, thereby improving the model training effect.

[0134] In some embodiments, the Montreal Forced Aligner (MFA) tool may be used to extract duration information corresponding to the sample speech, and align the sample text with the sample speech.

[0135] The initial model refers to the speech synthesis model to be trained.

[0136] In some embodiments, the initial model is a model that improves the network structure of the VITS model in the related art. For example, the initial model adopts the network structure of the text encoder, posterior encoder, and projection modules in the VITS model; at the same time, the initial model modifies the network structure and training method of the decoder, duration predictor, and other modules in the VITS model; in addition, the initial model deletes the monotonic alignment search (MAS) module in the VITS model and adds a fundamental frequency predictor.

[0137] Step S302, using the initial model to generate the synthesized speech corresponding to the sample text; wherein the initial model includes an initial fundamental frequency predictor and an initial decoder; the initial fundamental frequency predictor is used to predict the predicted fundamental frequency information corresponding to the sample data; the initial decoder is used to generate the corresponding synthesized speech based on the fundamental frequency information and latent variables corresponding to the sample data.

[0138] Here, the sample text is input into the initial model so as to use the initial model to generate predicted speech corresponding to the sample text, that is, synthesized speech.

[0139] The initial model includes an initial fundamental frequency predictor and an initial decoder, wherein the initial fundamental frequency predictor is used to predict the predicted fundamental frequency corresponding to the sample data; the initial decoder is used to generate the corresponding synthesized speech based on the fundamental frequency information and latent variables corresponding to the sample data. In this way, by extracting the predicted fundamental frequency corresponding to the sample text and using the fundamental frequency information for the generation of synthesized speech, the trained speech synthesis model can use the predicted fundamental frequency information to control the generation process of the synthesized speech, thereby solving the problem of pitch discontinuity in the synthesized speech in the related art, and further improving the sound quality of the synthesized speech.

[0140] Step S303: updating the initial model based on the loss between the sample speech and the synthesized speech to obtain a trained speech synthesis model.

[0141] Here, a preset loss function is used to calculate the loss between the sample speech and the synthesized speech, and the calculated loss is used to update the initial model until the initial model meets the preset convergence conditions to obtain a trained speech synthesis model.

[0142] In some embodiments, the preset loss function may be a mean squared error (MSE) function, a mean absolute error (MAE) function, a KL divergence (Kullback-LeiblerDivergence) function, or the like.

[0143] In the speech synthesis model training method provided by the present application, first, sample data and an initial model are determined; then, the initial model is used to generate a synthesized speech corresponding to the sample text, wherein the initial model includes an initial fundamental frequency predictor and an initial decoder, the initial fundamental frequency predictor is used to predict the predicted fundamental frequency information corresponding to the sample data, and the initial decoder is used to generate the corresponding synthesized speech based on the fundamental frequency information corresponding to the sample data and the hidden variable; finally, based on the loss between the sample speech and the synthesized speech, the initial model is updated to obtain the trained speech synthesis model. In this way, on the one hand, the initial model is trained with sample data, so that the initial model can learn the mapping relationship between the sample text and the sample speech, so that it has the function of generating speech from text; on the other hand, the initial fundamental frequency predictor in the initial model predicts the predicted fundamental frequency corresponding to the sample data, and the initial decoder in the initial model generates the synthesized speech based on the fundamental frequency information corresponding to the sample data and the hidden variable, so that the trained speech synthesis model can extract the fundamental frequency information from the text information, and use the fundamental frequency information to control the speech synthesis process, thereby solving the problem of pitch discontinuity in the synthesized speech in the related art, and then achieving the effect of improving the sound quality of the synthesized speech.

[0144] In some embodiments, the initial model includes an initial a priori encoder and an initial a posteriori encoder.

[0145] Thus, before using the initial model to generate the synthesized speech corresponding to the sample speech, that is, before the above step S302, the method further includes the following steps S304 to S306:

[0146] Step S304: using the initial a priori encoder to generate a priori latent variables corresponding to the sample text.

[0147] Here, the initial prior encoder refers to the neural network in the initial model used to generate prior latent variables based on sample text.

[0148] The sample text is input into the initial prior encoder to generate a priori latent variables using the initial prior encoder.

[0149] In some embodiments, before the sample text is input into the initial a priori encoder, at least one phoneme corresponding to the sample text and a phoneme ID corresponding to each phoneme are determined, so that at least one phoneme ID is input into the a priori encoder.

[0150] In some embodiments, using an initial a priori encoder, first, text embedding processing is performed on at least one phoneme ID to obtain a predicted text vector representation of at least one phoneme ID; then, the predicted text vector representation is mapped to a priori latent variables.

[0151] Step S305: Based on the sample speech, using the initial a posteriori encoder, generate a posteriori latent variables corresponding to the sample text.

[0152] Here, the initial posterior encoder is used to generate a posterior latent variables corresponding to the sample speech based on the sample speech input. Through the initial posterior encoder, the model can better capture the potential features of the sample speech and convert them into a posterior latent variables to achieve efficient end-to-end speech synthesis.

[0153] In some embodiments, a designated extraction tool may be used to extract a corresponding linear spectrum from the sample speech, and the linear spectrum may be used as an input of an initial a posteriori encoder.

[0154] In some embodiments, the Librosa tool may be used to extract the corresponding linear spectrum from the sample speech.

[0155] In some embodiments, the network structure of the initial a posteriori encoder is similar to the network structure of the a posteriori encoder in the VITS model.

[0156] Step S306: based on the loss between the a priori latent variable and the a posteriori latent variable, update the initial a priori encoder and the initial a posteriori encoder until a preset convergence condition is met.

[0157] Here, based on the preset loss function, the loss between the posterior latent variable and the prior latent variable is calculated.

[0158] In some embodiments, the preset loss function may be a MES loss function, a MAE loss function, a KL divergence loss function, or the like.

[0159] In some embodiments, the preset convergence condition may include the number of updates to the initial a priori encoder and the initial a posteriori encoder reaching a preset number threshold; it may also include the loss between the a posteriori latent variable and the a priori latent variable being less than a preset loss threshold.

[0160] In some embodiments, the sample text has a corresponding sample fundamental frequency.

[0161] Here, the sample fundamental frequency refers to the fundamental frequency information extracted from the sample speech using a specified extraction tool. Here, the sample fundamental frequency is used as a true value label corresponding to the sample text.

[0162] In some embodiments, REAPER (Robust Epoch And Pitch Estimator), Praat phonetics software (doing phonetics by computer) or other tools with a fundamental frequency extraction function may be used to extract fundamental frequency information from the sample speech as the sample fundamental frequency.

[0163] Thus, the speech synthesis method provided by the present application further includes the following steps S307 to S308:

[0164] Step S307: Generate a predicted fundamental frequency based on the a posteriori latent variable and using the initial fundamental frequency predictor.

[0165] Here, the posterior latent variable generated by the posterior encoder is input into the initial fundamental frequency predictor, so as to use the initial fundamental frequency predictor to predict the predicted fundamental frequency corresponding to the sample text from the prior latent variable.

[0166] Step S308: updating the initial fundamental frequency predictor based on the loss between the predicted fundamental frequency and the sample fundamental frequency until a preset convergence condition is met.

[0167] Here, a preset loss function is used to calculate the loss between the predicted fundamental frequency corresponding to the sample text and the sample fundamental frequency, and the loss is used to update the initial fundamental frequency predictor until the preset convergence condition is met to obtain the trained fundamental frequency predictor.

[0168] In some embodiments, the preset loss function may be a MES loss function, a MAE loss function, a KL divergence loss function, or the like.

[0169] In some embodiments, the preset convergence condition may include that the number of updates to the initial fundamental frequency predictor reaches a preset number threshold; or may include that the loss between the predicted fundamental frequency and the sample fundamental frequency is less than a preset loss threshold.

[0170] In some embodiments, the initial a priori encoder includes an initial duration predictor and an initial mapping module; the sample text includes at least one phoneme; each of the phonemes has a sample duration;

[0171] In this way, the initial a priori encoder is used to generate a priori latent variables corresponding to the sample text, that is, the above step S304 can be implemented as the following steps S3041 to S3042:

[0172] Step S3041, using the initial duration predictor to generate a predicted duration of at least one phoneme in the sample text;

[0173] Step S3042, based on the at least one phoneme and the predicted duration, using the initial mapping module, generating the prior latent variable; or, based on the at least one phoneme and the sample duration, using the initial mapping module, generating the prior latent variable.

[0174] Here, the specific implementation method of determining at least one phoneme from the sample text can be found in the detailed description above and will not be repeated here.

[0175] Sample duration refers to the true value label of the phoneme duration corresponding to each phoneme in the sample text.

[0176] In some embodiments, the corresponding sample duration of each phoneme may be manually annotated; or the sample duration corresponding to each phoneme may be extracted from the sample speech using a designated extraction tool.

[0177] In this way, at least one phoneme in the sample text is input into the initial duration predictor to obtain the predicted duration corresponding to each phoneme.

[0178] In some embodiments, the initial duration predictor is updated based on the loss between the predicted duration and the sample duration until a preset convergence condition is met.

[0179] Here, the initial duration predictor is used as an independent module to be trained, and the loss between the predicted duration and the sample duration is used to update the initial duration predictor. In this way, by independently training and updating the initial duration predictor, the training accuracy of the initial duration predictor can be improved, thereby obtaining a more accurate duration predictor.

[0180] In some embodiments, a preset loss function is used to calculate the loss between the predicted duration and the sample duration.

[0181] In some embodiments, the preset loss function may be a MES loss function, a MAE loss function, a KL divergence loss function, or the like.

[0182] In some embodiments, the preset convergence condition may include that the number of updates to the initial duration predictor reaches a preset threshold number of times; it may also include that the loss between the predicted duration and the sample duration is less than a preset loss threshold.

[0183] Here, during the model training process, at least one phoneme and the predicted duration corresponding to the at least one phoneme can be used as input information of the initial mapping module, and the initial mapping module can be used to generate a priori variables; the sample duration can also be directly input into the initial mapping module, so as to use the initial mapping module to generate a priori latent variables based on at least one phoneme and the sample duration. In both cases, the loss calculated based on the priori latent variables and the posteriori latent variables is used to update the neural network in the initial model except the initial decoder.

[0184] In the case where at least one phoneme and a sample duration are used as input information of the initial mapping module, in the speech synthesis model obtained after training, the predicted duration generated by the trained duration predictor is used as input information of the mapping module.

[0185] In some embodiments, the use of the initial model to generate the synthesized speech corresponding to the sample text, that is, the above step S302, can be implemented as the following step S3021:

[0186] The synthesized speech is generated based on the sample fundamental frequency and the a posteriori latent variable using the initial decoder.

[0187] Here, the sample fundamental frequency corresponding to the sample speech and the posterior latent variable generated by the posterior encoder are used as input information of the initial decoder, and the initial decoder is used to generate synthetic speech.

[0188] Since the posterior latent variable is generated by the posterior encoder based on the sample speech, the posterior latent variable can be used as the true value label information corresponding to the sample text. In this way, using the posterior latent variable as the input of the initial decoder can improve the accuracy of the initial decoder input information.

[0189] Here, in the speech synthesis model obtained after training, the predicted pitch generated by the trained pitch predictor is used as input information of the decoder.

[0190] In some embodiments, the initial model is updated based on the loss between the sample text corresponding to the sample speech and the synthesized speech to obtain a trained speech synthesis model, that is, the above step S303 can be implemented as the following steps S3031 to S3032:

[0191] Step S3031, generating a sample spectrum corresponding to the sample speech and a predicted spectrum corresponding to the synthesized speech respectively;

[0192] Step S3032: updating the initial decoder based on the loss between the sample spectrum and the predicted spectrum until a preset convergence condition is met.

[0193] Here, the spectrum of speech refers to the representation of speech signals in the frequency domain, which describes the energy distribution or intensity of different frequency components in the speech signal. Based on the spectrum information of speech, the feature vector corresponding to the speech information can be constructed, thereby realizing the accurate identification of speech information.

[0194] In some embodiments, algorithms such as short-time Fourier transform (STFT), fast Fourier transform (FFT), and Mel-Frequency Cepstral Coefficients (MFCCs) can be used to generate a sample spectrum corresponding to the sample speech and a predicted spectrum corresponding to the synthesized speech, respectively.

[0195] In some embodiments, a preset loss function is used to calculate the loss between the predicted spectrum and the sample spectrum.

[0196] In some embodiments, the preset loss function may be a MES loss function, a MAE loss function, a KL divergence loss function, or the like.

[0197] In some embodiments, the preset convergence condition may include that the number of updates to the initial decoder reaches a preset number threshold; or may include that the loss between the predicted spectrum and the sample spectrum is less than a preset loss threshold.

[0198] In this way, since the synthesized speech is generated based on the sample fundamental frequency and the posterior latent variables, the loss calculated based on the synthesized speech and the sample speech can be used alone to update the initial decoder, thereby making the training process of the initial decoder more stable.

[0199] Next, combine Figure 4 , an embodiment of training an initial model using the speech synthesis model training method provided in this application is described.

[0200] Here, firstly, sample data including sample text and sample speech is prepared; wherein the sample speech has a corresponding sample fundamental frequency, and each phoneme in the sample text has corresponding sample duration information.

[0201] like Figure 4As shown, first, a sample text 401 and speaker information 400 corresponding to the sample text 401 are input into an initial text encoder 410 .

[0202] Here, the speaker information 400 is embedded information obtained by encoding the speaker's ID information, and is input into the initial text encoder 410 as an explicit feature.

[0203] The initial text encoder 410 performs embedding processing on the phoneme IDs of multiple phonemes in the sample text 401 to obtain a text vector representation 402 .

[0204] The initial text encoder 410 transmits the text vector representation 402 to the initial duration predictor 420 and the length regularization module 430 respectively.

[0205] The initial duration predictor 420 generates a predicted duration 403 corresponding to each phoneme based on the text vector representation 402 .

[0206] Afterwards, the MSE loss L1 between the predicted duration 403 and the sample duration 404 is calculated, and the loss L1 is used to update the initial duration predictor 420 until a preset convergence condition is met.

[0207] The sample duration 404 is input into the length regularization module 430 .

[0208] Here, after the model training is completed, the predicted duration generated by the trained duration predictor 420 is input into the length regularization module 430 .

[0209] The length regularization module 430 performs upsampling on the text vector representation 402 of multiple phonemes based on the phoneme duration to obtain multiple frame vector representations 405 corresponding to the multiple phonemes; then, the length regularization module 430 transmits the multiple frame vector representations 405 to the initial mapping module 440.

[0210] The initial mapping module 440 generates a covariance matrix and a mean sequence corresponding to the prior latent variables 406 based on the multiple frame vector representations 405 , and transmits the covariance matrix and the mean sequence to the Flow module 450 .

[0211] The sample speech 407 is input into the initial a posteriori encoder 460. Specifically, the linear spectrum corresponding to the sample speech 407 is input into the initial a posteriori encoder 460.

[0212] The initial posterior encoder 460 generates a covariance matrix and a mean sequence corresponding to the posterior latent variable 408 , and transmits the covariance matrix and the mean sequence to the Flow module 450 .

[0213] Based on the covariance matrix and mean sequence corresponding to the prior latent variable 406, the Flow module 450 is used to generate the prior latent variable 406; based on the covariance matrix and mean sequence corresponding to the posterior latent variable 408, the Flow module 450 is used to generate the posterior latent variable 408.

[0214] Here, the KL loss L2 between the mean and variance corresponding to the prior latent variable 406 and the mean and variance corresponding to the posterior latent variable 408 is calculated, and L2 is used to update the entire initial model except the initial decoder 480.

[0215] The Flow module 450 transmits the a posteriori latent variables 408 to the initial decoder 480 .

[0216] Here, the speaker information 400 and the posterior latent variable 408 are input into the initial pitch predictor 470 .

[0217] Based on the posterior latent variable 408 , the initial fundamental frequency predictor 470 generates a predicted fundamental frequency 409 corresponding to the sample text.

[0218] Here, the loss L3 between the predicted fundamental frequency 409 and the sample fundamental frequency 411 is calculated, and the initial fundamental frequency predictor 470 is updated using L3.

[0219] Thereafter, the sample base frequency 411 is input into the initial decoder 480 .

[0220] The initial decoder 480 generates synthesized speech 412 based on the sample fundamental frequency 411 and the a posteriori latent variable 408 .

[0221] Here, the sample Mel spectrum and the predicted Mel spectrum corresponding to the sample speech 407 and the synthesized speech 412 are determined respectively; the loss L4 between the sample Mel spectrum and the predicted Mel spectrum is calculated, and the loss L4 is used to update the initial decoder 480.

[0222] Next, combine Figure 5 , the training process of the initial decoder in the speech synthesis model training method provided in this application is explained.

[0223] like Figure 5 As shown, the initial decoder includes an initial sine transform module 510, an initial up-sampling module 520, an initial pre-convolution module 530, an initial down-sampling module 540 and an initial HiFiGan decoder 550; wherein,

[0224] The sample base frequency 501 is input into the initial sine transform module 510 .

[0225] Here, first, the VUV information corresponding to the sample fundamental frequency 501 is determined; then the initial sine transformation module 510 is used to perform a sinusoidal transformation on the sample fundamental frequency 501 and the corresponding VUV information to obtain a corresponding sinusoidal waveform representation.

[0226] The initial sine transform module 510 transmits the generated sinusoidal waveform representation to the initial upsampling module 520 .

[0227] Here, Gaussian noise 502 is input into the initial upsampling module 520 .

[0228] The initial up-sampling module 520 performs up-sampling on the sinusoidal waveform and Gaussian noise 502 to obtain up-sampled feature information.

[0229] The initial up-sampling module 520 transmits the up-sampled feature information to the initial pre-convolution module 530 .

[0230] The initial pre-convolution module 530 performs global feature extraction on the up-sampled feature information to obtain corresponding global features. Afterwards, the initial pre-convolution module 530 transmits the global features to the initial down-sampling module 540.

[0231] The initial downsampling module 540 performs downsampling on the global features to obtain downsampled feature information. Afterwards, the initial downsampling module 540 transmits the downsampled feature information to the initial HiFiGan decoder 550.

[0232] Here, the posterior latent variables 503 are input into the initial HiFiGan decoder 550 .

[0233] The initial HiFiGan decoder 550 generates synthesized speech 504 using the posterior latent variables 503 and the downsampled feature information.

[0234] Thereafter, based on the loss between the sample speech and the synthesized speech 504, the initial decoder is updated.

[0235] It can be seen from the above embodiments that the speech synthesis model training method provided in the present application, by adopting the sample fundamental frequency corresponding to the input sample text and the sample duration corresponding to multiple phonemes in the sample text, separately trains the initial duration predictor, initial fundamental frequency predictor, initial decoder and other networks, so that the model training stage can be more stable and the rhythm can be better.

[0236] Based on the foregoing embodiments, the present application provides a speech synthesis device, which includes the units included and the modules included in the units, which can be implemented by a processor in a computer device; of course, it can also be implemented by a specific logic circuit; in the implementation process, the processor can be a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP) or a field programmable gate array (FPGA), etc.

[0237] Figure 6 A schematic diagram of the structure of a speech synthesis device provided in this application is shown in FIG. Figure 6 As shown, the speech synthesis device 600 includes: a first determination module 610, a second determination module 620 and a first generation module 630, wherein:

[0238] A first determination module 610, used to determine a latent variable corresponding to the text information;

[0239] A second determination module 620, configured to determine fundamental frequency information corresponding to the text information based on the latent variable;

[0240] The first generating module 630 is used to generate the synthesized speech corresponding to the text information based on the latent variable and the fundamental frequency information.

[0241] In some embodiments, the first determining module 610 is used to:

[0242] Determining a phoneme duration corresponding to at least one phoneme in the text information;

[0243] Based on the phoneme duration, upsampling the text vector representation corresponding to the at least one phoneme to obtain at least one frame vector representation corresponding to the at least one phoneme;

[0244] The latent variable is determined based on the at least one frame vector representation.

[0245] In some embodiments, the first generating module 630 is used to:

[0246] Based on the baseband information and the noise information, generating baseband embedding information;

[0247] The synthesized speech is generated based on the fundamental frequency embedding information and the latent variable.

[0248] In some embodiments, the first generating module 630 is used to:

[0249] Performing a sine transform on the baseband information to obtain a sine wave representation;

[0250] Baseband embedding information is generated based on the sinusoidal wave representation and the noise information.

[0251] Based on the foregoing embodiments, the present application also provides a speech synthesis model training device, which includes the units included and the modules included in the units, which can be implemented by a processor in a computer device; of course, it can also be implemented by a specific logic circuit; in the implementation process, the processor can be a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP) or a field programmable gate array (FPGA), etc.

[0252] Figure 7 A schematic diagram of the structure of a speech synthesis model training device provided in this application, such as Figure 7 As shown, the speech synthesis model training device 700 includes: a third determination module 710, a second generation module 720 and a first update module 730, wherein:

[0253] The third determination module 710 is used to determine sample data and an initial model; the sample data includes sample text and sample speech corresponding to the sample text;

[0254] The second generation module 720 is used to generate the synthesized speech corresponding to the sample text using the initial model; wherein the initial model includes an initial fundamental frequency predictor and an initial decoder; the initial fundamental frequency predictor is used to predict the predicted fundamental frequency information corresponding to the sample data; the initial decoder is used to generate the corresponding synthesized speech based on the fundamental frequency information and latent variables corresponding to the sample data;

[0255] The first updating module 730 is used to update the initial model based on the loss between the sample speech and the synthesized speech to obtain a trained speech synthesis model.

[0256] In some embodiments, the initial model includes an initial a priori encoder and an initial a posteriori encoder;

[0257] The device 700 further includes:

[0258] The third generating module 721 is used to generate a priori latent variables corresponding to the sample text using the initial a priori encoder;

[0259] A fourth generating module 722, configured to generate a posteriori latent variables corresponding to the sample text based on the sample speech and using the initial posteriori encoder;

[0260] The second updating module 740 is used to update the initial a priori encoder and the initial a posteriori encoder based on the loss between the a priori latent variable and the a posteriori latent variable until a preset convergence condition is met.

[0261] In some embodiments, the sample text has a corresponding sample fundamental frequency; the apparatus 700 further includes:

[0262] A fifth generating module 723 is used to generate a predicted fundamental frequency based on the a posteriori latent variable and using the initial fundamental frequency predictor;

[0263] The third updating module 750 is configured to update the initial fundamental frequency predictor based on the loss between the predicted fundamental frequency and the sample fundamental frequency until a preset convergence condition is met.

[0264] In some embodiments, the initial a priori encoder includes an initial duration predictor and an initial mapping module; the sample text includes at least one phoneme; each of the phonemes has a sample duration; the third generation module 721 is used to:

[0265] Using the initial duration predictor, generating a predicted duration of at least one phoneme in the sample text;

[0266] Based on the at least one phoneme and the predicted duration, the prior latent variable is generated using the initial mapping module; or, based on the at least one phoneme and the sample duration, the prior latent variable is generated using the initial mapping module.

[0267] In some embodiments, the initial duration predictor is updated based on the loss between the predicted duration and the sample duration until a preset convergence condition is met.

[0268] In some embodiments, the second generation module 720 is used to generate the synthesized speech based on the sample fundamental frequency and the posterior latent variable using the initial decoder.

[0269] In some embodiments, the first updating module 730 is used to:

[0270] Respectively generating a sample spectrum corresponding to the sample speech and a predicted spectrum corresponding to the synthesized speech;

[0271] Based on the loss between the sample spectrum and the predicted spectrum, the initial decoder is updated until a preset convergence condition is met.

[0272] The description of the above device embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiment. For technical details not disclosed in the device embodiment of the present application, please refer to the description of the method embodiment of the present application for understanding.

[0273] It should be noted that in the embodiment of the present application, if the above-mentioned speech synthesis method and speech synthesis model training method are implemented in the form of a software function module and sold or used as an independent product, they can also be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, which is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a disk or an optical disk. In this way, the embodiment of the present application is not limited to any specific hardware, software or firmware, or any combination of hardware, software and firmware.

[0274] An embodiment of the present application provides a computer device, including a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the program, some or all of the steps in the above method are implemented.

[0275] The embodiment of the present application provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, some or all of the steps in the above method are implemented. The computer-readable storage medium can be transient or non-transient.

[0276] An embodiment of the present application provides a computer program, including a computer-readable code. When the computer-readable code is run in a computer device, a processor in the computer device executes some or all of the steps for implementing the above method.

[0277] The embodiment of the present application provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and when the computer program is read and executed by a computer, some or all of the steps in the above method are implemented. The computer program product can be implemented specifically by hardware, software or a combination thereof. In some embodiments, the computer program product is specifically embodied as a computer storage medium, and in other embodiments, the computer program product is specifically embodied as a software product, such as a software development kit (SDK) and the like.

[0278] It should be noted here that the description of the various embodiments above tends to emphasize the differences between the various embodiments, and the same or similar aspects can be referenced to each other. The description of the above device, storage medium, computer program and computer program product embodiments is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the embodiments of the device, storage medium, computer program and computer program product of this application, please refer to the description of the method embodiment of this application for understanding.

[0279] It should be noted that Figure 8 A schematic diagram of a hardware entity of an electronic device in this application, such as Figure 8 As shown, the hardware entity of the electronic device 800 includes: a processor 801, a communication interface 802 and a memory 803, wherein:

[0280] The processor 801 generally controls the overall operation of the electronic device 800 .

[0281] The communication interface 802 enables the electronic device to communicate with other terminals or servers through a network.

[0282] The memory 803 is configured to store instructions and applications executable by the processor 801, and can also cache data to be processed or processed by the processor 801 and each module in the electronic device 800 (for example, image data, audio data, voice communication data, and video communication data), which can be implemented by flash memory (FLASH) or random access memory (Random Access Memory, RAM). Data transmission can be performed between the processor 801, the communication interface 802, and the memory 803 through the bus 804.

[0283] It should be understood that "one embodiment" or "an embodiment" mentioned throughout the specification means that specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the present application, the size of the serial number of each step / process mentioned above does not mean the order of execution, and the execution order of each step / process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application. The serial numbers of the embodiments of the present application mentioned above are for description only and do not represent the advantages and disadvantages of the embodiments.

[0284] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0285] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0286] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0287] In addition, all functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may be a separate unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0288] A person skilled in the art can understand that all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM), magnetic disks or optical disks, etc., various media that can store program codes.

[0289] Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can essentially or in other words, the part that contributes to the relevant technology can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0290] The above is only an implementation method of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application.

Claims

1. A speech synthesis method, characterized in that: include: Determine the latent variables corresponding to the text information; Based on the latent variable, determining fundamental frequency information corresponding to the text information; Based on the latent variable and the fundamental frequency information, a synthesized speech corresponding to the text information is generated.

2. The method according to claim 1, characterized in that The determining of the hidden variable corresponding to the text information includes: Determining a phoneme duration corresponding to at least one phoneme in the text information; Based on the phoneme duration, upsampling the text vector representation corresponding to the at least one phoneme to obtain at least one frame vector representation corresponding to the at least one phoneme; The latent variable is determined based on the at least one frame vector representation.

3. The method according to claim 1, characterized in that The step of generating a synthesized speech corresponding to the text information based on the latent variable and the fundamental frequency information includes: Based on the baseband information and the noise information, generating baseband embedding information; The synthesized speech is generated based on the fundamental frequency embedding information and the latent variable.

4. The method according to claim 3, characterized in that: The generating baseband embedding information based on the baseband information and the noise information includes: Performing a sine transform on the baseband information to obtain a sine wave representation; Baseband embedding information is generated based on the sinusoidal wave representation and the noise information.

5. A speech synthesis model training method, characterized in that: include: Determine sample data and an initial model; the sample data includes sample text and sample speech corresponding to the sample text; Using the initial model, generating a synthesized speech corresponding to the sample text; wherein the initial model includes an initial fundamental frequency predictor and an initial decoder; the initial fundamental frequency predictor is used to predict the predicted fundamental frequency information corresponding to the sample data; the initial decoder is used to generate the corresponding synthesized speech based on the fundamental frequency information and latent variables corresponding to the sample data; Based on the loss between the sample speech and the synthesized speech, the initial model is updated to obtain a trained speech synthesis model.

6. The method according to claim 5, characterized in that The initial model includes an initial a priori encoder and an initial a posteriori encoder; Before using the initial model to generate the synthesized speech corresponding to the sample speech, the method further includes: Using the initial a priori encoder, generating a priori latent variables corresponding to the sample text; Based on the sample speech, using the initial posterior encoder, generating a posterior latent variable corresponding to the sample text; Based on the loss between the priori latent variable and the a posteriori latent variable, the initial a priori encoder and the initial a posteriori encoder are updated until a preset convergence condition is met.

7. The method according to claim 6, characterized in that The sample text has a corresponding sample fundamental frequency; the method further includes: Based on the a posteriori latent variables, using the initial fundamental frequency predictor, generating a predicted fundamental frequency; Based on the loss between the predicted fundamental frequency and the sample fundamental frequency, the initial fundamental frequency predictor is updated until a preset convergence condition is met.

8. The method according to claim 7, characterized in that The initial a priori encoder includes an initial duration predictor and an initial mapping module; the sample text includes at least one phoneme; each of the phonemes has a sample duration; The step of using the initial a priori encoder to generate a priori latent variables corresponding to the sample text includes: Using the initial duration predictor, generating a predicted duration of at least one phoneme in the sample text; Based on the at least one phoneme and the predicted duration, the prior latent variable is generated using the initial mapping module; or, based on the at least one phoneme and the sample duration, the prior latent variable is generated using the initial mapping module.

9. The method according to claim 8, characterized in that Also includes: Based on the loss between the predicted duration and the sample duration, the initial duration predictor is updated until a preset convergence condition is met.

10. The method according to claim 6, characterized in that The step of using the initial model to generate a synthesized speech corresponding to the sample text includes: The synthesized speech is generated based on the sample fundamental frequency and the a posteriori latent variable using the initial decoder.

11. The method according to claim 10, characterized in that The updating of the initial model based on the loss between the sample text corresponding to the sample speech and the synthesized speech to obtain a trained speech synthesis model includes: Respectively generating a sample spectrum corresponding to the sample speech and a predicted spectrum corresponding to the synthesized speech; Based on the loss between the sample spectrum and the predicted spectrum, the initial decoder is updated until a preset convergence condition is met.

12. A speech synthesis device, characterized in that: The device comprises: A first determination module is used to determine the latent variables corresponding to the text information; A second determination module, used to determine the voice fundamental frequency information corresponding to the text information based on the latent variable; The first generating module is used to generate the speech information corresponding to the text information based on the latent variable and the speech fundamental frequency information.

13. A speech synthesis model training device, characterized in that: include: A third determination module is used to determine sample data and an initial model; The sample data has corresponding sample speech; The initial model includes an initial pitch predictor and an initial decoder; The initial fundamental frequency predictor is used to predict the voice fundamental frequency information corresponding to the text information, so that the initial decoder generates the corresponding voice information based on the voice fundamental frequency information; A second generation module is used to generate a synthetic speech corresponding to the sample text using the initial model; The first updating module is used to update the initial model based on the loss between the sample text corresponding to the sample speech and the synthesized speech to obtain a trained speech synthesis model.

14. A computer device comprising a memory and a processor, wherein the memory stores a computer program executable on the processor, wherein: When the processor executes the program, the steps in the method according to any one of claims 1 to 4 or claims 5 to 11 are implemented.

15. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 4 or claims 5 to 11 are implemented.

16. A computer program product, comprising a non-transitory computer-readable storage medium storing a computer program, wherein when the computer program is read and executed by a computer, the steps of the method according to any one of claims 1 to 4 or claims 5 to 11 are implemented.

Images