Speech synthesis method, speech synthesis device and readable storage medium

Abstract

The present application relates to the field of computer technology, and discloses a speech synthesis method, a speech synthesis device, and a readable storage medium. The method comprises: encoding text information to obtain text features corresponding to the text information; processing the text features based on a rhythm prediction model and local rhythm change parameters to obtain random rhythm features corresponding to the text features; wherein the random rhythm features include random fundamental frequency features, random energy features, and random duration features; and generating speech information corresponding to the text information based on the random rhythm features and text features. The present application generates speech information based on random fundamental frequency features, random energy features, and random duration features, thereby improving the emotional richness and naturalness of the generated speech information.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a speech synthesis method, a speech synthesis device and a readable storage medium. Background Art

[0002] In recent years, with the advancement of artificial intelligence technology, speech synthesis technology (Text to Speech) has also developed rapidly. Speech synthesis technology can convert text data into speech data based on the timbre, pitch, identity, emotion, accent, etc. selected by the user. Speech data contains rich and variable rhythmic information. How to make the rhythm of the synthesized speech data natural and varied is an important task in the field of speech synthesis. Summary of the invention

[0003] In order to solve the above problems, the embodiment of the present application provides a speech synthesis method, which can improve the emotional richness and naturalness of the generated speech information. Specifically, the embodiment of the present application discloses the following technical solutions:

[0004] A first aspect of an embodiment of the present application provides a speech synthesis method, which includes: encoding text information to obtain text features corresponding to the above text information; processing the above text features based on a rhythm prediction model and local rhythm change parameters to obtain random rhythm features corresponding to the above text features; wherein the above random rhythm features include random fundamental frequency features, random energy features and random duration features; and generating speech information corresponding to the above text information based on the above random rhythm features and the above text features.

[0005] In some embodiments, the above-mentioned text features are processed based on the rhythm prediction model and the local rhythm change parameters to obtain the random rhythm features corresponding to the above-mentioned text features, including: processing the above-mentioned text features based on the random rhythm predictor in the above-mentioned rhythm prediction model and the above-mentioned local rhythm change parameters to obtain the above-mentioned random rhythm features; wherein the above-mentioned random rhythm predictor includes a random fundamental frequency predictor, a random energy predictor and a random duration predictor.

[0006] In some embodiments, the above-mentioned text features are processed based on the random prosody predictor and the above-mentioned local prosody change parameters in the above-mentioned prosody prediction model to obtain the above-mentioned random prosody features, including: performing noise processing on the above-mentioned text features based on the above-mentioned random prosody predictor and the above-mentioned local prosody change parameters to obtain the noisy text features; performing noise reduction processing on the above-mentioned noisy text features based on the above-mentioned random prosody predictor and the above-mentioned local prosody change parameters to obtain the above-mentioned random prosody features.

[0007] In some embodiments, the above-mentioned processing of the above-mentioned text features based on the random prosody predictor and the above-mentioned local prosody change parameters in the above-mentioned prosody prediction model to obtain the above-mentioned random prosody features includes: processing the above-mentioned text features based on the above-mentioned random fundamental frequency predictor and the above-mentioned local prosody change parameters to obtain the above-mentioned random fundamental frequency features; processing the above-mentioned text features and the above-mentioned random fundamental frequency features based on the above-mentioned random energy predictor and the above-mentioned local prosody change parameters to obtain the above-mentioned random energy features; processing the above-mentioned text features, the above-mentioned random fundamental frequency features and the above-mentioned random energy features based on the above-mentioned random duration predictor and the above-mentioned local prosody change parameters to obtain the above-mentioned random duration features; determining the above-mentioned random prosody features based on the above-mentioned random fundamental frequency features, the above-mentioned random energy features and the above-mentioned random duration features.

[0008] In some embodiments, the method further includes: processing the text features based on the fixed prosody predictor in the prosody prediction model to obtain fixed prosody features corresponding to the text features; wherein the fixed prosody predictor includes a fixed fundamental frequency predictor, a fixed energy predictor and a fixed duration predictor; the fixed prosody features include fixed fundamental frequency features, fixed energy features and fixed duration features; the generating speech information corresponding to the text information based on the random prosody features and the text features includes: generating the speech information based on the random prosody features, the fixed prosody features and the text features.

[0009] In some embodiments, the above-mentioned voice information is generated according to the above-mentioned random prosodic features, the above-mentioned fixed prosodic features and the above-mentioned text features, including: determining the target prosodic features according to the first control coefficient, the second control coefficient, the above-mentioned random prosodic features and the above-mentioned fixed prosodic features; wherein the above-mentioned first control coefficient is used to determine the weight of the above-mentioned random prosodic features, and the above-mentioned second control coefficient is used to determine the weight of the above-mentioned fixed prosodic features; and generating the above-mentioned voice information according to the above-mentioned target prosodic features and the above-mentioned text features.

[0010] In some embodiments, generating the voice information according to the random prosodic features, the fixed prosodic features and the text features includes: determining voice features according to the random prosodic features, the fixed prosodic features and the text features; and generating the voice information according to the voice features.

[0011] In some embodiments, the method further includes: obtaining sample speech information; obtaining sample fundamental frequency information, sample energy information and sample duration information corresponding to the sample speech information, and encoding the sample fundamental frequency information, the sample energy information and the sample duration information to obtain sample prosody features; wherein the sample prosody features include sample fundamental frequency features, sample energy features and sample duration features; determining a random prosody loss function corresponding to the random prosody predictor according to the sample prosody features and the random prosody features, and optimizing the random prosody predictor according to a first optimization parameter of the random prosody loss function; wherein the random prosody loss function includes a random fundamental frequency loss function, a random energy loss function and a random duration loss function; determining a fixed prosody loss function corresponding to the fixed prosody predictor according to the sample prosody features and the fixed prosody features, and optimizing the fixed prosody predictor according to a second optimization parameter of the fixed prosody loss function; wherein the fixed prosody loss function includes a fixed fundamental frequency loss function, a fixed energy loss function and a fixed duration loss function.

[0012] In some embodiments, the above-mentioned encoding processing of the above-mentioned text information to obtain text features corresponding to the above-mentioned text information includes: encoding the above-mentioned text information based on a first encoder to obtain character-level features corresponding to the above-mentioned text information; encoding the above-mentioned text information based on a second encoder to obtain phoneme-level features corresponding to the above-mentioned text information; encoding the features obtained by adding the above-mentioned character-level features and the above-mentioned phoneme-level features based on a third encoder to obtain the above-mentioned text features.

[0013] A second aspect of an embodiment of the present application provides a speech synthesis device, which includes: an encoding module, configured to obtain text information, and encode the above text information to obtain text features corresponding to the above text information; a random prosody prediction module, configured to process the above text features based on a prosody prediction model and local prosody change parameters to obtain random prosody features corresponding to the above text features; wherein the above random prosody features include random fundamental frequency features, random energy features and random duration features; a generation module, configured to generate speech information corresponding to the above text information based on the above random prosody features and the above text features.

[0014] A third aspect of an embodiment of the present application provides a computer-readable storage medium, which stores computer program instructions. When a computer reads the instructions, the speech synthesis method described in the first aspect is executed.

[0015] A fourth aspect of an embodiment of the present application provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer executes the speech synthesis method described in the first aspect above.

[0016] In the speech synthesis method provided in the embodiment of the present application, the text information can be encoded and processed to obtain text features corresponding to the text information, and the text features can be processed based on the rhythm prediction model and the local rhythm change parameters to obtain random rhythm features corresponding to the text features. Among them, the random rhythm features include random fundamental frequency features, random energy features and random duration features. Afterwards, the speech information corresponding to the text information is generated according to the random rhythm features and the text features. Applying the technical solution provided by the present application, the random fundamental frequency features, random energy features and random duration features of the text features can be predicted, and the speech information corresponding to the text information can be generated based on the random fundamental frequency features, random energy features and random duration features, thereby improving the emotional richness and naturalness of the generated response speech. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 A flowchart of a speech synthesis method provided in some embodiments of the present application;

[0019] Figure 2 A schematic diagram of encoding text information provided in some embodiments of the present application;

[0020] Figure 3 A schematic diagram of a diffusion model provided for some embodiments of the present application;

[0021] Figure 4 A schematic diagram of a random rhythm predictor provided in some embodiments of the present application;

[0022] Figure 5 A flowchart for training a random prosody predictor provided for some embodiments of the present application;

[0023] Figure 6 A schematic diagram of a VITS model provided for some embodiments of the present application;

[0024] Figure 7A schematic diagram of a speech synthesis device provided in some embodiments of the present application;

[0025] Figure 8 A schematic diagram of an electronic device according to some embodiments of the present application. DETAILED DESCRIPTION

[0026] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present invention and to make the above-mentioned purposes, features and advantages of the embodiments of the present invention more obvious and understandable, the technical solutions in the embodiments of the present invention are further described in detail below with reference to the accompanying drawings.

[0027] Existing speech synthesis methods usually predict the prosodic features of text information based on the VITS model (Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech), and synthesize speech information based on the text information and prosodic features. However, the VITS model only predicts the duration features based on the random duration predictor, and the prosodic features are not only reflected in the duration, but also in multiple phonemes such as fundamental frequency and energy. Therefore, the VITS model ignores the complex information in the rhythm and the interactivity between these information, and lacks fine-grained modeling of the prosodic features. At the same time, the flow-based models in the VITS model have weak noise resistance. Therefore, the random duration predictor modeled based on the flow model will become extremely unstable and difficult to train after the introduction of noise information. The duration control method of only increasing or decreasing the noise also causes the VITS model to lack the control ability when predicting the duration features.

[0028] Based on the above technical problems, the present application provides a speech synthesis method, a speech synthesis device and a storage medium, which can predict the character-level prosodic features (including fundamental frequency, energy, duration and other features) of text information and control the degree of prosodic variation, thereby improving the prosodic richness and naturalness of speech information. The speech synthesis method provided by the present application is described in detail below in conjunction with the accompanying drawings.

[0029] Figure 1 A flowchart of a speech synthesis method provided in some embodiments of the present application. Figure 1 As shown, the speech synthesis method may include steps 110 to 130.

[0030] Step 110: Encode the text information to obtain text features corresponding to the text information.

[0031] refer to Figure 2 A schematic diagram of encoding text information is shown in FIG. Figure 2 As shown, the text information can be encoded based on the first encoder (such as the Bert pre-trained model, etc.) to obtain the character-level features corresponding to the text information. The pre-trained Bert language model can obtain the complex relationship between words and sentences in the text information, and generate character-level features corresponding to the text information. The character-level features can reflect complex rhythmic information, thereby effectively optimizing the rhythmic effect of the synthesized voice information. In addition, the text information can be encoded based on the second encoder (such as an acoustic model based on deep learning, a speech recognition model, etc.) to obtain the phoneme-level features corresponding to the text information. The character-level features are then mapped through the embedding layer (Embedding) and converted to the same dimension as the phoneme-level features through a convolutional network; finally, the features obtained by adding the two are encoded based on the third encoder (i.e., the text encoder) to obtain the text features corresponding to the text information.

[0032] It should be noted that the specific types of the first encoder, the second encoder and the third encoder are not limited in the embodiments of the present application.

[0033] Step 120 , processing the text features based on the prosody prediction model and the local prosody change parameters to obtain random prosody features corresponding to the text features.

[0034] In some embodiments, the prosody prediction model includes a random prosody predictor and a fixed prosody predictor, wherein the random prosody predictor includes a random fundamental frequency predictor, a random energy predictor, and a random duration predictor; and the fixed prosody predictor includes a fixed fundamental frequency predictor, a fixed energy predictor, and a fixed duration predictor.

[0035] The rhythm prediction model can model fine-grained emotions (i.e., character-level rhythm), thereby obtaining random rhythm features that are richer and more random in emotional expression, as well as fixed rhythm features that are more stable in emotional expression. Among them, random rhythm features include random fundamental frequency features, random energy features, and random duration features; fixed rhythm features include fixed fundamental frequency features, fixed energy features, and fixed duration features.

[0036] In some examples, the random prosody predictor can be built based on Denoising Diffusion Probabilistic Models (DDPM). Figure 3 A schematic diagram of a diffusion model is shown in Figure . Figure 3 As shown in the figure, the diffusion model is a generative model based on the forward denoising process and the reverse denoising process. The forward denoising process converts the features into Gradually convert to a white noise The reverse denoising process converts this white noise into Gradually transform into features The following describes in detail the training process of the random prosody predictor and the process of determining the random prosody features based on the random prosody predictor.

[0037] refer to Figure 4 A schematic diagram of a random rhythm predictor is shown in FIG. Figure 4 As shown, the process of training the random prosody predictor and determining the random prosody features based on the random prosody predictor can be performed simultaneously, that is, the training process and the inference process of the random prosody predictor can be performed simultaneously.

[0038] In some embodiments, the text feature can be processed based on a random prosody predictor and a local prosody change parameter (i.e., a time step, or a diffusion number) to obtain a random prosody feature. Exemplarily, the text feature can be processed based on a random fundamental frequency predictor and a local prosody change parameter to obtain a random fundamental frequency feature corresponding to the text feature; then, the text feature predicted by the fundamental frequency can be processed based on a random energy predictor and a local prosody change parameter to obtain a random energy feature corresponding to the text feature; then, the text feature predicted by the fundamental frequency and energy can be processed based on a random duration predictor and a local prosody change parameter to obtain a random duration feature corresponding to the text feature.

[0039] Among them, the process of processing text features based on random fundamental frequency predictor, random energy predictor and random duration predictor to obtain random fundamental frequency features, random energy features and random duration features respectively is the same, that is, firstly performing noise processing on the text features according to the local rhythm change parameters to obtain the noisy text features; then performing noise reduction processing on the noisy text features according to the local rhythm change parameters to obtain the text features containing random rhythm features. The specific process and related formulas can be referred to the following description:

[0040] For example, when the text feature is subjected to noise processing, the text feature may be subjected to gradual noise processing according to the local rhythm variation parameter, that is, the text feature After T time steps of forward diffusion, the noisy text features are obtained. The time step can be set according to the actual situation. The implementation of the forward noise addition process can refer to formula (1):

[0041] (1)

[0042] in, is the variance parameter of the noise corresponding to the tth time step, It can increase with the increase of time step T; is the identity matrix.

[0043] For example, after the noise is added, the features of the text after the noise is added are analyzed. Perform reverse denoising. The reverse denoising process is used to denoise the text features after adding noise. Sampling by time step, through the neural network Gradually reduce noise and obtain the denoised text features , that is, the text features containing random rhythmic features. The implementation of the reverse denoising process can refer to formula (2):

[0044] (2)

[0045] in, is the noise factor, It is a neural network used to predict the noise corresponding to time step t; the neural network can be based on the U-shaped convolutional neural network (U-Net), while adding residual connection and skip connection mechanisms to the text features through a multi-layer convolutional network It performs multiple downsampling and then multiple upsampling to restore it, and adds a transformer layer in the middle of up and down sampling for self-supervision to improve the expression ability.

[0046] It should be noted that by increasing or decreasing the time step of the random prosody predictor, the emotional expression effect of the random prosody feature can be increased or decreased accordingly. For example, increasing the time step can make the random prosody predictor more powerful in noise reduction, and the output random prosody feature has higher accuracy and more significant effect on emotional expression.

[0047] Figure 5 A flowchart of training a random prosody predictor and a fixed prosody predictor is provided for some embodiments of the present application, such as Figure 5 As shown, the method includes steps 510 to 540.

[0048] Step 510, obtaining sample voice information.

[0049] In some embodiments, at least one sample speech may be obtained from a sample speech database.

[0050] Step 520, obtaining sample fundamental frequency information, sample energy information and sample duration information corresponding to the sample speech information, and encoding the sample fundamental frequency information, sample energy information and sample duration information to obtain sample prosodic features.

[0051] In some embodiments, the sample prosodic features include sample fundamental frequency features, sample energy features and sample duration features, and the sample fundamental frequency features, sample energy features and sample duration features are all used to characterize the local prosodic features of the sample (i.e., the prosodic features of the sample at the character level).

[0052] Exemplarily, the pysptk.sptk.rapt package can be used to extract the fundamental frequency of the sample speech information, and then the sample fundamental frequency information at the character level can be obtained by cropping, transforming and interpolating the sample speech information. Also, the librosa package is used to perform a short-time Fourier Transform (STFT) operation on the sample speech information to obtain amplitude and phase information, and the amplitude information is squared and squared for each column and then squared to obtain character-level sample energy information. Also, the sample speech information is forced to be aligned with the sample text information corresponding to the sample speech information through a monotonic alignment search (MAS), thereby obtaining the sample duration information corresponding to the sample speech data. Then, the obtained sample fundamental frequency information, sample energy information and sample duration information are encoded to obtain sample fundamental frequency features, sample energy features and sample duration features.

[0053] Step 530 : determining a random prosody loss function corresponding to the random prosody predictor according to the sample prosody features and the random prosody features, and optimizing the random prosody predictor according to a first optimization parameter of the random prosody loss function.

[0054] In some embodiments, the random prosody loss function includes a random fundamental frequency loss function, a random energy loss function, and a random duration loss function. Exemplarily, the random prosody loss function corresponding to the random prosody predictor can be determined according to the sample prosody feature and the random prosody feature output by each random prosody predictor, and the random prosody predictor can be optimized according to the first optimization parameter of the random prosody loss function.

[0055] For example, the random fundamental frequency loss function corresponding to the random fundamental frequency predictor can be determined based on the random fundamental frequency characteristics and sample fundamental frequency characteristics output by the random fundamental frequency predictor; and the random energy loss function corresponding to the random energy predictor can be determined based on the random energy characteristics and sample energy characteristics output by the random energy predictor; and the random duration loss function corresponding to the random duration predictor can be determined based on the random duration characteristics and sample duration characteristics output by the random duration predictor.

[0056] The determination of the random base frequency loss function, random energy loss function or random duration loss function can refer to formula (3). Noise addition and noise reduction are performed to obtain the predicted random rhythmic features Then, according to the sample rhythm characteristics and predicted random rhythmic features , determine the random rhythm loss function , and according to the first optimization parameter For random rhythm loss function The optimization is performed so that the predicted noise is as close as possible to the actual added noise.

[0057] (3)

[0058] in, is the estimate of the text feature corresponding to time step t, An estimate of the text features corresponding to the previous time step, is the first optimization parameter.

[0059] In some embodiments, in addition to determining the random prosody features corresponding to the text features based on the random prosody predictor, the text features can also be processed based on the fixed prosody predictor to obtain the fixed prosody features corresponding to the text features. Among them, the fixed fundamental frequency predictor / fixed energy predictor / fixed duration predictor can be a predictor including two layers of long short-term memory (LSTM) and one fully connected layer (Fully Connected Layer), which can make the output of the predictor have a relatively stable structure. After the text features are input into the fixed fundamental frequency predictor / fixed energy predictor / fixed duration predictor, the fixed fundamental frequency features, fixed energy features and fixed duration features corresponding to the text features can be obtained respectively. Among them, the formula for processing the text features based on the fixed fundamental frequency predictor / fixed energy predictor / fixed duration predictor to obtain the fixed fundamental frequency features / fixed energy features / fixed duration features corresponding to the text features can refer to formula (4):

[0060] (4)

[0061] in, is the text feature, Represents the text features through Convolutional Neural Networks (CNN) Extract fixed rhythmic features. is the diagonal covariance matrix, is the variance constant. (Right now ) is a Gaussian distribution, we can As the predicted value, or sample the predicted value from the generated Gaussian distribution .

[0062] Step 540 : determining a fixed prosody loss function corresponding to the fixed prosody predictor according to the sample prosody features and the fixed prosody features, and optimizing the fixed prosody predictor according to a second optimization parameter of the fixed prosody loss function.

[0063] In some embodiments, a fixed prosody loss function corresponding to the fixed prosody predictor can be determined based on the fixed prosody features and sample prosody features predicted by the fixed prosody predictor, and the fixed prosody predictor can be optimized according to the second optimization parameter in the fixed prosody loss function. The fixed prosody loss function includes a fixed fundamental frequency loss function, a fixed energy loss function, and a fixed duration loss function. The formula for determining the fixed prosody loss function can refer to formula (5):

[0064] (5)

[0065] in, is a fixed rhythm loss function; is the number of predictions; is the local rhythmic feature of the i-th sample; is the fixed rhythm feature of the ith prediction, as in formula (4): .

[0066] Exemplarily, the fixed prosody predictor can be optimized using algorithms such as gradient descent. When optimizing the fixed prosody predictor using the gradient descent algorithm, the formula For the second optimization parameter Optimize. is the learning rate, is a fixed rhythm loss function For the second optimization parameter gradient.

[0067] After determining the above random rhythm loss function and the above fixed rhythm loss function, the loss function corresponding to the random rhythm predictor can be determined: It can be understood that optimizing the random prosody predictor and the fixed prosody predictor according to the random prosody loss function and the fixed prosody loss function is to optimize the prosody prediction model.

[0068] In some embodiments, the speech information is generated according to the random prosodic features, the fixed prosodic features and the text features, including: determining the target prosodic features according to the first control coefficient, the second control coefficient, the random prosodic features and the fixed prosodic features; wherein the first control coefficient is used to determine the weight of the random prosodic features, and the second control coefficient is used to determine the weight of the fixed prosodic features.

[0069] In some examples, after determining the random prosodic features and the fixed prosodic features corresponding to the text information, the weights of the random prosodic features and the fixed prosodic features can be adjusted respectively by using the first control coefficient corresponding to the random prosodic features and the second control coefficient corresponding to the fixed prosodic features, thereby adjusting the diversity of the prosody and the stability of the prosody respectively, and obtaining the target prosodic features corresponding to the text information. For example, the first control coefficient and the second control coefficient can be adjusted to appropriate values, so that the local prosody of the speech information subsequently generated based on the random prosodic features and the fixed prosodic features has both good variability and good stability.

[0070] Through the above scheme, the present application utilizes the sampling generation characteristics of the diffusion model itself within the distribution, so that the predicted random rhythmic features have both the overall characteristics within the distribution and the one-to-many randomness, so that different rhythms can be determined for the same text information. At the same time, controlling the time step corresponding to the noise reduction process can correspondingly control the degree of rhythm change, thereby improving the naturalness and controllability of the rhythm. In addition, while predicting the duration characteristics of the text features, the present application also adds the prediction of the fundamental frequency and energy characteristics of the text features, so that the character-level rhythmic features of the text information can be determined jointly according to the fundamental frequency, energy and duration characteristics, thereby improving the emotional richness and naturalness of the character-level rhythmic features. In addition, the present application predicts the fundamental frequency, energy and duration of the text features by a random rhythm predictor and a fixed rhythm predictor, and adjusts the weights of the two according to the control coefficient, so that the variability and stability of the rhythm can be effectively regulated.

[0071] Step 130: Generate voice information corresponding to the text information according to the random prosodic features and the text features.

[0072] In some embodiments, after determining the above-mentioned target prosodic features, the text features containing the target prosodic features (i.e., the target text features) can be input into the speech synthesis model. The speech synthesis model can determine the corresponding speech features based on the target text features, and generate speech information corresponding to the text information based on the speech features.

[0073] For example, the speech synthesis model may be a variational inference with adversarial learning for end-to-end text-to-speech (VITS) model. Figure 6 A schematic diagram of a VITS model is shown in FIG. Figure 6As shown, the target text features determined based on the prosody prediction model can be input into the stream model in the VITS model, so as to obtain the speech features corresponding to the target text features based on the stream model; then, the speech features are processed based on the decoder in the VITS model to convert the speech features into corresponding audio waveforms, so as to obtain the speech information corresponding to the text information.

[0074] In some embodiments, continue to refer to Figure 6 , training the VITS model can improve the quality and naturalness of the generated speech information. For the training process of the VITS model, please refer to the following example:

[0075] 1. Audio Reconstruction Loss

[0076] The audio reconstruction loss is used to measure the difference between the Mel spectrum generated by the VITS model and the true Mel spectrum. For example, the linear spectrum corresponding to the training sample speech can be extracted first, and the posterior encoder (PosteriorEncoder) in the VITS model can obtain the Mel spectrum corresponding to the training sample speech based on the linear spectrum corresponding to the training sample speech. (The mel spectrum is the training sample speech The real Mel spectrum of Output implicit features , and implicit features The corresponding mean and variance, and according to the implicit features The corresponding mean and variance generate the posterior distribution ; Finally, the implicit features are transformed into Decoded into estimated answering speech , and extract its Mel spectrum . Thus, the loss function corresponding to audio reconstruction is obtained Please refer to formula (6):

[0077] (6)

[0078] 2. KL Divergence Loss

[0079] KL discreteness loss is used to measure the posterior distribution and the conditional prior distribution The difference between the two. The loss function corresponding to KL divergence Refer to formula (7):

[0080] (7)

[0081] Among them, the conditional prior distribution is the target text feature output by the random prosody predictor; after the target text feature is improved in expressiveness through the flow model, it is combined with the above implicit feature output by the posterior encoder Perform alignment processing to obtain the alignment matrix .

[0082] 3. Adversarial Training Loss

[0083] Adversarial training loss is achieved by introducing a discriminator into the VITS model. To distinguish between real audio and synthesized audio generated by the decoder, thereby improving the quality of the generated response speech. The adversarial training loss consists of the following three parts:

[0084] a. Discriminator Loss

[0085] The discriminator loss is used to measure the discriminator For training sample speech and synthesized audio The discriminator loss corresponds to the loss function Refer to formula (8):

[0086] (8)

[0087] b. Generator Loss

[0088] The generator loss is used to measure the synthesized audio Discriminator Identified as training sample speech The loss function corresponding to the generator loss is Refer to formula (9):

[0089] (9)

[0090] c. Feature Matching Loss

[0091] Feature matching loss is used to measure the synthetic audio And training sample speech In the discriminator The difference between different levels of features. The loss function corresponding to the feature matching loss is Refer to formula (10):

[0092] (10)

[0093] in, is the number of layers of the discriminator network, is the discriminator The characteristics of the layer, For the The number of features of the layer.

[0094] Through the above scheme, the audio reconstruction quality, the alignment of the potential representation and the authenticity of the generated audio in the VITS model are comprehensively considered. The model is optimized through audio reconstruction loss, KL discreteness loss and adversarial training loss. By optimizing these loss functions, the VITS model can generate high-quality and natural speech information.

[0095] By applying the technical solution of the present application, the random prosodic features at the character level corresponding to the text information can be determined based on the diffusion model, so that the generated voice information has subtle emotional changes locally. At the same time, controlling the time step in the diffusion model can control the degree of rhythm change accordingly, so that the naturalness and controllability of the rhythm can be improved. In addition, while predicting the duration characteristics of the text features, the present application also adds the prediction of the fundamental frequency and energy characteristics of the text features, so that the character-level prosodic features of the text information can be jointly determined based on the fundamental frequency, energy and duration characteristics, thereby further improving the emotional richness and naturalness of the character-level prosodic features. In addition, the present application also predicts the fundamental frequency, energy and duration of the text features through a random prosodic predictor and a fixed prosodic predictor, and adjusts the weights of the two according to the control coefficient, so that the variability and stability of the rhythm can be effectively regulated.

[0096] Figure 7 Schematic diagram of a speech synthesis device provided in some embodiments of the present application. Figure 7 As shown, the speech synthesis device 700 includes an encoding module 701 , a random prosody prediction module 702 and a generation module 703 .

[0097] The encoding module 701 is configured to obtain text information, and perform encoding processing on the text information to obtain text features corresponding to the text information.

[0098] The random prosody prediction module 702 is configured to process text features based on the prosody prediction model and the local prosody change parameter to obtain random prosody features corresponding to the text features.

[0099] Among them, the random rhythmic features include random fundamental frequency features, random energy features and random duration features.

[0100] The generating module 703 is configured to generate voice information corresponding to the text information according to the random prosodic features and the text features.

[0101] In some embodiments, the random prosody prediction module 702 is specifically configured to: process text features based on a random prosody predictor and local prosody change parameters in a prosody prediction model to obtain random prosody features; wherein the random prosody predictor includes a random fundamental frequency predictor, a random energy predictor, and a random duration predictor.

[0102] In some embodiments, the random prosody prediction module 702 is specifically configured to: perform noise processing on the text features based on the random prosody predictor and the local prosody change parameters to obtain the noisy text features; perform noise reduction processing on the noisy text features based on the random prosody predictor and the local prosody change parameters to obtain the random prosody features.

[0103] In some embodiments, the random prosody prediction module 702 is specifically configured to: process text features based on a random fundamental frequency predictor and local prosody change parameters to obtain random fundamental frequency features; process text features and random fundamental frequency features based on a random energy predictor and local prosody change parameters to obtain random energy features; process text features, random fundamental frequency features and random energy features based on a random duration predictor and local prosody change parameters to obtain random duration features; determine random prosody features based on random fundamental frequency features, random energy features and random duration features.

[0104] like Figure 7 As shown, the speech synthesis apparatus 700 further includes a fixed prosody prediction module 704 .

[0105] In some embodiments, the fixed prosody prediction module 704 is configured to: process text features based on a fixed prosody predictor in a prosody prediction model to obtain fixed prosody features corresponding to the text features; wherein the fixed prosody predictor includes a fixed fundamental frequency predictor, a fixed energy predictor, and a fixed duration predictor; the fixed prosody features include fixed fundamental frequency features, fixed energy features, and fixed duration features; the generation module 703 is specifically configured to: generate speech information based on random prosody features, fixed prosody features, and text features.

[0106] In some embodiments, the generation module 703 is specifically configured to: determine the target prosodic feature based on the first control coefficient, the second control coefficient, the random prosodic feature and the fixed prosodic feature; wherein the first control coefficient is used to determine the weight of the random prosodic feature, and the second control coefficient is used to determine the weight of the fixed prosodic feature; generate speech information based on the target prosodic feature and the text feature.

[0107] In some embodiments, the generation module 703 is specifically configured to: determine speech features according to random prosodic features, fixed prosodic features and text features; and generate speech information according to the speech features.

[0108] like Figure 7 As shown, the speech synthesis device 700 further includes a training module 705 .

[0109] In some embodiments, the training module 705 is configured to: obtain sample speech information; obtain sample fundamental frequency information, sample energy information and sample duration information corresponding to the sample speech information, and encode the sample fundamental frequency information, sample energy information and sample duration information to obtain sample prosody features; wherein the sample prosody features include sample fundamental frequency features, sample energy features and sample duration features; determine the random prosody loss function corresponding to the random prosody predictor according to the sample prosody features and the random prosody features, and optimize the random prosody predictor according to the optimization parameters of the random prosody loss function; wherein the random prosody loss function includes the random fundamental frequency loss function, the random energy loss function and the random duration loss function; determine the fixed prosody loss function corresponding to the fixed prosody predictor according to the sample prosody features and the fixed prosody features, and optimize the fixed prosody predictor according to the second optimization parameters of the fixed prosody loss function; wherein the fixed prosody loss function includes the fixed fundamental frequency loss function, the fixed energy loss function and the fixed duration loss function.

[0110] In some embodiments, the encoding module 701 is specifically configured as follows: based on the first encoder, encode the text information to obtain character-level features corresponding to the text information; based on the second encoder, encode the text information to obtain phoneme-level features corresponding to the text information; based on the third encoder, encode the features obtained by adding character-level features and phoneme-level features to obtain text features.

[0111] Figure 8 A schematic diagram of an electronic device provided for some embodiments of the present application. In some embodiments, the electronic device includes one or more processors and a memory. The memory is configured to store one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the speech synthesis method in the above embodiment.

[0112] like Figure 8 As shown, the electronic device 800 includes: a processor (processor) 801 and a memory 802. Exemplarily, the electronic device 800 may also include: a communication interface (Communications Interface) 803 and a communication bus 804.

[0113] The processor 801, the memory 802 and the communication interface 803 communicate with each other via a communication bus 804. The communication interface 803 is used to communicate with other devices such as a client or a network element of another server.

[0114] In some embodiments, the processor 801 is used to execute the program 805, and specifically, the relevant steps in the above-mentioned speech synthesis method embodiment can be executed. Specifically, the program 805 can include program code, and the program code includes computer executable instructions.

[0115] Exemplarily, the processor 801 may be a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement some embodiments of the present application. The electronic device 800 may include one or more processors, which may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.

[0116] In some embodiments, the memory 802 is used to store the program 805. The memory 802 may include a high-speed RAM memory, and may also include a non-volatile memory (NVM), such as at least one disk memory.

[0117] Program 805 can be specifically called by processor 801 to enable electronic device 800 to perform speech synthesis method operations.

[0118] Some embodiments of the present application provide a computer-readable storage medium, which stores at least one executable instruction. When the executable instruction is executed on the electronic device 800, the electronic device 800 executes the speech synthesis method in the above embodiment.

[0119] The executable instructions can be specifically used to enable the electronic device 800 to perform speech synthesis method operations.

[0120] For example, the computer readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc (CD-ROM), a magnetic tape, a floppy disk, an optical data storage device, etc.

[0121] The beneficial effects that can be achieved by the computer-readable storage medium provided in some embodiments of the present application can be referred to the beneficial effects in the corresponding speech synthesis method provided above, and will not be repeated here.

[0122] It should be noted that, in the application, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "including a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0123] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0124] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, which can be embodied in any computer-readable medium for use by an instruction execution system, apparatus or device (such as a computer-based system, a system including a processor or other system that can fetch instructions from an instruction execution system, apparatus or device and execute instructions), or used in combination with these instruction execution systems, apparatuses or devices.

[0125] For the purposes of this specification, a "computer-readable medium" can be any apparatus that can contain, store, communicate, propagate, or transport the program for use by an instruction execution system, apparatus, or device or in conjunction with such instruction execution system, apparatus, or device.

[0126] More specific examples (a non-exhaustive list) of computer readable media include the following: an electrical connection having one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM).

[0127] In addition, the computer readable medium can even be a paper or other suitable medium on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other medium, and then editing, interpreting or processing in other suitable ways as necessary, and then storing it in the computer memory. It should be understood that various parts of the present application can be implemented in hardware, software, firmware or a combination thereof.

[0128] In the above embodiments, multiple steps or methods may be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it may be implemented by any one of the following technologies known in the art or a combination thereof: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0129] The above-described embodiments of the present application do not constitute a limitation on the protection scope of the present application.

Claims

1. A speech synthesis method, characterized in that: include: Encoding the text information to obtain text features corresponding to the text information; Performing noise processing on the text features based on a random prosody predictor and a local prosody change parameter in a prosody prediction model to obtain a noisy text feature; Based on the random prosody predictor and the local prosody change parameter, the noisy text feature is subjected to denoising to obtain a random prosody feature; wherein the random prosody predictor includes a random fundamental frequency predictor, a random energy predictor and a random duration predictor, and the random prosody feature includes a random fundamental frequency feature, a random energy feature and a random duration feature; The speech information corresponding to the text information is generated according to the random prosodic features and the text features.

2. The method according to claim 1, characterized in that The step of performing denoising on the noisy text features based on the random prosody predictor and the local prosody change parameter to obtain random prosody features includes: Performing denoising on the noisy text feature based on the random fundamental frequency predictor and the local prosody change parameter to obtain the random fundamental frequency feature; Performing denoising on the noisy text feature and the random fundamental frequency feature based on the random energy predictor and the local prosody change parameter to obtain the random energy feature; Based on the random duration predictor and the local rhythm change parameter, the noisy text feature, the random fundamental frequency feature and the random energy feature are subjected to denoising to obtain the random duration feature; The random rhythm feature is determined based on the random fundamental frequency feature, the random energy feature and the random duration feature.

3. The method according to claim 1 or 2, characterized in that: The method further comprises: The text feature is processed based on the fixed prosody predictor in the prosody prediction model to obtain a fixed prosody feature corresponding to the text feature; wherein the fixed prosody predictor includes a fixed fundamental frequency predictor, a fixed energy predictor and a fixed duration predictor; and the fixed prosody feature includes a fixed fundamental frequency feature, a fixed energy feature and a fixed duration feature; The step of generating voice information corresponding to the text information according to the random prosodic feature and the text feature includes: The speech information is generated according to the random prosodic feature, the fixed prosodic feature and the text feature.

4. The method according to claim 3, characterized in that The generating the voice information according to the random prosodic feature, the fixed prosodic feature and the text feature comprises: Determine a target prosodic feature according to a first control coefficient, a second control coefficient, the random prosodic feature and the fixed prosodic feature; wherein the first control coefficient is used to determine the weight of the random prosodic feature, and the second control coefficient is used to determine the weight of the fixed prosodic feature; The speech information is generated according to the target prosodic feature and the text feature.

5. The method according to claim 3, characterized in that: The generating the voice information according to the random prosodic feature, the fixed prosodic feature and the text feature comprises: Determining speech features according to the random prosodic features, the fixed prosodic features and the text features; The voice information is generated according to the voice feature.

6. The method according to claim 3, characterized in that The method further comprises: Obtain sample voice information; Acquire sample fundamental frequency information, sample energy information and sample duration information corresponding to the sample speech information, and encode the sample fundamental frequency information, the sample energy information and the sample duration information to obtain sample prosodic features; wherein the sample prosodic features include sample fundamental frequency features, sample energy features and sample duration features; Determine a random prosody loss function corresponding to the random prosody predictor according to the sample prosody feature and the random prosody feature, and optimize the random prosody predictor according to a first optimization parameter of the random prosody loss function; wherein the random prosody loss function includes a random fundamental frequency loss function, a random energy loss function and a random duration loss function; According to the sample prosody features and the fixed prosody features, a fixed prosody loss function corresponding to the fixed prosody predictor is determined, and the fixed prosody predictor is optimized according to a second optimization parameter of the fixed prosody loss function; wherein the fixed prosody loss function includes a fixed fundamental frequency loss function, a fixed energy loss function and a fixed duration loss function.

7. The method according to claim 1 or 2, characterized in that: The encoding process of the text information to obtain text features corresponding to the text information includes: Encoding the text information based on a first encoder to obtain character-level features corresponding to the text information; Encoding the text information based on a second encoder to obtain phoneme-level features corresponding to the text information; The feature obtained by adding the character-level feature and the phoneme-level feature is encoded based on a third encoder to obtain the text feature.

8. A speech synthesis device, characterized in that: include: The encoding module is configured to obtain text information and perform encoding processing on the text information to obtain text features corresponding to the text information; The random prosody prediction module is configured to perform noise processing on the text features based on the random prosody predictor and the local prosody change parameters in the prosody prediction model to obtain the noisy text features; and to perform noise reduction processing on the noisy text features based on the random prosody predictor and the local prosody change parameters to obtain random prosody features; wherein the random prosody predictor includes a random fundamental frequency predictor, a random energy predictor and a random duration predictor, and the random prosody features include random fundamental frequency features, random energy features and random duration features; The generating module is configured to generate the speech information corresponding to the text information according to the random prosodic features and the text features.

9. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor, the speech synthesis method according to any one of claims 1 to 7 is implemented.

10. A computer program product, characterized in that The computer program product comprises a computer program stored on a non-transitory computer-readable storage medium, wherein the computer program comprises program instructions, and when the program instructions are executed by a computer, the computer is caused to execute the speech synthesis method according to any one of claims 1 to 7.

Images