Self-adaptive emotional speech synthesis method based on complete end-to-end

Abstract

The invention discloses a self-adaptive emotion speech synthesis method based on complete end-to-end. The method comprises the following steps: firstly, acquiring an emotional voice data set; constructing a conditional variation auto-encoder network, training the conditional variation auto-encoder network by using the emotional speech data set to obtain the trained conditional variation auto-encoder network, and further constructing an adaptive emotional speech synthesis network; and finally, inputting a target text, generating phonemes corresponding to the target text, inputting the reference emotional speech and the phonemes corresponding to the target text into an adaptive emotional speech synthesis network, and outputting emotional synthesis speech which corresponds to the target text and is consistent with the emotional of the reference emotional speech by the network. According to the method, the synthesized voice can be endowed with richer and finer emotional colors only by inputting the reference emotional audio without the help of an emotional label, the emotional controllability with finer granularity generated by the voice is realized, and the naturalness and expressive force of emotional voice synthesis are improved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of artificial intelligence and emotional speech synthesis, and particularly relates to a completely end-to-end adaptive emotional speech synthesis method. BACKGROUND

[0002] Human speech is essentially expressive, and accurately and controllably conveying expressions from text has wide applications in human-computer interaction and audio content generation. The focus of the field of emotional speech synthesis is the presentation of emotional expression. The core of emotional speech synthesis is to use deep learning models and neural networks to simulate the characteristics and emotional expression of human speech. Traditional speech synthesis can usually only generate reading style voices, which lack emotional color, resulting in synthesized speech lacking vitality and appeal. In contrast, emotional speech synthesis generates voices with rich emotional dimensions, making synthesized voices more resonant with people, thereby improving the realism and emotional connection of communication.

[0003] Humans can speak with different emotions, which leads to a wide variety of conversations. However, it is difficult to accurately represent speech emotions. In addition, speech emotions themselves have super-fragmental characteristics and are complexly intertwined with various acoustic cues such as timbre, pitch, and rhythm. It is not easy to obtain emotional speech data. It is very difficult to record multiple sentences for a long time while maintaining emotions. In addition, due to the ambiguity of emotional labels, it is easy to observe samples with inconsistent emotional labels in open-source emotional speech data sets. Even if it is possible to find a correct emotional label, the expression of emotion is limited. Current research on emotional speech synthesis mainly focuses on imitating the average style of a specific emotional type. However, the style control of synthesized speech is often limited to discrete emotional categories, requiring training data recorded by the target speaker in the target style. Limited by the emotional categories of the emotional data set, most emotional speech synthesis models can only generate existing emotions in the data set, such as happiness, surprise, sadness, etc., but human emotions are diverse, which greatly limits the emotional diversity of generated speech.

[0004] In summary, the existing research has the following problems: human emotions are diverse, but it is difficult to accurately represent emotions because emotions themselves have super-fragmental characteristics and are complexly intertwined with various acoustic cues such as timbre, pitch, and rhythm. Due to the ambiguity of emotional labels, it is easy to observe samples with inconsistent emotional labels in open-source emotional speech data sets. Current research on emotional speech synthesis mainly focuses on imitating the average style of a specific emotional type, and the expression of emotion is limited, resulting in insufficient emotional diversity of generated speech. SUMMARY

[0005] In view of the defects and deficiencies in the background art, the present application provides a fully end-to-end adaptive emotional speech synthesis method.

[0006] In order to achieve the above technical purpose, the present application is realized by the following technical scheme:

[0007] One kind is based on the adaptive emotional speech synthesis method of fully end-to-end

[0008] S1: obtain an emotional speech data set;

[0009] S2: a fully end-to-end conditional variational autoencoder network is constructed, and the conditional variational autoencoder network is trained using the emotional speech data set to obtain a trained conditional variational autoencoder network; then, an adaptive emotional speech synthesis network is constructed based on the trained conditional variational autoencoder network;

[0010] S3: input the target text, generate the phoneme corresponding to the target text, input the reference emotional speech and the phoneme corresponding to the target text into the adaptive emotional speech synthesis network, and the network outputs the emotional synthesized speech corresponding to the target text and consistent with the emotion of the reference emotional speech.

[0011] The conditional variational autoencoder network includes a prior encoder module, a posterior encoder module and a decoder module, the prior encoder module and the posterior encoder module are connected, the posterior encoder module and the decoder module are connected, in the training process, the real emotional speech and the corresponding phoneme in the emotional speech data set are used as the input of the prior encoder module, the real emotional speech in the emotional speech data set is also used as the input of the posterior encoder module, the prior encoder module is used to generate a prior distribution, the posterior encoder module is used to generate a latent variable z and is used as the input of the prior encoder module and the decoder module, and the decoder module is used to generate emotional synthesized speech.

[0012] The adaptive emotional speech synthesis network is constructed based on the trained conditional variational autoencoder network, comprising:

[0013] The adaptive emotional speech synthesis network is composed of the prior encoder module in the trained conditional variational autoencoder network and the decoder in the decoder module, which is used to generate a latent variable z and is used as the input of the decoder module.

[0014] The prior encoder module comprises:

[0015] The text embedding c corresponding to the input phoneme is generated text The emotional representation extraction network extracts the emotional representation c corresponding to the input speech emo The emotional representation and the text embedding c textAfter fusion, the text is input into a text encoder to obtain a text embedding h containing target sentiment information. text The duration predictor is based on text embeddings containing target sentiment information. text Predict the synthesis duration; the length adjuster adjusts the text embedding h with target sentiment information. text The text embedding is extended from the phoneme level to the frame level by the actual duration, resulting in an extended text embedding. The extended text embedding is fused with the positional encoding and then input into the frame prior network. The latent variable z generated by the posterior encoder module is used as the input to the stream network.

[0016] The posterior encoder module includes a connected Mel spectrum extractor and a posterior encoder. The real emotional speech y is used as the input of the Mel spectrum extractor, and the posterior encoder outputs the latent variable z.

[0017] The decoder module includes a decoder, a multi-scale discriminator, and a multi-period discriminator. The input to the decoder module is sliced ​​before being fed back into the decoder; the synthesized speech output by the decoder... The outputs of the multi-scale discriminator and the multi-period discriminator are used together as the output of the decoder module, serving as the inputs of the multi-scale discriminator and the multi-period discriminator.

[0018] The prior encoder module in the trained conditional variational autoencoder network includes:

[0019] Generate text embeddings corresponding to the input phonemes. text The emotion representation extraction network extracts the emotion representation c corresponding to the input speech. emo Integrating emotional representations with text embedding text After fusion, the text is input into a text encoder to obtain a text embedding h containing target sentiment information. text The duration predictor is based on text embeddings containing target sentiment information. text Predict the synthesis duration; the length adjuster adjusts the text embedding h with target sentiment information. text The predicted synthesis duration extends the text embedding from the phoneme level to the frame level, resulting in an extended text embedding. The extended text embedding is then fused with the positional encoding and input into the frame prior network. The prior distribution p(z|c) output by the frame prior network is sampled to obtain the prior feature variable f(z), which is used as the input to the stream network. The stream network outputs the latent variable z, which is used as the output of the prior encoder module.

[0020] II. A computer device

[0021] The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to perform the steps of a fully end-to-end adaptive emotional speech synthesis method.

[0022] Three, a computer readable storage medium

[0023] The computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of the fully end-to-end adaptive emotion speech synthesis method.

[0024] Four, a computer program product

[0025] The product includes a computer program / instruction, which is executed by a processor to implement the steps of the fully end-to-end adaptive emotion speech synthesis method.

[0026] The present application has the following beneficial effects:

[0027] The present application realizes the emotion adaptation of any speech through three modules of a priori encoder module, a posteriori encoder module and decoder module, that is, for the emotion appearing in any speech, the synthesis can be realized through the network model during reasoning.

[0028] The present application does not need to rely on emotion labels, and only needs to input reference emotion audio to generate speech with emotion, so that it is not limited to discrete emotion categories, thereby giving the synthesized speech more rich and delicate emotional color, realizing more fine-grained emotion controllable speech generation, and improving the naturalness and expressiveness of emotion speech synthesis.

[0029] The present application better simulates the generation of human speech by using a fully end-to-end speech synthesis technology. BRIEF DESCRIPTION OF DRAWINGS

[0030] Figure 1 The flowchart of the method of the present application.

[0031] Figure 2 The model structure diagram during the training of the conditional variational autoencoder network in the present application.

[0032] Figure 3 The model structure diagram of the adaptive emotion speech synthesis network in the present application. DETAILED DESCRIPTION

[0033] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all.

[0034] As Figure 1 shown, the fully end-to-end adaptive emotion speech synthesis method proposed by the present application specifically includes the following steps:

[0035] S1: Obtain an emotional speech dataset;

[0036] In a feasible implementation, the emotional speech dataset is obtained or collected, and the speaker of each real emotional speech in the emotional speech dataset is labeled, that is, the same speaker is labeled the same. For example, the "0001_000371.wav" speech is labeled as "Speaker 1", and the "0005_001092.wav" is labeled as "Speaker 5". Then, the Chinese text corresponding to each real emotional speech in the emotional speech dataset is normalized and converted into phonemes by a text front-end module. Then, forced alignment is performed using a speech-text alignment tool (such as Montreal Forced Aligner, MFA), and the real duration d of each phoneme of each speech in the generated emotional speech dataset is displayed, thereby obtaining a final emotional speech dataset. For example, the real duration d of the text "very lovely, just like you." is "24 10 7 10 8 23 20 1454 7 8 10 11 9 20 24". Therefore, the emotional speech dataset in S1 of the present application includes all phonemes corresponding to each text, the real duration of the phonemes, and the real emotional speech.

[0037] In a feasible implementation, a dataset including all phonemes corresponding to each text, the real duration of the phonemes, and the real emotional speech is directly used as the emotional speech dataset.

[0038] S2: A completely end-to-end conditional variational autoencoder network is constructed, the conditional variational autoencoder network is trained using the emotional speech dataset, and a trained conditional variational autoencoder network is obtained; then, a self-adaptive emotional speech synthesis network is constructed based on the trained conditional variational autoencoder network.

[0039] As shown in Figure 2 The conditional variational autoencoder network (CVAE) includes a prior encoder module, a posterior encoder module, and a decoder module. The prior encoder module and the posterior encoder module are connected, and the posterior encoder module and the decoder module are connected. In the training process, the real emotional speech in the emotional speech dataset and the corresponding phonemes are used as the input of the prior encoder module, that is, the real emotional speech input into the prior encoder module and the posterior encoder module is the same. The real emotional speech in the emotional speech dataset is also used as the input of the posterior encoder module. The prior encoder module is used to generate a prior distribution p(z|c). The posterior encoder module is used to generate a latent variable z and serve as the input of the prior encoder module and the decoder module. The decoder module is used to generate an emotional synthesized speech The following formula is satisfied:

[0040] z = Enc(y) ~ q(z|y)

[0041]

[0042] where Enc(y) ~ q(z|y) denotes that the latent variable z follows the posterior distribution q(z|y) of the given real emotional speech y after being processed by the prior encoder module; Dec(z) ~ p(y|z) denotes that the emotional synthesized speech y follows the prior distribution p(y|z) of the given latent variable z after being processed by the decoder module. follows the prior distribution p(y|z) of the given latent variable z.

[0043] The prior encoder module comprises:

[0044] generates a text embedding c corresponding to the phonemes of the input text , and the emotional representation extraction network extracts an emotional representation c corresponding to the input speech emo . Optionally, the emotional representation extraction network adopts an emotion2vec model network based on Ali, is pre-trained on a large amount of unmarked emotional data through self-supervised learning, so that it can effectively learn generalizable speech emotional features. It disentangles the relationship between speaker identity and speech content while extracting the representation, achieving emotional adaptation to any speech. In training, the real emotional speech is used as the reference emotional speech. The emotional representation extracted by the emotional representation extraction network is a one-dimensional vector with a length of 1024, which is stored using an npy file. For example, for the "0001_000371.wav" speech, a "0001_000371.npy" file will be generated. The emotional representation c emo is processed through a linear layer to align with the dimension of the text embedding, and then fused with the text embedding to input into the text encoder to obtain a text embedding h with target emotional information text (i.e., the condition c of the prior encoder module); optionally, the text encoder is composed of a plurality of transformer blocks and a linear projection layer connected in sequence. The number of transformer blocks is 6 or 12. In order to enable the model to better capture the emotional features of the reference speech, the emotional representation c emo is also input into the third transformer block of the text encoder. The duration predictor predicts the synthesized duration based on the text embedding h with target emotional information text .

[0045] In a feasible implementation, in order to enable the model to better predict the synthesized duration , adversarial learning is introduced, and a conditional discriminator is used to train the duration predictor. The text embedding h with target emotional information text is used as the input of the generator G dur , i.e., the input of the duration predictor.

[0046] Because for emotional speech, the acoustic variations within each phoneme are much more subtle, and each phoneme carries rich emotional information. Only the phoneme-level mean μ is used. θ and variance σ θ This is far from sufficient. Therefore, to generate more nuanced and emotionally controllable speech, a length adjuster and a frame prior network are introduced. The length adjuster adjusts the text embedding h with target emotional information. text Furthermore, the actual duration extends the text embedding from the phoneme level to the frame level, resulting in an extended text embedding; in this invention, the text embedding with emotional information h text On the one hand, the input is fed into the duration predictor to generate the synthetic duration. The duration loss is calculated using the actual duration *d*, and simultaneously input into a length adjuster along with *d* to extend it from the phoneme level to the frame level. The extended text embedding is then fused with positional encoding before being fed into the frame prior network. Positional encoding is only position-dependent, independent of content, and is used to convey the positional information of each frame-level element, allowing the model to consider sequence information during learning. The frame prior network is used for post-processing of its input to achieve a finer-grained representation of information. The frame prior network processes the frame-level sequence to calculate the frame-level mean μ. θ and variance σ θ The latent variable z generated by the posterior encoder module serves as the input to the streaming network. The streaming network transforms the latent variable z into a prior feature variable f(z) to enhance the flexibility of the prior distribution. The KL divergence loss is generated based on the outputs of the streaming network and the frame prior network. During training, the network weights of the sentiment representation extraction network are frozen, while the network weights of other modules are not frozen.

[0047] In one feasible implementation, the frame prior network consists of multiple transformer block layers, enhancing its ability to model frame-level information, enabling it to generate a more fine-grained normal distribution and characterize the mean μ. θ and variance σ θ This allows for a finer-grained representation of information. The formula is expressed as:

[0048]

[0049] Where det is the Jacobian determinant; f θ (z) is the prior feature function, used to map the latent variable z to the space containing the prior feature variable f(z); N(f θ (z); μ θ (c),σ θ (c) is a mean μ θ (c), with variance σ θ (c) is a Gaussian (normal) distribution.

[0050] In one possible implementation, the flow network is composed of multiple affine coupling layers to enhance the flexibility of the prior distribution, while adding a small transformer block with residual connections in the flow network to capture long-term dependencies. The flow network can reversibly transform a normal distribution into a more general distribution, and the trained flow network can implement the reverse transformation.

[0051] The posterior encoder module includes a connected mel-spectrogram extractor and a posterior encoder, the real emotional speech y is input into the mel-spectrogram extractor, and the posterior encoder outputs the latent variable z. The mel-spectrogram extractor is used to process the real emotional speech y and generate the mel-spectrogram x mel . A plurality of WaveNet residual blocks in the posterior encoder extract a series of hidden vectors based on the mel-spectrogram x mel , which are then linearly projected to estimate the mean and variance of the posterior distribution p(z|y). Subsequently, the latent variable z is derived after the estimated mean and variance are reparameterized and output.

[0052] The decoder module includes a decoder, a multi-scale discriminator, and a multi-period discriminator. The input of the decoder module is sliced and then input into the decoder; specifically, the latent variable z is randomly sliced according to the set length, so that the sliced audio segment is the same as the set length. Therefore, only part of the latent variable z is input into the decoder to generate the corresponding speech segment. The synthesized speech output by the decoder is input into the multi-scale discriminator and the multi-period discriminator, and the outputs of the multi-scale discriminator and the multi-period discriminator are taken together as the output of the decoder module. The multi-period discriminator and the multi-scale discriminator are used to enhance the quality of the synthesized audio.

[0053] Based on the trained conditional variational autoencoder network, an adaptive emotional speech synthesis network is constructed, including:

[0054] As shown in Figure 3 , the adaptive emotional speech synthesis network (i.e. the model structure diagram during inference) is composed of the prior encoder module in the trained conditional variational autoencoder network and the decoder in the decoder module, which is used to generate the latent variable z and input into the decoder module. The input of the prior encoder module includes the reference emotional speech and the phonemes corresponding to the target text, and there is no corresponding relationship between the reference emotional speech and the target text. The output of the synthesized speech has the same emotion as that in the reference emotional speech.

[0055] The prior encoder module in the trained conditional variational autoencoder network includes:

[0056] The input phonemes correspond to the text embedding c text generated by the text embedding network, and the emotional representation extraction network extracts the emotional representation c emoDuring training, authentic emotional speech is used as the reference emotional speech. Emotional representations are embedded into the text. text After fusion, the text is input into a text encoder to obtain a text embedding h containing target sentiment information. text The duration predictor is based on text embeddings containing target sentiment information. text Predict the synthesis duration; the length adjuster adjusts the text embedding h with target sentiment information. text The predicted synthesis duration extends the text embedding from the phoneme level to the frame level, resulting in an extended text embedding. The extended text embedding is then fused with the positional encoding and input into the frame prior network. The prior distribution p(z|c) output by the frame prior network is sampled to obtain the prior feature variable f(z), which is used as the input to the stream network. The stream network outputs the latent variable z, which is used as the output of the prior encoder module.

[0057] The overall discriminator is composed of both the multi-period discriminator and the multi-scale discriminator. The other network modules in the conditional variational autoencoder network besides the multi-period discriminator and the multi-scale discriminator are referred to as the overall generator.

[0058] The loss during the training process of a conditional variational autoencoder network satisfies the following formula:

[0059] L = L cvae +L dur +L adv (G dur )+L adv (G)+L fm (G)

[0060] L cvae =L recon +D KL (q(z|y)||p(z|c))

[0061]

[0062] L dur =MSE(G dur (h text ), d)

[0063]

[0064] Where L is the total loss of the overall generator, L cvae For the conditional variational autoencoder loss; L dur For time loss; L adv (G dur L represents the adversarial loss of the generator in the duration predictor; adv (D dur) is the adversarial loss of the discriminator in the duration predictor, used to train the discriminator in the duration predictor; L adv (G) is the adversarial loss of the overall generator; L adv (D) is the adversarial loss of the overall discriminator, used to train the overall discriminator; L fm (G) is the feature matching loss; it improves the effect of generated speech by comparing the synthesized speech features with the real speech features, to stabilize the training. L recon is the reconstruction loss, which improves the quality of synthesized speech (similarity with real speech) by comparing the mel-spectrogram of synthesized speech with the mel-spectrogram of real speech; D KL (||) is the KL divergence loss; q(z|y) is the posterior distribution; p(z|c) is the prior distribution; x mel is the mel-spectrogram feature of real emotional speech y; is the mel-spectrogram feature of synthesized speech ; ||| is the absolute error, i.e., taking the absolute value of the calculation result; MSE(, ) is the mean square error, i.e., calculating the average of the square of the difference between the predicted value and the true value; d is the real duration; G dur () is the synthesized duration is the expectation symbol, indicating the expectation calculation on all possible inputs; is the expectation value of the text embedding h text with target emotional information; D dur () is the output of the discriminator in the duration predictor; is the joint expectation value of the real duration d and the text embedding h text with target emotional information; D(G(z)) is the output of the overall discriminator; is the expectation value of the latent variable z; is the joint expectation value of real emotional speech y and latent variable z; x fm is the representation of real emotional speech y in the feature space; is the representation of synthesized speech in the feature space.

[0065] S3: generate phonemes corresponding to the target text, input the reference emotional speech and the phonemes corresponding to the target text into the adaptive emotional speech synthesis network, and the network outputs emotional synthesized speech corresponding to the target text and consistent with the emotion of the reference emotional speech

[0066] In S3, the target text is converted into phonemes and then into text embedding C text , and the reference emotional speech is converted into emotional representation C emo, which are then fused and input into the text encoder to obtain the text embedding with the target emotion information at the phoneme level, i.e., condition h text , which is input into the duration predictor to generate the synthesized duration The condition h text is then input into the length regulator together with the synthesized duration , so as to extend h text from the phoneme level to the frame level, h text is then fused with the position encoding and input into the frame prior network for post-processing to generate the frame-level mean μ θ and variance σ θ , using which the frame-level mean μ θ and variance σ θ are used to generate f(z), which is converted into the latent variable z by the flow network. Finally, the latent variable z is input into the decoder to reconstruct the synthesized speech , i.e., to generate the emotion-synthesized speech corresponding to the text and consistent with the emotion of the reference emotion-synthesized speech

[0067] The purpose of using the emotion representation extraction network in the present application is to obtain the emotion representation of the reference emotion-synthesized speech while disentangling the relationship between the speaker identity and the speech content, so as to realize emotion adaptation of any speech at the time of synthesis, i.e., inputting the speech of any speaker, the model can have the emotion of the reference speech while keeping the original speaker's timbre unchanged.

[0068] The purpose of adding the length regulator and the frame prior network in the prior encoder module is to extend the condition c from the phoneme level to the frame level, so as to realize more fine-grained representation of information and generate more delicate emotion-controllable speech. Because for the speech with emotion, each phoneme internally carries rich emotion information and acoustic changes. The representation at the phoneme level is far from enough.

[0069] The purpose of deleting the monotonic alignment search of the original model and using the Montreal forced alignment tool is to display the duration of each phoneme and input it into the duration predictor, so that the duration of the generated speech is closer to the real speech. Finally, more fine-grained emotion-controllable speech generation is realized and the quality and emotion effect of the speech generation are improved.

[0070] The above merely describes the preferred specific embodiments of the present application, but the protection scope of the present application is not limited thereto, any person skilled in the art can make equivalent replacements or changes to the technical solutions and the inventive concept of the present application within the technical range disclosed by the present application, which should be covered within the protection scope of the present application.

Claims

1. A fully end-to-end based adaptive emotional speech synthesis method, characterized by, The method comprises the following steps: S1: obtaining an emotional speech dataset; S2: constructing a fully end-to-end conditional variational autoencoder network, training the conditional variational autoencoder network using the emotional speech dataset to obtain a trained conditional variational autoencoder network, and then constructing an adaptive emotional speech synthesis network based on the trained conditional variational autoencoder network; S3: inputting a target text to generate phonemes corresponding to the target text, inputting a reference emotional speech and the phonemes corresponding to the target text into the adaptive emotional speech synthesis network, and outputting an emotional synthesized speech corresponding to the target text and consistent with the emotion of the reference emotional speech.

2. The fully end-to-end based adaptive emotional speech synthesis method according to claim 1, wherein, The conditional variational autoencoder network comprises a prior encoder module, a posterior encoder module, and a decoder module, the prior encoder module and the posterior encoder module are connected, the posterior encoder module and the decoder module are connected, in the training process, real emotional speeches in the emotional speech dataset and corresponding phonemes are taken as inputs of the prior encoder module, the real emotional speeches in the emotional speech dataset are also taken as inputs of the posterior encoder module, the prior encoder module is used to generate a prior distribution, the posterior encoder module is used to generate a latent variable z and taken as inputs of the prior encoder module and the decoder module, and the decoder module is used to generate an emotional synthesized speech.

3. The fully end-to-end based adaptive emotional speech synthesis method according to claim 1, wherein, The adaptive emotional speech synthesis network is constructed based on the trained conditional variational autoencoder network, comprising: The adaptive emotional speech synthesis network is composed of the prior encoder module in the trained conditional variational autoencoder network and a decoder in the decoder module, and is used to generate the latent variable z and taken as an input of the decoder module.

4. The fully end-to-end based adaptive emotional speech synthesis method according to claim 2, wherein, The prior encoder module comprises: Generate text embeddings corresponding to the input phonemes. text The emotion representation extraction network extracts the emotion representation c corresponding to the input speech. emo Integrating emotional representations with text embedding text After fusion, the text is input into a text encoder to obtain a text embedding h containing target sentiment information. text The duration predictor is based on text embeddings containing target sentiment information. text Predict the synthesis duration; the length adjuster adjusts the text embedding h with target sentiment information. text The text embedding is extended from the phoneme level to the frame level by the actual duration, resulting in an extended text embedding. The extended text embedding is fused with the positional encoding and then input into the frame prior network. The latent variable z generated by the posterior encoder module is used as the input to the stream network.

5. The fully end-to-end based adaptive emotional speech synthesis method according to claim 2, wherein, The posterior encoder module comprises a mel-frequency spectrum extractor and a posterior encoder connected thereto, the real emotional speech y is taken as an input of the mel-frequency spectrum extractor, and the posterior encoder outputs the latent variable z.

6. The fully end-to-end based adaptive emotional speech synthesis method according to claim 2, wherein, The decoder module comprises a decoder, a multi-scale discriminator, and a multi-period discriminator, and the input of the decoder module is sliced and then input into the decoder; Synthesized speech output by the decoder As input to the multi-scale discriminator and the multi-cycle discriminator, the outputs of the multi-scale discriminator and the multi-cycle discriminator are taken together as output of the decoder module.

7. The fully end-to-end based adaptive emotional speech synthesis method according to claim 3, wherein, The prior encoder module in the trained conditional variational autoencoder network comprises: Generate text embeddings corresponding to the input phonemes. text The emotion representation extraction network extracts the emotion representation c corresponding to the input speech. emo Integrating emotional representations with text embedding text After fusion, the text is input into a text encoder to obtain a text embedding h containing target sentiment information. text The duration predictor is based on text embeddings containing target sentiment information. text Predict the synthesis duration; the length adjuster adjusts the text embedding h with target sentiment information. text The predicted synthesis duration extends the text embedding from the phoneme level to the frame level, resulting in an extended text embedding. The extended text embedding is then fused with the positional encoding and input into the frame prior network. The prior distribution p(z|c) output by the frame prior network is sampled to obtain the prior feature variable f(z), which is used as the input to the stream network. The stream network outputs the latent variable z, which is used as the output of the prior encoder module.

8. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that, The processor executes the computer program to realize the steps of the adaptive emotional speech synthesis method based on a fully end-to-end network in any one of claims 1 to 7.

9. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to realize the steps of the adaptive emotional speech synthesis method based on a fully end-to-end network in any one of claims 1 to 7.

10. A computer program product comprising computer programs / instructions, characterized in that, The computer program / instruction is executed by the processor to realize the steps of the adaptive emotional speech synthesis method based on a fully end-to-end network in any one of claims 1 to 7.

Images