Expressive speech synthesis enhancement method and system based on conditional embedding
By adopting a conditional embedding method in the speech synthesis technology, combining emotion prediction, prologue prediction and feature fusion, the problem of unnatural speech expression in the existing technology is solved, and a more natural and delicate emotional speech synthesis is achieved.
Patent Information
- Application Number
- CN202411512380.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-28
- Publication Date
- 2025-02-14
AI Technical Summary
When existing emotional speech synthesis technology processes data sets with emotional attributes, the emotional expression of synthetic audio is not natural enough and difficult to match the emotional atmosphere to be conveyed by the text.
Using an expressive speech synthesis enhancement method based on conditional embedding, the emotional predictor generates the characteristics of the emotional label and the speaker label through the emotion predictor to generate global condition vectors; combining the pronunciation predictor and text encoder to generate frame-level potential features and prediction duration. Use these features and duration to generate delicate and authentic emotional voice.
Accurate control of the emotional expression of synthetic speech is achieved, and the generated speech is more natural and richer, and can accurately convey the expected emotions.
Smart Images
Figure CN119446110A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of speech synthesis technology, and in particular to an expressive speech synthesis enhancement method based on conditional embedding, an expressive speech synthesis enhancement system based on conditional embedding, a computer device, a computer-readable storage medium and a computer program product. Background Art
[0002] The statements in this section merely provide background art related to the present invention and do not necessarily constitute prior art.
[0003] Speech synthesis technology, also widely known as text-to-speech (TTS) technology, is an artificial intelligence technology that converts text information into spoken language. In recent years, thanks to the rapid development of deep learning and artificial intelligence, speech synthesis technology has undergone a qualitative innovation. In this field of technology, emotional speech synthesis technology is particularly eye-catching. It aims not only to achieve accurate conversion of text content to speech, but also strives to incorporate rich emotional colors into the synthesized speech, making the speech sound more natural, full and contagious.
[0004] The inventors found in their research that although emotional speech synthesis technology has shown great potential, the existing technology still has some limitations when processing datasets with emotional attributes. These limitations are mainly manifested in that the emotional expression of the synthesized audio is not natural enough and sometimes it is difficult to match the emotional atmosphere that the text wants to convey. Summary of the invention
[0005] In order to address the deficiencies of the prior art, the present invention provides an expressive speech synthesis enhancement method and system based on conditional embedding, which can achieve precise control of the emotional expression of synthesized speech and generate more delicate and realistic emotional speech.
[0006] In order to achieve the above object, the present invention adopts the following technical solution:
[0007] In a first aspect, the present invention provides an expressive speech synthesis enhancement method based on conditional embedding.
[0008] An expressive speech synthesis enhancement method based on conditional embedding includes the following processes:
[0009] The emotion predictor generates a corresponding emotion label according to the text, and the emotion label is input into the conditional encoder together with the speaker label to perform feature fusion to generate a global conditional vector;
[0010] Processing the text and converting it into a phoneme sequence, encoding the phoneme sequence through a text encoder, passing the encoding result to a prosody predictor, and generating frame-level latent features together with a global conditional variable;
[0011] The frame-level latent features are mapped into a set of Gaussian distributions μ and σ through a linear projection layer, and a random duration prediction period generates a predicted duration according to a global condition vector and the encoding result, and the distribution of the text is expanded using the predicted duration;
[0012] According to the expanded text distribution, the normalized flow is sampled, and the latent variables are generated through the inverse flow transform. Finally, the decoder generates the speech output with the corresponding emotional characteristics.
[0013] In a second aspect, the present invention provides an expressive speech synthesis enhancement system based on conditional embedding, comprising the following process:
[0014] The global condition vector generating unit is configured as follows: the emotion predictor generates a corresponding emotion label according to the text, and the emotion label is input into the condition encoder together with the speaker label to perform feature fusion to generate a global condition vector;
[0015] A frame-level latent feature generation unit is configured to: process the text and convert it into a phoneme sequence, encode the phoneme sequence through a text encoder, pass the encoding result to a prosody predictor, and generate a frame-level latent feature together with a global conditional variable;
[0016] The predicted duration generation unit is configured as follows: the frame-level potential features are mapped into a set of Gaussian distributions μ and σ through a linear projection layer, a random duration prediction period generates a predicted duration according to a global condition vector and the encoding result, and a distribution of the text is expanded using the predicted duration;
[0017] The speech output unit is configured to: sample a normalized flow according to the expanded text distribution, generate latent variables through inverse flow transformation, and finally generate speech output with corresponding emotional characteristics through a decoder.
[0018] In a third aspect, the present invention provides a computer device, comprising: a processor and a computer-readable storage medium;
[0019] a processor adapted to execute a computer program;
[0020] A computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by the processor, the expressive speech synthesis enhancement method based on conditional embedding as described in the first aspect of the present invention is implemented.
[0021] In a fourth aspect, the present invention provides a computer-readable storage medium storing a computer program, wherein the computer program is suitable for being loaded by a processor and executing the expressive speech synthesis enhancement method based on conditional embedding as described in the first aspect of the present invention.
[0022] In a fifth aspect, the present invention provides a computer program product, comprising a computer program, which, when executed by a processor, implements the expressive speech synthesis enhancement method based on conditional embedding as described in the first aspect of the present invention.
[0023] Compared with the prior art, the present invention has the following beneficial effects:
[0024] 1. The present invention expands the phoneme-level parameters to the frame-level parameters through the prosody predictor, and generates natural prosodic features in combination with the global conditional variables, so that the synthesized speech is more prosodic and natural.
[0025] 2. The present invention uses a conditional encoder to convert emotion and speaker labels into embedding vectors, and generates a global conditional embedding through feature fusion to ensure that the generated speech conforms to the speaker characteristics and accurately conveys the expected emotion.
[0026] 3. The present invention uses the pre-trained RoBERTa model to extract and predict emotional information, and integrates it into the speech synthesis process to achieve precise control and delicate expression of emotions.
[0027] Advantages of additional aspects of the present invention will be given in part in the following description, and in part will become obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS
[0028] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.
[0029] Figure 1 A schematic diagram of model training provided in Example 1 of the present invention;
[0030] Figure 2 A schematic diagram of an expressive speech synthesis enhancement method based on conditional embedding provided in Example 1 of the present invention;
[0031] Figure 3 A schematic diagram of an expressive speech synthesis enhancement system based on conditional embedding provided in Example 2 of the present invention;
[0032] Figure 4 A schematic diagram of a computer device provided in Example 3 of the present invention. DETAILED DESCRIPTION
[0033] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.
[0034] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.
[0035] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.
[0036] Embodiment 1:
[0037] This implementation proposes an expressive speech synthesis enhancement method based on conditional embedding, which can achieve precise control of the emotional expression of synthesized speech and generate more delicate and realistic emotional speech. Specifically, it includes the following processes:
[0038] S1: Data preprocessing stage.
[0039] A series of preprocessing operations are performed on the text and audio in the sentiment dataset to ensure the quality and consistency of the data.
[0040] S1.1: Specific process of voice data processing.
[0041] All audio samples are processed into mono format, the sampling rate is uniformly set to 22050Hz, and they are encoded in PCM16bit format. All audio is free of background noise interference. All audio samples with a duration of less than 1 second and their corresponding texts are removed. Finally, linear spectrum features are extracted from each audio file, which will be used as input for the model later.
[0042] S1.2: Specific process of text data processing.
[0043] All texts remove non-text elements, eliminate redundant spaces, including redundant spaces between words and spaces at the beginning and end of lines, correct inappropriate line breaks and indentations, and use simplified Chinese and UTF-8 encoding for all texts. The International Phonetic Alphabet (IPA) is then used to convert the text into phoneme sequences.
[0044] S1.3: Divide the sentiment dataset into validation set, test set and training set according to a certain ratio.
[0045] S2: Model building phase.
[0046] Build an emotional speech synthesis model, which includes an emotion predictor, a prosody predictor, a conditional encoder, and a speech synthesis model. Figure 1 The emotional speech synthesis model constructed in step S2 includes an emotion predictor, a prosody predictor, a conditional encoder and a speech synthesis model.
[0047] S2.1: Specific process of the emotion predictor.
[0048] First, the RoBERTa model is fine-tuned using a text dataset labeled with sentiment labels to better adapt it to specific sentiment classification tasks. After fine-tuning, the input text is converted into a sentiment feature vector through the RoBERTa model, and then fed into a linear layer to map these deep features to a predefined sentiment category space to generate a specific sentiment category label. v is the feature vector output from the RoBERTa model, W and b are the weights and biases of the linear layer, and the final output y can be used as a sentiment label. This process can be expressed by the following formula:
[0049] y=softmax(W·v+b)(1).
[0050] S2.2: Specific process of the prosody predictor.
[0051] The conditional variable H and the phoneme-level language information h generated by the text encoder are received and further refined through six one-dimensional residual convolutional layers with a kernel size of 17 to predict the rhythmic changes at the frame level. Then, a set of Gaussian distribution parameters μ and σ are generated through a linear projection layer. In order to enhance the model's ability to express emotional rhythmic changes, we apply a normalization flow f θ , this technique can transform simple distributions into more complex distributions through reversible transformations, thereby improving the expressive power of the prior distribution:
[0052]
[0053] S2.3: Specific process of conditional encoder.
[0054] Receive the emotion and speaker labels, and use the embedding layer to convert these discrete labels into their respective embedding vectors e and g. Then, these two vectors are stacked in the feature dimension, and the features are fused through the splicing operation to generate a comprehensive feature vector h:
[0055] h = concat(e,g)(3);
[0056] The vector of formula (3) is then further encoded into a conditional vector H, which is used for global conditioning throughout the model to ensure that the generated speech output is both consistent with the characteristics of the specific speaker and accurately conveys the intended emotion.
[0057] S2.4: The speech synthesis model is based on VITS, including a priori encoder, a posteriori encoder, random duration predictor, decoder and discriminator. The VITS (Variational Inference with adversarial learning for end-to-end Text-to-Speech) model here is an end-to-end speech synthesis model based on variational inference, standardized flow and adversarial training.
[0058] S2.4.1: Specific process of the prior encoder.
[0059] The text encoder converts the input text phoneme sequence into a fixed-dimensional vector representation through the embedding layer, and then uses the attention mechanism to weight the embedded vector and normalize the flow f θ It is used to enhance the flexibility of the prior distribution.
[0060] S2.4.2: Specific process of the posterior encoder.
[0061] The linear spectrum of the input audio is extracted using a one-dimensional convolutional layer, which is then fused with the conditional vector H and sent to the WaveNet module for further processing. The processed signal is again passed through a one-dimensional convolutional layer to extract higher-level features, which are then decomposed into μ and σ, and finally the latent variable z is generated through a sampling step.
[0062] S2.4.3: Specific process of random duration predictor.
[0063] During the training phase, the module first accepts the text encoding information h and combines it with the duration d obtained by forced alignment. These data are used to parameterize a standard normal distribution. During the inference phase, the module uses the text duration d predicted during training to expand the distribution of the text so that the number of distributions is equal to the number of audio features.
[0064] S2.4.4: Decoder specific process.
[0065] The generator of HiFi-GAN V1 is adopted, which consists of a stack of transformation convolutions, each of which is followed by a multi-receptive field fusion module (MRF) to add the conditional vector H to the input latent variable z, and then gradually upsample it to finally restore it to real audio.
[0066] S2.4.5: The specific process of the discriminator.
[0067] Using HiFi-GAN’s multi-cycle discriminator, the speech signal is processed through multiple layers of one-dimensional convolutional layers, each followed by an activation function to increase nonlinearity and help capture complex features. Then, features are further extracted and compressed through a series of inverse convolution operations and downsampling layers to generate a final feature vector. This feature vector is used to determine whether the input signal is real speech or synthesized by the generator, helping the generator to optimize speech quality.
[0068] S3: Model training phase.
[0069] The preprocessed text and audio data are input into the emotional speech synthesis model for training. During the training process, the model parameters are optimized by calculating the values of various loss functions, thereby gradually improving the performance of the model and the quality of the generated speech.
[0070] S3.1: Specific process of the training phase.
[0071] The preprocessed text phoneme sequence is encoded by the text encoder and then input into the prosody predictor together with the conditional variable H to generate the P distribution. At the same time, the preprocessed audio linear spectrum is also encoded into the Q distribution; the Q distribution is sampled to obtain z and processed by flow to generate zp, which is matched with the P distribution to generate the duration d, and then zp is brought into each P distribution to maximize the likelihood value; the text encoding h and the duration d are input into the random duration predictor to calculate the duration prediction loss L dur , copy the P distribution according to the predicted duration of d, make the number of distributions equal to the number of audio features, slice the latent variable z and pass it into the decoder to generate audio with emotion. During the entire training process, the embedding vector H generated by the conditional encoder is input to each module as a global condition; to improve the accuracy of emotion modeling, the real emotion label is used as the input of the conditional encoder, and the loss L between the real label and the predicted label is calculated pre .
[0072] In addition to the above losses, it also includes the KL divergence loss L between the Q distribution and the P distribution kl , Mel reconstruction loss L of input audio features and output audio features recon , the adversarial loss L between the generator and the discriminator adv , and feature matching loss L fm . These losses work together to optimize the performance of the model.
[0073] S3.2: The main loss during training is:
[0074] S3.2.1: Mel reconstruction loss L recon :
[0075] L recon =||Mel pre-Mel tar ||(4);
[0076] S3.2.2: KL divergence loss L kl :
[0077] L kl =D kl (q(z|x)||p(z|c))(5);
[0078] S3.2.3: Sentiment prediction loss L pre :
[0079]
[0080] S3.2.4: Adversarial Loss L adv :
[0081] L adv (G) = E z [(D(G(z))-1) 2 ](7);
[0082] L adv (D)=E (y,z) [(D(y)-1) 2 +(D(G(z)))](8);
[0083] S3.2.5: Feature matching loss L fm :
[0084]
[0085] S3.2.6: Duration prediction loss L dur :
[0086]
[0087] S4: Model reasoning stage.
[0088] like Figure 2 As shown in Figure 1, the input text, speaker label and emotion label are received to generate speech output with corresponding emotion characteristics. First, the input text is processed and converted into a phoneme sequence through step 1. Then, these phoneme sequences are encoded into h through the text encoder and then passed to the prosody predictor to generate frame-level latent features together with the conditional variable H. Then, they are mapped into a set of Gaussian distributions μ and σ through the linear projection layer, and the distribution of the text is expanded using the predicted duration d to sample f. θ(z), generates latent variable z through inverse flow transformation, and finally generates audio through decoder. In this process, the emotion predictor generates corresponding emotion labels based on the text, which are input into the conditional encoder together with the speaker label to generate the global conditional vector H through feature fusion, and supports users to manually specify emotion labels.
[0089] Embodiment 2:
[0090] like Figure 3 As shown, this implementation provides an expressive speech synthesis enhancement system based on conditional embedding, including:
[0091] The global condition vector generating unit is configured as follows: the emotion predictor generates a corresponding emotion label according to the text, and the emotion label is input into the condition encoder together with the speaker label to perform feature fusion to generate a global condition vector;
[0092] A frame-level latent feature generation unit is configured to: process the text and convert it into a phoneme sequence, encode the phoneme sequence through a text encoder, pass the encoding result to a prosody predictor, and generate a frame-level latent feature together with a global conditional variable;
[0093] The predicted duration generation unit is configured as follows: the frame-level potential features are mapped into a set of Gaussian distributions μ and σ through a linear projection layer, a random duration prediction period generates a predicted duration according to a global condition vector and the encoding result, and a distribution of the text is expanded using the predicted duration;
[0094] The speech output unit is configured to: sample a normalized flow according to the expanded text distribution, generate latent variables through inverse flow transformation, and finally generate speech output with corresponding emotional characteristics through a decoder.
[0095] The specific processes of the above-mentioned units are described in Example 1 and will not be repeated here.
[0096] It is understandable that the above-mentioned various units can be separately or all combined into one or several other units to constitute, or some (some) units therein can also be split into multiple smaller units in function to constitute, which can achieve the same operation without affecting the realization of the technical effects of the embodiments of the present application. The above-mentioned units are divided based on logical functions. In practical applications, the function of a unit can also be realized by multiple units, or the function of multiple units can be realized by one unit. In other embodiments of the present application, the system can also include other units. In practical applications, these functions can also be implemented with the assistance of other units, and can be implemented by the collaboration of multiple units.
[0097] According to another embodiment of the present application, the system described in this embodiment can be constructed, and the method of Example 1 of the present application can be implemented by running a computer program (including program code) capable of executing the steps involved in the corresponding method described in Example 1 on a general computing device such as a computer including processing elements and storage elements such as a central processing unit (CPU), random access memory (RAM), and read-only memory (ROM). The computer program can be recorded on, for example, a computer-readable recording medium, and loaded into the above-mentioned computing device through the computer-readable recording medium and run therein.
[0098] Embodiment 3:
[0099] like Figure 4 As shown, this implementation provides an electronic device, which includes a processor 1001, a communication interface 1002, and a computer-readable storage medium 1003. The processor 1001, the communication interface 1002, and the computer-readable storage medium 1003 may be connected via a bus or other means.
[0100] Among them, the communication interface 1002 is used to receive and send data, the computer-readable storage medium 1003 can be stored in the memory of the electronic device, the computer-readable storage medium 1003 is used to store a computer program, the computer program includes program instructions, and the processor 1001 is used to execute the program instructions stored in the computer-readable storage medium 1003.
[0101] Processor 1001 (or CPU (Central Processing Unit)) is the computing core and control core of the electronic device, which is suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions to implement corresponding method processes or corresponding functions.
[0102] The processor 1001 is configured to execute the following process:
[0103] The emotion predictor generates a corresponding emotion label according to the text, and the emotion label is input into the conditional encoder together with the speaker label to perform feature fusion to generate a global conditional vector;
[0104] Processing the text and converting it into a phoneme sequence, encoding the phoneme sequence through a text encoder, passing the encoding result to a prosody predictor, and generating frame-level latent features together with a global conditional variable;
[0105] The frame-level latent features are mapped into a set of Gaussian distributions μ and σ through a linear projection layer, and a random duration prediction period generates a predicted duration according to a global condition vector and the encoding result, and the distribution of the text is expanded using the predicted duration;
[0106] According to the expanded text distribution, the normalized flow is sampled, and the latent variables are generated through the inverse flow transform. Finally, the decoder generates the speech output with the corresponding emotional characteristics.
[0107] The specific process is described in Example 1 and will not be repeated here.
[0108] Embodiment 4:
[0109] This implementation provides a computer-readable storage medium (Memory), which is a memory device in an electronic device for storing programs and data. It is understandable that the computer-readable storage medium here can include both built-in storage media in the electronic device and, of course, extended storage media supported by the electronic device. The computer-readable storage medium provides a storage space that stores the processing system of the electronic device.
[0110] In addition, the storage space also stores one or more instructions suitable for being loaded and executed by the processor, and these instructions may be one or more computer programs (including program codes). It should be noted that the computer-readable storage medium here may be a high-speed RAM memory, or a non-volatile memory, such as at least one disk memory; optionally, it may also be at least one computer-readable storage medium located away from the aforementioned processor.
[0111] In one embodiment, the computer-readable storage medium stores one or more instructions; the processor loads and executes the one or more instructions stored in the computer-readable storage medium to implement the following process:
[0112] The emotion predictor generates a corresponding emotion label according to the text, and the emotion label is input into the conditional encoder together with the speaker label to perform feature fusion to generate a global conditional vector;
[0113] Processing the text and converting it into a phoneme sequence, encoding the phoneme sequence through a text encoder, passing the encoding result to a prosody predictor, and generating frame-level latent features together with a global conditional variable;
[0114] The frame-level latent features are mapped into a set of Gaussian distributions μ and σ through a linear projection layer, and a random duration prediction period generates a predicted duration according to a global condition vector and the encoding result, and the distribution of the text is expanded using the predicted duration;
[0115] According to the expanded text distribution, the normalized flow is sampled, and the latent variables are generated through the inverse flow transform. Finally, the decoder generates the speech output with the corresponding emotional characteristics.
[0116] The specific process is described in Example 1 and will not be repeated here.
[0117] Embodiment 5:
[0118] The present implementation provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the electronic device performs the following process:
[0119] The emotion predictor generates a corresponding emotion label according to the text, and the emotion label is input into the conditional encoder together with the speaker label to perform feature fusion to generate a global conditional vector;
[0120] Processing the text and converting it into a phoneme sequence, encoding the phoneme sequence through a text encoder, passing the encoding result to a prosody predictor, and generating frame-level latent features together with a global conditional variable;
[0121] The frame-level latent features are mapped into a set of Gaussian distributions μ and σ through a linear projection layer, and a random duration prediction period generates a predicted duration according to a global condition vector and the encoding result, and the distribution of the text is expanded using the predicted duration;
[0122] According to the expanded text distribution, the normalized flow is sampled, and the latent variables are generated through the inverse flow transform. Finally, the decoder generates the speech output with the corresponding emotional characteristics.
[0123] The specific process is described in Example 1 and will not be repeated here.
[0124] A person skilled in the art can appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. A person skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.
[0125] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loading and executing computer program instructions on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable device. The computer instruction can be stored in a computer-readable storage medium or transmitted by a computer-readable storage medium. The computer instruction can be transmitted from a website site, a computer, a server or a data center to another website site, a computer, a server or a data center by wired (for example, coaxial cable, optical fiber, digital line (DSL)) or wireless (for example, infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that a computer can access or a data processing device such as a server, a data center, etc. that contains one or more available media integration. Available media can be magnetic media (for example, floppy disk, hard disk, tape), optical media (for example, DVD), or semiconductor media (for example, solid-state drive (Solid State Disk, SSD)) and the like.
[0126] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.
Claims
1. A method for enhancing expressive speech synthesis based on conditional embedding, characterized in that: The process includes: The emotion predictor generates a corresponding emotion label according to the text, and the emotion label is input into the conditional encoder together with the speaker label to perform feature fusion to generate a global conditional vector; Processing the text and converting it into a phoneme sequence, encoding the phoneme sequence through a text encoder, passing the encoding result to a prosody predictor, and generating frame-level latent features together with a global conditional variable; The frame-level latent features are mapped into a set of Gaussian distributions μ and σ through a linear projection layer, and a random duration prediction period generates a predicted duration according to a global condition vector and the encoding result, and the distribution of the text is expanded using the predicted duration; According to the expanded text distribution, the normalized flow is sampled, and the latent variables are generated through the inverse flow transform. Finally, the decoder generates the speech output with the corresponding emotional characteristics.
2. The method for enhancing expressive speech synthesis based on conditional embedding according to claim 1, characterized in that: Processing and converting text into phoneme sequences, including: All texts have non-text elements removed; redundant spaces are eliminated, including redundant spaces between words and spaces at the beginning and end of lines; inappropriate line breaks and indentations are corrected; all texts are unified in simplified Chinese and UTF-8 encoding, and then the international phonetic alphabet is used to convert the text into a phoneme sequence.
3. The method for enhancing expressive speech synthesis based on conditional embedding according to claim 1, characterized in that: The sentiment predictor generates corresponding sentiment labels based on the text, including: The text is converted into a sentiment feature vector through the fine-tuned RoBERTa model, and then sent to a linear layer to map the sentiment feature vector to a predefined sentiment category space to generate a sentiment label.
4. The method for enhancing expressive speech synthesis based on conditional embedding according to claim 1, characterized in that: The encoding result is passed to the prosody predictor to generate frame-level latent features together with the global conditional variables, including: According to the global conditional vector and the encoding results generated by the text encoder, six one-dimensional residual convolution layers are used to predict the rhythmic changes at the frame level and obtain the frame-level latent features.
5. The method for enhancing expressive speech synthesis based on conditional embedding according to claim 1, characterized in that: The emotion label and the speaker label are input into the conditional encoder together, and feature fusion is performed to generate a global conditional vector, including: The emotion label and the speaker label are converted into their respective embedding vectors e and g by using the embedding layer. The embedding vectors e and g are stacked in the feature dimension, and the features are fused through the splicing operation to generate a comprehensive feature vector, which is encoded as a global condition vector.
6. The method for enhancing expressive speech synthesis based on conditional embedding according to claim 1, characterized in that: The training process includes: the preprocessed text phoneme sequence is encoded by the text encoder and then input into the prosody predictor together with the global condition variable H to generate P distribution; the preprocessed audio linear spectrum is also encoded into Q distribution; the Q distribution is sampled to obtain z and processed by flow to generate zp, which is then matched with the P distribution to generate duration d; Bring zp into each P distribution to maximize the likelihood value, input the text encoding h and duration d into the random duration predictor, and calculate the duration prediction loss L dur , copy the P distribution according to the predicted duration of d, make the number of distributions equal to the number of audio features, slice the latent variable z and pass it into the decoder to generate audio with emotion; During the entire training process, the embedding vector H generated by the conditional encoder is input to each module as a global condition. The true sentiment label is used as the input of the conditional encoder, and the loss L between the true label and the predicted label is calculated. pre ; During the training process, the KL divergence loss L between the Q distribution and the P distribution is also included kl , Mel reconstruction loss L of input audio features and output audio features recon , the adversarial loss L between the generator and the discriminator adv And the feature matching loss L fm .
7. An expressive speech synthesis enhancement system based on conditional embedding, characterized in that: The process includes: The global condition vector generating unit is configured as follows: the emotion predictor generates a corresponding emotion label according to the text, and the emotion label is input into the condition encoder together with the speaker label to perform feature fusion to generate a global condition vector; A frame-level latent feature generation unit is configured to: process the text and convert it into a phoneme sequence, encode the phoneme sequence through a text encoder, pass the encoding result to a prosody predictor, and generate a frame-level latent feature together with a global conditional variable; The predicted duration generation unit is configured as follows: the frame-level potential features are mapped into a set of Gaussian distributions μ and σ through a linear projection layer, a random duration prediction period generates a predicted duration according to a global condition vector and the encoding result, and a distribution of the text is expanded using the predicted duration; The speech output unit is configured to: sample a normalized flow according to the expanded text distribution, generate latent variables through inverse flow transformation, and finally generate speech output with corresponding emotional characteristics through a decoder.
8. A computer device, characterized in that: include: a processor and a computer readable storage medium; a processor adapted to execute a computer program; A computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by the processor, the expressive speech synthesis enhancement method based on conditional embedding as described in any one of claims 1 to 6 is implemented.
9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which is suitable for being loaded by a processor and executing the expressive speech synthesis enhancement method based on conditional embedding as described in any one of claims 1 to 6.
10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the method for enhancing expressive speech synthesis based on conditional embedding is implemented as described in any one of claims 1 to 6.