Voice synthesis model training method, voice synthesis method, device and apparatus

Abstract

The present disclosure provides a speech synthesis model training method, a speech synthesis method, an apparatus and a device, belonging to the technical field of computer. The first speech synthesis model comprises a language network and a diffusion network; the method comprises: processing a sample text and a sample audio through the language network to obtain a first audio feature, the sample text and the sample audio are consistent in content, and the first audio feature comprises a content feature of the sample text and an acoustic feature of the sample audio; performing noise prediction on the sample audio added with reference noise based on the first audio feature through the diffusion network to obtain a predicted noise; determining a first training loss based on the reference noise and the predicted noise; and updating parameters of the language network and the diffusion network in the first speech synthesis model based on the first training loss to obtain a second speech synthesis model. The above technical solution achieves the purpose of end-to-end training of the speech synthesis model, simplifies the training steps, and is beneficial to improving the training efficiency of the model.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a training method for a speech synthesis model, a speech synthesis method, an apparatus, and a device. Background Technology

[0002] With the rapid development of computer technology, speech synthesis technology has been widely applied in people's daily lives. Examples include voice assistants, navigation systems, smart home control, and reading software. It provides users with a more natural and convenient way to interact by converting text information into audible speech.

[0003] Currently, speech synthesis technology is mainly based on network models. Network models typically divide the text-to-speech processing into two stages: text-to-semantics and semantics-to-acoustics. First, the encoder in the network model is trained in a supervised manner to learn the semantic information of the text; then, the language model in the network model is trained in an unsupervised manner to learn how to convert semantic information into acoustic features; finally, speech is generated based on the acoustic features.

[0004] However, the above technical solutions require separate training for text-to-semantics and semantic-to-acoustic models, which affects the training efficiency of the model. Summary of the Invention

[0005] This disclosure provides a training method, speech synthesis method, apparatus, and device for a speech synthesis model, achieving end-to-end training of the speech synthesis model, simplifying the training steps, and improving the training efficiency of the model. The technical solution of this disclosure is as follows:

[0006] According to one aspect of the present disclosure, a method for training a speech synthesis model is provided. The first speech synthesis model includes a language network and a diffusion network. The language network is used to predict audio features of text, and the diffusion network is used to reconstruct audio based on the audio features of the text. The method includes:

[0007] The sample text and sample audio are processed by the language network to obtain a first audio feature, wherein the sample text and the sample audio have the same content, and the first audio feature includes the content features of the sample text and the acoustic features of the sample audio.

[0008] Based on the first audio feature, noise prediction is performed on the sample audio with added reference noise through the diffusion network to obtain the predicted noise.

[0009] Based on the reference noise and the predicted noise, a first training loss is determined, which represents the degree of difference between the reference noise and the predicted noise.

[0010] update parameters of the language network and the diffusion network in the first speech synthesis model based on the first training loss, to obtain a second speech synthesis model.

[0011] According to another aspect of the embodiments of the present disclosure, a speech synthesis method is provided, which comprises:

[0012] obtaining input text and prompt audio, the prompt audio being used to indicate acoustic information for speech synthesis;

[0013] processing the input text and the prompt audio through a language network in a second speech synthesis model to obtain intermediate audio features corresponding to the input text, the second speech synthesis model being trained based on the training method of the speech synthesis model described above;

[0014] performing denoising processing on the intermediate audio features through a diffusion network in the second speech synthesis model to obtain target audio features, and generating output audio based on the target audio features.

[0015] According to another aspect of the embodiments of the present disclosure, a training device of a speech synthesis model is provided, a first speech synthesis model comprising a language network and a diffusion network, the language network being used to predict audio features of text, and the diffusion network being used to reconstruct audio based on audio features of text; the device comprises:

[0016] a processing unit configured to perform processing on sample text and sample audio through the language network to obtain first audio features, the sample text and the sample audio being consistent in content, and the first audio features comprising content features of the sample text and acoustic features of the sample audio;

[0017] a prediction unit configured to perform noise prediction on the sample audio added with reference noise based on the first audio features through the diffusion network to obtain predicted noise;

[0018] a first determination unit configured to determine a first training loss based on the reference noise and the predicted noise, the first training loss being used to represent a difference degree between the reference noise and the predicted noise;

[0019] an update unit configured to update parameters of the language network and the diffusion network in the first speech synthesis model based on the first training loss, to obtain a second speech synthesis model.

[0020] In some embodiments, the processing unit comprises:

[0021] a clipping subunit configured to perform, based on content of the sample text, clipping a sample audio segment from the sample audio, the sample audio segment including sound of all or part of the content of the sample text;

[0022] a processing subunit configured to perform, by the language network, processing of the sample text and the sample audio segment to obtain the first audio feature.

[0023] In some embodiments, the clipping subunit is configured to perform obtaining, from the sample text, secondary content other than key content, the key content being used to represent a theme of the sample text; and clipping, based on the secondary content, the sample audio segment from the sample audio, the sample audio segment having content of the secondary content.

[0024] In some embodiments, the clipping subunit is configured to perform obtaining, from the content of the sample text, an emotion keyword, the emotion keyword being used to represent an emotion conveyed by the content of the sample text; and clipping, based on the emotion keyword, the sample audio segment from the sample audio, the sample audio segment including the emotion keyword.

[0025] In some embodiments, the sample audio includes speech of a plurality of objects;

[0026] the clipping subunit is configured to perform, for any object of the plurality of objects, clipping, based on the content of the sample text and an acoustic feature of the object, a sample audio segment corresponding to the object from the sample audio.

[0027] In some embodiments, the prediction unit is configured to perform obtaining a second audio feature corresponding to the sample audio and a reference noise feature, the reference noise feature being used to represent the reference noise; adding, by the diffusion network, the reference noise feature to the second audio feature to obtain a third audio feature; and performing, by the diffusion network, noise prediction on the third audio feature conditioned on the first audio feature to obtain a predicted noise feature, the predicted noise feature being used to represent the predicted noise.

[0028] the first determination unit is configured to perform determining, based on the reference noise feature and the predicted noise feature, the first training loss.

[0029] In some embodiments, the apparatus further comprises:

[0030] The second determining unit is configured to determine a second training loss based on the first audio feature and a second audio feature corresponding to the sample audio, the second audio feature including a content feature and an acoustic feature in the sample audio, the second training loss being used to represent a difference degree between the first audio feature and the second audio feature.

[0031] The updating unit is configured to update the language network and the diffusion network in the first speech synthesis model based on the first training loss and the second training loss to obtain the second speech synthesis model.

[0032] In some embodiments, the apparatus further includes:

[0033] The third determining unit is configured to remove the predicted noise from the sample audio to which the reference noise is added to obtain target audio, and determine a third training loss based on the sample audio and the target audio, the third training loss being used to represent a difference degree between the sample audio and the target audio.

[0034] The updating unit is configured to update the language network and the diffusion network in the first speech synthesis model based on the first training loss and the third training loss to obtain the second speech synthesis model.

[0035] In some embodiments, the processing unit includes:

[0036] The obtaining sub-unit is configured to obtain a phoneme sequence of the sample text and an acoustic feature of the sample audio.

[0037] The processing sub-unit is configured to process the phoneme sequence and the acoustic feature through the language network to obtain the first audio feature.

[0038] In some embodiments, the obtaining sub-unit is configured to perform preprocessing on the sample text, the preprocessing including at least one of word segmentation, noise removal, text normalization, and multi-pronunciation annotation, and convert the preprocessed sample text into the phoneme sequence based on a corresponding relationship between characters and phonemes.

[0039] According to another aspect of the embodiments of the present disclosure, a speech synthesis apparatus is provided, and the apparatus includes:

[0040] The obtaining unit is configured to obtain input text and prompt audio, the prompt audio being used to indicate acoustic information for speech synthesis.

[0041] The first processing unit is configured to perform processing on the input text and the prompt audio by a language network in a second speech synthesis model to obtain intermediate audio features corresponding to the input text, the second speech synthesis model being trained based on the training method of the speech synthesis model.

[0042] The second processing unit is configured to perform denoising processing on the intermediate audio features by a diffusion network in the second speech synthesis model to obtain target audio features, and generate output audio based on the target audio features.

[0043] According to another aspect of the embodiments of the present disclosure, an electronic device is provided, which includes:

[0044] one or more processors;

[0045] a memory for storing program code executable by the processor;

[0046] The processor is configured to execute the program code to implement the training method of the speech synthesis model or the speech synthesis method.

[0047] According to another aspect of the embodiments of the present disclosure, a computer readable storage medium is provided, when program code in the computer readable storage medium is executed by a processor of an electronic device, the electronic device can execute the training method of the speech synthesis model or the speech synthesis method.

[0048] According to another aspect of the embodiments of the present disclosure, a computer program product is provided, which includes computer programs / instructions that are executed by a processor to implement the training method of the speech synthesis model or the speech synthesis method.

[0049] The embodiments of the present disclosure provide a training method of a speech synthesis model, which adopts content-consistent sample text and sample audio to jointly train a language network and a diffusion network in the speech synthesis model, so that the speech synthesis model can learn as a whole to convert text into audio containing acoustic features, achieving the purpose of end-to-end training of the speech synthesis model, simplifying the training steps, and being conducive to improving the training efficiency of the model; and the diffusion network is adopted to learn to reconstruct audio, the speech synthesis model is trained by a first training loss, so that the difference between the reference noise and the predicted noise becomes smaller and smaller, i.e., the diffusion network can accurately learn the noise in the sample audio, ensuring that the diffusion network can reconstruct more accurate audio, i.e., being conducive to improving the accuracy of the speech synthesis model, so as to generate audio with higher expressiveness and naturalness, and higher quality.

[0050] In addition, compared with the scheme of "learning semantic information of text first, and then converting the semantic information into audio containing acoustic features" in the related art, the scheme does not need to encode the text semantically, thereby avoiding the dependence of audio reconstruction on the accuracy of semantic encoding, and facilitating the guarantee of the reliability and stability of the speech synthesis model.

[0051] It should be understood that the foregoing general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0052] The accompanying drawings incorporated in the specification and forming a part of it, illustrate embodiments consistent with the present disclosure and together with the description, serve to explain the principles of the present disclosure, and are not intended to limit the present disclosure in any manner.

[0053] Figure 1 is a schematic diagram of an implementation environment of a training method of a speech synthesis model according to an exemplary embodiment.

[0054] Figure 2 is a flowchart of a training method of a speech synthesis model according to an exemplary embodiment.

[0055] Figure 3 is a flowchart of another training method of a speech synthesis model according to an exemplary embodiment.

[0056] Figure 4 is a schematic diagram of a training process of a first speech synthesis model according to an exemplary embodiment.

[0057] Figure 5 is a flowchart of a speech synthesis method according to an exemplary embodiment.

[0058] Figure 6 is a block diagram of a training device of a speech synthesis model according to an exemplary embodiment.

[0059] Figure 7 is a block diagram of a speech synthesis device according to an exemplary embodiment.

[0060] Figure 8 is a block diagram of a terminal according to an exemplary embodiment.

[0061] Figure 9 is a block diagram of a server according to an exemplary embodiment. DETAILED DESCRIPTION

[0062] In order to make ordinary people in the art better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be described clearly and completely below in conjunction with the drawings.

[0063] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0064] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data for analysis, stored data, displayed data, etc.) and signals involved in the present disclosure are authorized by the user or fully authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of relevant countries and regions. For example, the sample text, sample audio, input text, prompt audio involved in the present disclosure are obtained under full authorization.

[0065] Figure 1 is a schematic diagram of an implementation environment of a training method of a speech synthesis model according to an exemplary embodiment. Referring to Figure 1 , the implementation environment specifically includes a terminal 101 and a server 102. The terminal 101 can be connected to the server 102 through a wireless network or a wired network.

[0066] The terminal 101 is at least one of a smart phone, a smart watch, a desktop computer, a laptop computer, an MP3 player, an MP4 player, and a laptop computer. The terminal 101 has an application program supporting speech synthesis installed and running thereon. The application program can be a voice assistant, a social application program, a navigation application program, or a video application program, etc., which is not limited by the embodiments of the present disclosure. The user can log in to the application program through the terminal 101 to obtain the services provided by the application program. Taking the navigation application program as an example, the user can perform voice interaction with the navigation application program through the terminal 101. Correspondingly, the navigation application program can understand the address input by the user, generate route text corresponding to the address, and convert the route text into navigation voice through speech synthesis technology to play the navigation voice to the user. The application program is connected to the server 102, and the server 102 provides background services.

[0067] The terminal 101 refers to one of a plurality of terminals, and the embodiment is exemplified by the terminal 101. It can be understood by those skilled in the art that the number of the terminals can be more or less. For example, the terminals can be several, or the terminals can be tens or hundreds, or more, and the number of the terminals and the type of the equipment are not limited in the embodiment.

[0068] In some embodiments, the server 102 is at least one of a server, a plurality of servers, a cloud computing platform, and a virtualization center. Optionally, the number of the servers can be more or less, and the embodiment is not limited thereto. Of course, the server 102 can also include other functional servers to provide more comprehensive and diversified services. In some embodiments, the server 102 undertakes the main computing work, and the terminal 101 undertakes the secondary computing work; or the server 102 undertakes the secondary computing work, and the terminal 101 undertakes the main computing work; or the server 102 and the terminal 101 cooperatively compute in a distributed computing architecture. The server 102 can be connected with the terminal 101 and other terminals through a wireless network or a wired network, and optionally, the number of the servers can be more or less, and the embodiment is not limited thereto.

[0069] Figure 2 is a flowchart of a training method of a speech synthesis model according to an exemplary embodiment, as shown in Figure 2 The training method of the speech synthesis model is applied in an electronic device, and includes the following steps:

[0070] In step 201, the electronic device processes the sample text and the sample audio through a language network in a first speech synthesis model to obtain first audio features. The sample text and the sample audio have consistent content, and the first audio features include content features of the sample text and acoustic features of the sample audio.

[0071] In the embodiment, the first speech synthesis model is a speech synthesis model to be trained. The first speech synthesis model includes a language network and a diffusion network. The language network is used to predict audio features of the text. The diffusion network is used to reconstruct audio based on the audio features of the text.

[0072] The electronic device inputs the sample text and the sample audio into the language network in the first speech synthesis model, processes the sample text and the sample audio through the language network, extracts the content features of the sample text and the acoustic features of the sample audio, and obtains the first audio features corresponding to the sample text. The first audio features are the audio features of the sample text predicted by the language network.

[0073] The sample text and the sample audio content are consistent. That is, the language network can learn acoustic features to be possessed in the audio features to be predicted by taking the sample audio as a prompt, and combine content features of the sample text with the acoustic features to obtain the first audio features. The acoustic features are used to indicate features of acoustic performances such as tone color, tone intensity, tone height, tone length, and the like in the sample audio. The language network can be any kind of Auto Regressive Language Model (ARLM), for example, GPT (Generative Pre-Trained Transformer), and the like. The architecture of the language network is not limited in the embodiments of the present disclosure.

[0074] In step 202, the electronic device performs noise prediction on the sample audio added with the reference noise based on the first audio features by the diffusion network in the first speech synthesis model to obtain predicted noise.

[0075] In the embodiments of the present disclosure, after the language network outputs the first audio features, the electronic device inputs the first audio features into the diffusion network in the first speech synthesis model, and performs noise prediction on the sample audio added with the reference noise based on the first audio features by the diffusion network to obtain predicted noise. Then, the electronic device removes the predicted noise from the sample audio added with the reference noise by the prediction network to obtain predicted audio. The predicted audio is the audio reconstructed based on the text features by the diffusion network.

[0076] The predicted audio is the audio finally synthesized by the first speech synthesis model based on the sample text. Since the predicted audio is obtained by removing the predicted noise from the sample audio added with the reference noise, and the sample audio and the reference noise are known, the accuracy of the predicted audio depends on the accuracy of the predicted noise. Accordingly, the performance of the diffusion network in determining the predicted noise can reflect the performance (i.e., the overall performance of the model) of the first speech synthesis model in outputting the predicted audio. Therefore, the electronic device can determine the loss of the first speech synthesis model according to the error of the predicted noise by the diffusion network, so as to perform model training. Please refer to the subsequent steps for details.

[0077] In step 203, the electronic device determines a first training loss based on the reference noise and the predicted noise, where the first training loss is used to represent the difference between the reference noise and the predicted noise.

[0078] In the embodiments of the present disclosure, the electronic device can determine the first training loss of the first speech synthesis model according to the difference between the reference noise and the predicted noise. The greater the first training loss is, the greater the difference between the reference noise and the predicted noise is; the smaller the first training loss is, the smaller the difference between the reference noise and the predicted noise is.

[0079] In the calculation of the first training loss, the electronic device can use a mean square error calculation method, a mean absolute error calculation method, a cross-entropy loss calculation method, etc. The calculation method of the first training loss is not limited in the embodiments of the present disclosure.

[0080] In step 204, the electronic device updates the parameters of the language network and the diffusion network in the first speech synthesis model based on the first training loss to obtain a second speech synthesis model.

[0081] In the embodiments of the present disclosure, the electronic device updates the parameters of the language network and the diffusion network in the first speech synthesis model to minimize the first training loss until the second speech synthesis model obtained after the parameter update satisfies the training stop condition. The training stop condition can be that the first training loss is lower than the first loss threshold, or that the number of parameter updates reaches a preset number, etc. The embodiments of the present disclosure are not limited in this regard.

[0082] The embodiments of the present disclosure provide a speech synthesis model training method, which uses content-consistent sample text and sample audio to jointly train the language network and the diffusion network in the speech synthesis model, so that the speech synthesis model can learn as a whole to convert text into audio containing acoustic features, achieving the purpose of end-to-end training of the speech synthesis model, simplifying the training steps, and being beneficial to improving the training efficiency of the model; and, the diffusion network is used to learn to reconstruct the audio, and the speech synthesis model is trained through the first training loss, so that the difference between the reference noise and the predicted noise becomes smaller and smaller, i.e., the diffusion network can accurately learn the noise in the sample audio, ensuring that the diffusion network can reconstruct more accurate audio, i.e., being beneficial to improving the accuracy of the speech synthesis model, so as to generate audio with higher expressiveness and naturalness, and higher quality.

[0083] In addition, compared with the scheme of "first learning semantic information of text, and then converting the semantic information into audio containing acoustic features" in the related art, the present scheme also does not need to encode the semantic information of the text, thereby avoiding the dependence of audio reconstruction on the accuracy of semantic encoding, and being beneficial to ensuring the reliability and stability of the speech synthesis model.

[0084] In some embodiments, the sample text and the sample audio are processed through the language network to obtain the first audio feature, including:

[0085] Based on the content of the sample text, a sample audio segment is intercepted from the sample audio, and the sample audio segment includes the sound of all or part of the content in the sample text.

[0086] The sample text and the sample audio segment are processed through the language network to obtain the first audio feature.

[0087] The scheme provided by the embodiments of the present disclosure can extract a sample audio segment from the sample audio according to the content of the sample text, and then process the sample text and the sample audio segment through a language network to obtain first audio features. Since the sample audio segment has a smaller data volume than the sample audio and contains the sound of all or part of the content of the sample text, the processing efficiency of the language network can be improved, and the acoustic features used for speech synthesis can be accurately learned, which is conducive to improving the accuracy of the audio features generated by the language network. Moreover, in the case where the sample audio segment only contains the sound of part of the content of the sample text, the language network can be prevented from learning the content of the sample text from the sample audio segment as much as possible, that is, the content learned by the language network is ensured to come from the sample text rather than the sample audio, which is conducive to improving the accuracy of the language network in learning the content of the sample text, thereby improving the accuracy and training efficiency of the speech synthesis model.

[0088] In some embodiments, the sample audio segment is extracted from the sample audio based on the content of the sample text, including:

[0089] The secondary content is obtained from the sample text, and the key content is used to represent the theme of the sample text.

[0090] The sample audio segment is extracted from the sample audio based on the secondary content, and the content of the sample audio segment is the secondary content.

[0091] The scheme provided by the embodiments of the present disclosure can extract a sample audio segment from the sample audio based on the secondary content in the sample text. Since the sample audio segment has a smaller data volume than the sample audio and contains the sound of the secondary content in the sample text, the processing efficiency of the language network can be improved, and the acoustic features used for speech synthesis can be accurately learned, which is conducive to improving the accuracy of the audio features generated by the language network. Moreover, since the sample audio segment only contains the secondary content in the sample text, the language network can be prevented from learning the key content of the sample text from the sample audio segment as much as possible, that is, the content learned by the language network is ensured to come from the sample text rather than the sample audio, which is conducive to improving the accuracy of the language network in learning the content of the sample text, thereby improving the accuracy and training efficiency of the speech synthesis model.

[0092] In some embodiments, the sample audio segment is extracted from the sample audio based on the content of the sample text, including:

[0093] The emotional keyword is obtained from the content of the sample text, and the emotional keyword is used to represent the emotion conveyed by the content of the sample text.

[0094] The sample audio segment is extracted from the sample audio based on the emotional keyword, and the sample audio segment includes the emotional keyword.

[0095] The scheme provided by the embodiment of the present disclosure is based on the emotional keywords in the sample text, and a sample audio segment is intercepted from the sample audio. Since the data amount of the sample audio segment is smaller than that of the sample audio and the sample audio segment contains the sound of part of the content in the sample text, the processing efficiency of the language network can be improved, and the acoustic characteristics used for speech synthesis can be accurately learned, which is beneficial to improving the accuracy of the audio features generated by the language network. Moreover, since the sample audio segment contains the emotional keywords in the sample text, the language network can learn the main content of the sample text from the sample audio segment as much as possible, that is, the content learned by the language network is ensured to come from the sample text rather than the sample audio, which is beneficial to improving the accuracy of the language network in learning the content of the sample text, and the emotion required by the audio of the sample text can also be learned from the sample audio segment, which is beneficial to generating audio with more accurate and natural emotion and higher quality.

[0096] In some embodiments, the sample audio includes speech of a plurality of objects;

[0097] Based on the content of the sample text, a sample audio segment is intercepted from the sample audio, including:

[0098] For any object in the plurality of objects, based on the content of the sample text and the acoustic characteristics of the object, a sample audio segment corresponding to the object is intercepted from the sample audio.

[0099] The scheme provided by the embodiment of the present disclosure is based on the emotional keywords in the sample text, and a sample audio segment is intercepted from the sample audio. Since the data amount of the sample audio segment is smaller than that of the sample audio and the sample audio segment contains the sound of part of the content in the sample text, the processing efficiency of the language network can be improved, and the acoustic characteristics used for speech synthesis can be accurately learned, which is beneficial to improving the accuracy of the language network. Moreover, since the sample audio segment contains the emotional keywords in the sample text, the language network can learn the main content of the sample text from the sample audio segment as much as possible, that is, the content learned by the language network is ensured to come from the sample text rather than the sample audio, which is beneficial to improving the accuracy of the language network in learning the content of the sample text, and the emotion required by the audio of the sample text can also be learned from the sample audio segment, which is beneficial to generating audio with more accurate and natural emotion and higher quality.

[0100] In some embodiments, by using the diffusion network, the sample audio to which the reference noise is added is subjected to noise prediction based on the first audio feature, to obtain a predicted noise, including:

[0101] The second audio feature corresponding to the sample audio and the reference noise feature are obtained, and the reference noise feature is used to represent the reference noise;

[0102] The reference noise feature is added to the second audio feature by using the diffusion network, to obtain a third audio feature;

[0103] The third audio feature is subjected to noise prediction by using the diffusion network, with the first audio feature as a condition, to obtain a predicted noise feature, and the predicted noise feature is used to represent the predicted noise;

[0104] determine a first training loss based on the reference noise and the predicted noise, including:

[0105] determine a first training loss based on the reference noise feature and the predicted noise feature.

[0106] The scheme provided by the embodiments of the present disclosure uses a diffusion network to learn to reconstruct audio, that is, first adds noise to sample audio through the diffusion network, and then predicts the added noise conditioned on the first audio feature predicted by the language network, so as to reconstruct the audio by removing the predicted noise. Since the sample audio and the reference noise are known, the accuracy of the predicted audio depends on the accuracy of the predicted noise, that is, the performance of the diffusion network in determining the predicted noise can reflect the performance of the first speech synthesis model in outputting the predicted audio (that is, the overall performance of the model). Accordingly, the speech synthesis model is trained through the first training loss, so that the difference between the reference noise and the predicted noise becomes smaller and smaller, that is., the diffusion network can accurately learn the noise in the sample audio, ensuring that the diffusion network can reconstruct more accurate audio, which is conducive to improving the accuracy of the speech synthesis model, so as to generate audio with higher expressiveness and naturalness and higher quality.

[0107] In some embodiments, the method further includes:

[0108] determine a second training loss based on the first audio feature and a second audio feature corresponding to the sample audio, the second audio feature including a content feature and an acoustic feature in the sample audio, the second training loss being used to represent a difference between the first audio feature and the second audio feature;

[0109] update parameters of the language network and the diffusion network in the first speech synthesis model based on the first training loss, including:

[0110] update parameters of the language network and the diffusion network in the first speech synthesis model based on the first training loss and the second training loss, to obtain a second speech synthesis model.

[0111] The scheme provided by the embodiments of the present disclosure, since the first audio feature is the feature of the audio of the sample text predicted by the language network, and the second audio feature is the feature of the real audio, therefore, the second training loss can accurately reflect the loss of the audio predicted by the language network. By training the speech synthesis model through the first training loss and the second training loss, the speech synthesis model is trained from two aspects of predicting noise by the diffusion network and predicting audio by the language network, which is conducive to improving the accuracy of the speech synthesis model, so as to generate audio with higher expressiveness and naturalness and higher quality.

[0112] In some embodiments, the method further includes:

[0113] remove the predicted noise from the sample audio to which the reference noise is added, to obtain target audio.

[0114] determine a third training loss based on the sample audio and the target audio, the third training loss being used to represent a degree of difference between the sample audio and the target audio;

[0115] update parameters of the language network and the diffusion network in the first speech synthesis model based on the first training loss to obtain a second speech synthesis model, including:

[0116] update parameters of the language network and the diffusion network in the first speech synthesis model based on the first training loss and the third training loss to obtain the second speech synthesis model.

[0117] The scheme provided by the embodiments of the present disclosure can accurately reflect the loss of the audio generated by the speech synthesis model as a whole, and train the speech synthesis model from the audio predicted by the speech synthesis model as a whole, which is conducive to improving the accuracy of the speech synthesis model so as to generate audio with higher expressiveness and naturalness and higher quality.

[0118] In some embodiments, the sample text and the sample audio are processed by the language network to obtain the first audio feature, including:

[0119] obtain a phoneme sequence of the sample text and an acoustic feature of the sample audio;

[0120] The phoneme sequence and the acoustic feature are processed by the language network to obtain the first audio feature.

[0121] The scheme provided by the embodiments of the present disclosure can train the ability of the language network to predict the audio feature according to the phoneme sequence and the acoustic feature, and further improve the accuracy of the language network in predicting the audio feature.

[0122] In some embodiments, the process of obtaining the phoneme sequence of the sample text includes:

[0123] The sample text is preprocessed, and the preprocessing includes at least one of word segmentation, noise removal, text normalization, and multi-pronunciation word pronunciation annotation;

[0124] The preprocessed sample text is converted into a phoneme sequence based on the correspondence between characters and phonemes.

[0125] The scheme provided by the embodiments of the present disclosure can standardize at least one of the content and the pronunciation of the sample text by preprocessing the sample text, so as to accurately convert the sample text into a phoneme sequence and improve the accuracy of the phoneme sequence.

[0126] The above Figure 2 The above Figure 3 is a flowchart of another method for training a voice synthesis model according to an example embodiment. Taking an electronic device as an example, the electronic device is provided, and referring to Figure 3 , the method comprises:

[0127] In step 301, the electronic device extracts a sample audio segment from the sample audio based on the content of the sample text, the sample text and the sample audio content are consistent, and the sample audio segment includes the sound of all or part of the content in the sample text.

[0128] In the embodiments of the present disclosure, the electronic device can directly train the first voice synthesis model based on the sample text and the sample audio. Alternatively, considering that the sample audio data is large, and accordingly the data amount of the audio features to be processed by the model is large, the electronic device can also extract part of the audio from the sample audio for training according to the sample text. The electronic device can randomly extract the sample audio segment from the sample audio, and can also extract the sample audio segment according to the content of the sample text, and the embodiments of the present disclosure do not limit the way of extracting the sample audio segment. For example, the part of the sample audio that contains the sound of all the content in the sample text is equivalent to removing the blank or background sound part in the sample audio. The following exemplary introduces three ways of extracting the sample audio segment, but is by no means limited thereto.

[0129] The first way is that the electronic device can extract the sample audio segment according to the secondary content in the sample text. Accordingly, the process of the electronic device extracting the sample audio segment from the sample audio comprises: the electronic device obtains the secondary content in the sample text except the key content. Then, the electronic device extracts the sample audio segment from the sample audio based on the secondary content, and the content of the sample audio segment is the secondary content. The key content is used to represent the theme of the sample text. The electronic device can determine the key content in the sample text by performing theme recognition on the sample text, so as to obtain the secondary content in the sample text. The secondary content can be auxiliary description (such as some adjectives) in the sample text, background information (such as the first sentence in the text), etc., which are not limited by the embodiments of the present disclosure.

[0130] The scheme provided by the embodiments of the present disclosure is based on the secondary content in the sample text, and a sample audio segment is intercepted from the sample audio. Since the sample audio segment has a smaller data volume than the sample audio and contains the sound of the secondary content in the sample text, the processing efficiency of the language network can be improved, and the acoustic features used for speech synthesis can be accurately learned, which is beneficial to improving the accuracy of the audio features generated by the language network. Moreover, since the sample audio segment only contains the secondary content in the sample text, the language network can be prevented from learning the key content of the sample text from the sample audio segment, that is, the content learned by the language network is ensured to come from the sample text rather than the sample audio, which is beneficial to improving the accuracy of the language network in learning the content of the sample text, thereby improving the accuracy and training efficiency of the speech synthesis model.

[0131] In the second mode, the electronic device can intercept the sample audio segment based on the emotional keywords in the sample text. Accordingly, the process in which the electronic device intercepts the sample audio segment from the sample audio includes that the electronic device acquires the emotional keywords from the content of the sample text. Then, the electronic device intercepts the sample audio segment from the sample audio based on the emotional keywords. The sample audio segment includes the emotional keywords. The emotional keywords are used to represent the emotion conveyed by the content of the sample text. The emotional keywords can be happy, angry, contemptuous, etc., which are not limited by the embodiments of the present disclosure. The electronic device can determine the emotional keywords in the sample text by performing emotion recognition on the sample text. The number of emotional keywords is not limited by the embodiments of the present disclosure.

[0132] The scheme provided by the embodiments of the present disclosure is based on the emotional keywords in the sample text, and a sample audio segment is intercepted from the sample audio. Since the sample audio segment has a smaller data volume than the sample audio and contains the sound of part of the content in the sample text, the processing efficiency of the language network can be improved, and the acoustic features used for speech synthesis can be accurately learned, which is beneficial to improving the accuracy of the audio features generated by the language network. Moreover, since the sample audio segment contains the emotional keywords in the sample text, the language network can be prevented from learning the main content of the sample text from the sample audio segment, that is, the content learned by the language network is ensured to come from the sample text rather than the sample audio, which is beneficial to improving the accuracy of the language network in learning the content of the sample text, and the emotion required by the audio of the sample text can also be learned from the sample audio segment, which is beneficial to generating audio with more accurate and natural emotion and higher quality.

[0133] In the third mode, the sample audio includes speech of multiple objects, and the electronic device can intercept speech of any object for model training. Accordingly, the process of the electronic device intercepting the sample audio segment from the sample audio includes: for any object in the multiple objects, the electronic device intercepts a sample audio segment corresponding to the object from the sample audio based on the content of the sample text and the acoustic feature of the object. The object can be the object to which the acoustic feature used for speech synthesis belongs. That is, the electronic device can first determine the object to which the acoustic feature used for speech synthesis belongs, and then intercept the corresponding sample audio segment from the sample audio. The object can be specified by a user, and the embodiments of the present disclosure do not limit this.

[0134] The embodiments of the present disclosure provide a scheme. The sample audio includes speech of multiple objects. A sample audio segment of any object can be intercepted from the sample audio according to an acoustic feature of the object, so as to generate audio based on the acoustic feature of the object subsequently. That is, the speech synthesis model can learn the acoustic feature of any object to generate audio, which guarantees the flexibility of model training, i.e., a user can select an audio segment of a certain object to train the model according to the user's own needs, so that the speech synthesis model can learn the acoustic feature of the object sufficiently, which is beneficial to subsequent generation of audio conforming to the object.

[0135] Before intercepting the sample audio segment from the sample text, the electronic device can first preprocess the sample text, and then intercept the sample audio segment according to the preprocessed sample text, so as to improve the quality of the sample audio segment.

[0136] The preprocessing includes at least one of word segmentation, noise removal, text normalization, and annotation of readings of multi-syllable characters. The word segmentation refers to dividing the sample text into multiple characters. The noise removal refers to removing meaningless symbols (such as punctuation marks, emoticons, etc.) in the sample text. The text normalization refers to converting English abbreviations, numbers, etc. in the sample text into standard forms. For example, converting “TTS” in the sample text into “Text To Speech”, and converting “857” in the sample text into “eight hundred and fifty-seven”. The annotation of readings of multi-syllable characters refers to annotating readings of multi-syllable characters in the sample text. In the process of annotating the readings, the electronic device can determine the readings of the multi-syllable characters according to the phrase to which the multi-syllable characters belong, and the embodiments of the present disclosure do not limit this. For example, in the case where the sample text includes the phrase “body weight”, the electronic device can annotate the reading of “weight” as “zhong-4th tone”. In the case where the sample text includes the phrase “repeat”, the electronic device can annotate the reading of “weight” as “chong-2nd tone”.

[0137] In step 302, the electronic device processes the sample text and the sample audio segment through the language network in the first speech synthesis model to obtain a first audio feature, where the first audio feature includes the content feature of the sample text and the acoustic feature of the sample audio.

[0138] In the embodiments of the present disclosure, the electronic device may input the sample text and the sample audio segment into the language network in the first speech synthesis model, and process the sample text and the sample audio segment through the language network to extract the content feature of the sample text and the acoustic feature of the sample audio segment, so as to obtain the first audio feature corresponding to the sample text. Alternatively, the electronic device may also extract the content feature of the sample text and the acoustic feature of the sample audio segment through other network models outside the language network, and then input the content feature of the sample text and the acoustic feature of the sample audio segment into the language network for processing, so as to obtain the first audio feature corresponding to the sample text. The embodiments of the present disclosure do not limit this. The first audio feature is the audio feature of the sample text predicted by the language network.

[0139] In some embodiments, the language network can predict the first audio feature according to the factor sequence of the sample text and the acoustic feature of the sample audio. Accordingly, the electronic device obtains the phoneme sequence of the sample text and the acoustic feature of the sample audio. Then, the electronic device processes the phoneme sequence and the acoustic feature through the language network to obtain the first audio feature. That is, the electronic device obtains the phoneme sequence of the sample text and the acoustic feature of the sample audio segment. Then, the electronic device processes the phoneme sequence and the acoustic feature through the language network to obtain the first audio feature.

[0140] Among them, the phoneme sequence includes the phonemes of multiple characters in the sample text. The phoneme sequence can represent the text content of the sample text. For example, when the sample text contains the character "今", the phoneme sequence includes the three phonemes "ji n" of this character. Then, the electronic device inputs the phoneme sequence of the sample text and the acoustic feature of the sample audio segment into the language network, and processes the phoneme sequence and the acoustic feature through the language network to predict the first audio feature. The solution provided by the embodiments of the present disclosure can train the ability of the language network to predict the audio feature according to the phoneme sequence and the acoustic feature by inputting the phoneme sequence of the sample text and the acoustic feature of the sample audio into the language network for processing, and further improve the accuracy of the language network in predicting the audio feature.

[0141] In the process of obtaining the phoneme sequence of the sample text, the electronic device can pre-process the sample text. The pre-processing includes at least one of word segmentation, denoising, text normalization, and multi-sound character pronunciation annotation. Then, the electronic device converts the pre-processed sample text into a phoneme sequence based on the correspondence between characters and phonemes. That is, the electronic device determines the factors of each character in the sample text according to the correspondence between characters and phonemes. Then, the electronic device sorts the audio of each character according to the arrangement order of each character in the sample text to obtain the phoneme sequence corresponding to the sample text. The scheme provided in the embodiments of the present disclosure can standardize at least one of the content and pronunciation of the sample text by pre-processing the sample text, so as to more accurately convert the sample text into a phoneme sequence, thereby improving the accuracy of the phoneme sequence.

[0142] In the process of obtaining the phoneme sequence, the electronic device can use a G2P (Grapheme-to-Phoneme, word element to phoneme) model to process the pre-processed sample text to obtain the phoneme sequence of the sample text. The embodiments of the present disclosure do not limit the manner of obtaining the phoneme sequence. In the process of obtaining the acoustic feature, the electronic device can use a Prompt Encoder to process the sample audio segment to obtain the acoustic feature of the sample audio segment, so as to reflect the acoustic information such as timbre, intensity, pitch, and length used for speech synthesis.

[0143] In step 303, the electronic device performs noise prediction on the sample audio to which the reference noise is added based on the first audio feature through the diffusion network in the first speech synthesis model, to obtain the predicted noise.

[0144] In the embodiments of the present disclosure, the diffusion network includes a forward diffusion process and a reverse diffusion process. The forward diffusion process of the diffusion network is to add noise to the sample audio. The reverse diffusion process of the diffusion network is a process of predicting and removing noise from the sample audio to which the noise is added. After the language network outputs the first audio feature, the first audio feature is input into the diffusion network in the first speech synthesis model. The electronic device performs noise prediction on the sample audio to which the reference noise is added based on the first audio feature through the diffusion network, to obtain the predicted noise. Then, the electronic device removes the predicted noise from the sample audio to which the reference noise is added through the prediction network, to obtain the predicted audio. The predicted audio is the audio used by the diffusion network for text-based audio feature reconstruction.

[0145] In the forward diffusion process, the electronic device adds the n-th noise (the n-th noise is the reference noise) to the sample audio through the diffusion network, so that the sample audio changes from a clear state to a completely random state. This process can be regarded as diffusion of data in noise, and by gradually increasing the noise, the information of the original data is gradually covered. In the reverse diffusion process, the electronic device gradually removes the predicted noise (i.e., learns the noise distribution in the forward diffusion process to remove the n-th noise) from the sample audio with the added reference noise through the diffusion network conditioned on the first audio feature, so that the predicted audio after removing the predicted noise meets the first audio feature.

[0146] In some embodiments, the above-mentioned processes of adding noise, predicting noise, and removing noise are all processed in the dimension of the feature. Accordingly, the electronic device obtains a second audio feature corresponding to the sample audio and a reference noise feature used to represent the reference noise. The electronic device adds the reference noise feature to the second audio feature through the diffusion network to obtain a third audio feature. Then, the electronic device performs noise prediction on the third audio feature through the diffusion network conditioned on the first audio feature to obtain a predicted noise feature. The predicted noise feature is used to represent the predicted noise. Subsequently, the electronic device can determine the first training loss based on the reference noise feature and the predicted noise feature.

[0147] The scheme provided by the embodiments of the present disclosure uses the diffusion network to learn to reconstruct the audio, that is, first adds noise to the sample audio through the diffusion network, and then predicts the added noise conditioned on the first audio feature predicted by the language network, so as to reconstruct the audio by removing the predicted noise. Since the sample audio and the reference noise are known, the accuracy of the predicted audio depends on the accuracy of the predicted noise, that is, the performance of the diffusion network in determining the predicted noise can reflect the performance of the first speech synthesis model in outputting the predicted audio (i.e., the overall performance of the model). Accordingly, the speech synthesis model is trained through the first training loss, so that the difference between the reference noise and the predicted noise becomes smaller and smaller, that is, the diffusion network can accurately learn the noise in the sample audio, which ensures that the diffusion network can reconstruct more accurate audio, that is, is conducive to improving the accuracy of the speech synthesis model, so as to generate audio with higher expressiveness and naturalness and higher quality.

[0148] The reference noise can be Gaussian noise randomly generated by the diffusion network, which is not limited by the embodiments of the present disclosure.

[0149] In step 304, the electronic device determines a first training loss based on the reference noise and the predicted noise, the first training loss being used to represent the difference between the reference noise and the predicted noise.

[0150] In the embodiments of the present disclosure, the electronic device can determine the first training loss of the first speech synthesis model according to the difference between the reference noise and the predicted noise. The greater the first training loss is, the greater the difference between the reference noise and the predicted noise is; the smaller the first training loss is, the smaller the difference between the reference noise and the predicted noise is.

[0151] In some embodiments, the first training loss is a mean square error loss. Accordingly, the electronic device calculates the mean square error loss between the reference noise feature and the predicted noise according to a mean square error loss function, and determines the mean square error loss as the first training loss. The reference noise feature is used to represent the reference noise; the predicted noise feature is used to represent the predicted noise. By determining the mean square error loss between the reference noise and the predicted noise as the second training loss of the speech synthesis model, the second training loss can accurately reflect the difference between the predicted noise and the reference noise predicted by the speech synthesis model, thereby improving the accuracy and efficiency of training the first speech synthesis model through the first training loss.

[0152] In step 305, the electronic device updates the parameters of the language network and the diffusion network in the first speech synthesis model based on the first training loss to obtain a second speech synthesis model.

[0153] In the embodiments of the present disclosure, the electronic device can update the parameters of the language network and the diffusion network in the first speech synthesis model in multiple rounds of iterations with the goal of minimizing the first training loss, until the second speech synthesis model obtained after the parameter update satisfies the training stop condition. The training stop condition can be that the first training loss is lower than a first loss threshold, or that the number of parameter updates reaches a preset number, and the present disclosure does not limit this.

[0154] In order to more clearly describe the training process of the first speech synthesis model, the training process is further described below in conjunction with the accompanying drawings. Figure 4 is a schematic diagram of a training process of a first speech synthesis model according to an exemplary embodiment. As shown in Figure 4As shown, the first speech synthesis model includes a language network and a diffusion network. The electronic device converts the sample text into a phoneme sequence (Phoneme embedding) through a G2P model. The electronic device extracts acoustic features of the sample audio (Prompt embedding) through a Prompt Encoder. Then, the electronic device inputs the phoneme sequence and the acoustic features into the language network in the first speech synthesis model, processes the phoneme sequence and the acoustic features through the language network, and obtains first audio features (a series of hidden layer variables). Then, the electronic device performs noise prediction on third audio features (second audio features added with reference noise features) based on the first audio features through the diffusion network in the first speech synthesis model, and obtains predicted noise features. Then, the electronic device determines a first training loss based on the reference noise features and the predicted noise features, trains the first speech synthesis model based on the first training loss, and obtains a second speech synthesis model.

[0155] In some embodiments, in addition to model training through the first training loss, the electronic device can also calculate a language network predicted audio feature loss to perform model training. Accordingly, the electronic device determines a second training loss based on the first audio features and second audio features corresponding to the sample audio. The second audio features include content features and acoustic features in the sample audio. The second training loss is used to represent the difference between the first audio features and the second audio features. Then, the electronic device updates parameters of the language network and the diffusion network in the first speech synthesis model based on the first training loss and the second training loss, and obtains the second speech synthesis model.

[0156] The second training loss can be a cross-entropy loss. The electronic device can use a cross-entropy loss function to calculate the cross-entropy loss between the first audio features and the second audio features corresponding to the sample audio, and take the cross-entropy loss as the second training loss. By determining the cross-entropy loss between the first audio features and the second audio features as the second training loss of the speech synthesis model, the second training loss can accurately reflect the difference between the audio features predicted by the language network and the real audio features, thereby improving the accuracy and efficiency of training the speech synthesis model through the second training loss.

[0157] The scheme provided by the embodiments of the present disclosure can accurately reflect the loss of the language network predicting the audio, train the speech synthesis model through the first training loss and the second training loss, realize training the speech synthesis model from the two aspects of predicting noise by the diffusion network and predicting audio by the language network, and improve the accuracy of the speech synthesis model, so as to generate audio with higher expressiveness and naturalness and higher quality.

[0158] In some embodiments, in addition to training the model through the first training loss, the electronic device can also train the model by calculating the loss of the audio predicted by the speech synthesis model as a whole. Accordingly, the electronic device removes the predicted noise from the sample audio to which the reference noise is added to obtain target audio. Then, the electronic device determines a third training loss based on the sample audio and the target audio. The third training loss is used to represent the difference between the sample audio and the target audio. Then, the electronic device updates the parameters of the language network and the diffusion network in the first speech synthesis model based on the first training loss and the third training loss to obtain a second speech synthesis model.

[0159] The process of removing noise and calculating the third training loss can be performed in the dimension of the feature. Accordingly, the electronic device removes the predicted noise feature from the third audio feature (the second audio feature to which the reference noise feature is added) to obtain a target audio feature. Then, the electronic device determines the third training loss based on the second audio feature corresponding to the sample audio and the target audio feature. Alternatively, the electronic device can calculate the third training loss in the dimension of the audio after obtaining the target audio. Optionally, the electronic device can extract the mel spectrogram of the sample audio and the mel spectrogram of the target audio, calculate the gap between the mel spectrogram of the sample audio and the mel spectrogram of the target audio, and take the gap as the third training loss. The calculation method of the third training loss is not limited in the embodiments of the present disclosure. The third training loss can be a mean square error loss, a cross-entropy loss, or a mean absolute error loss, and the embodiments of the present disclosure are not limited in this regard.

[0160] The scheme provided by the embodiments of the present disclosure can accurately reflect the loss of the language network predicting the audio, train the speech synthesis model through the first training loss and the second training loss, realize training the speech synthesis model from the two aspects of predicting noise by the diffusion network and predicting audio by the language network, and improve the accuracy of the speech synthesis model, so as to generate audio with higher expressiveness and naturalness and higher quality.

[0161] In some embodiments, the electronic device can further update parameters of the language network and the diffusion network in the first speech synthesis model based on the first training loss, the second training loss, and the third training loss to obtain a second speech synthesis model, which will not be described herein again.

[0162] In some embodiments, during the model training process described above, the electronic device can use at least one optimization method such as gradient clipping, learning rate decay, and regularization to optimize the performance of the first speech synthesis model. For example, the electronic device can adjust the learning rate during the training process through a learning rate scheduler, so that the learning rate corresponding to different iteration rounds is not completely the same. Alternatively, the electronic device can scale or truncate the gradient of back propagation during the model training process using the back propagation algorithm, so that the gradient is kept within a reasonable range, thereby avoiding the occurrence of gradient explosion or gradient disappearance. Alternatively, the electronic device can perform gradient clipping according to the absolute value of the gradient or the norm of the gradient.

[0163] After the model training is completed, the electronic device can further use at least one method such as Mel Cepstral Distortion (MCD) and PESQ (Perceptual evaluation of speech quality) to evaluate the quality of the speech generated by the second speech synthesis model. In the case where the quality of the generated speech does not meet the preset standard, the electronic device can continue to fine-tune the parameters of the language network and the diffusion network in the second speech synthesis model to improve the naturalness and expressiveness of speech synthesis.

[0164] The embodiments of the present disclosure provide a training method of a speech synthesis model, which uses content-consistent sample text and sample audio to jointly train a language network and a diffusion network in the speech synthesis model, so that the speech synthesis model can learn as a whole to convert text into audio containing acoustic features, achieving the purpose of end-to-end training of the speech synthesis model, simplifying the training steps, and being conducive to improving the training efficiency of the model. Furthermore, the present solution uses a diffusion network to learn to reconstruct audio, and trains the speech synthesis model through a first training loss, so that the difference between the reference noise and the predicted noise becomes smaller and smaller, i.e., the diffusion network can accurately learn the noise in the sample audio, ensuring that the diffusion network can reconstruct more accurate audio, i.e., being conducive to improving the accuracy of the speech synthesis model, so as to generate audio with higher expressiveness and naturalness, and higher quality.

[0165] In addition, compared with the solution in the related art that "first learns semantic information of text, and then converts the semantic information into audio containing acoustic features", the present solution also does not need to encode the semantic information of the text, thereby avoiding the dependence of audio reconstruction on the accuracy of semantic encoding, and being conducive to ensuring the reliability and stability of the speech synthesis model.

[0166] The above embodiments mainly introduce the training process of the speech synthesis model. In some embodiments, the electronic device can implement a speech synthesis task through the obtained second speech synthesis model. Figure 5 is a flowchart of a speech synthesis method according to an example embodiment, which is performed by an electronic device, referring to Figure 5 The method comprises the following steps.

[0167] In step 501, the electronic device obtains input text and prompt audio, and the prompt audio is used to indicate acoustic information for speech synthesis.

[0168] In the embodiments of the present disclosure, the input text is text to be synthesized by speech. For example, the input text is navigation text generated by a navigation application, or reply text of a voice assistant to a user, and the like, which is not limited in the embodiments of the present disclosure. The prompt audio is used to provide acoustic features for speech synthesis. For example, the prompt audio can be a segment of speech of a star selected by a user. Before speech synthesis, the electronic device obtains the input text and the prompt audio. The electronic device can preprocess the input text before inputting into the second speech synthesis model. For details, refer to the preprocessing process in step 301, which is not described herein.

[0169] In step 502, the electronic device processes the input text and the prompt audio through a language network in the second speech synthesis model to obtain intermediate audio features corresponding to the input text.

[0170] In the embodiments of the present disclosure, the electronic device can input the input text and the prompt audio into the language network in the second speech synthesis model, process the input text and the prompt audio through the language network, extract the content features of the input text and the acoustic features of the sample audio segment, to obtain the intermediate audio features corresponding to the input text. Alternatively, the electronic device can also extract the content features of the input text and the acoustic features of the prompt audio through other network models other than the language network, and then input the content features of the input text and the acoustic features of the prompt audio into the language network for processing, to obtain the intermediate audio features corresponding to the input text. The embodiments of the present disclosure are not limited in this regard. The intermediate audio features are the audio features of the input text predicted by the language network. For details, refer to the obtaining process of the phoneme sequence and the acoustic features in step 302, which is not described herein.

[0171] In step 503, the electronic device denoises the intermediate audio features through a diffusion network in the second speech synthesis model to obtain target audio features, and generates output audio based on the target audio features.

[0172] In the embodiments of the present disclosure, after the intermediate audio feature is output in the language network, the intermediate audio feature is input into a diffusion network in the second speech synthesis model. The electronic device performs denoising processing on the intermediate audio feature through the diffusion network to obtain a target audio feature. Then, the electronic device converts the target audio feature into output audio. The content of the output audio is consistent with the content of the input text, and the speech synthesis of the input text is realized.

[0173] The embodiments of the present disclosure provide a speech synthesis method, which can synthesize speech for any input text through a second speech synthesis model to obtain output audio consistent with the content of the input text. Since the second speech synthesis model is trained based on the training method of the above speech synthesis model, the output audio consistent with the content of the input text can be accurately obtained through the second speech synthesis model, and the accuracy of speech synthesis is improved.

[0174] In addition, since the input text is preprocessed and phoneme sequence conversion is performed in the process of speech synthesis, the content of the input text can be standardized, thereby facilitating the speech synthesis model to generate more accurate output audio based on the standardized phoneme sequence, reducing the cases of missing words, missing words and pronunciation errors, and improving the accuracy of speech synthesis.

[0175] All the optional technical solutions described above can be combined to form optional embodiments of the present disclosure, which will not be described one by one here.

[0176] Figure 6 is a block diagram of a training device of a speech synthesis model according to an exemplary embodiment. Referring to Figure 6 The first speech synthesis model includes a language network and a diffusion network, the language network is used for predicting audio features of text, and the diffusion network is used for reconstructing audio based on the audio features of the text; the device comprises:

[0177] The processing unit 601 is configured to perform processing on the sample text and the sample audio through the language network to obtain first audio features, the content of the sample text and the sample audio is consistent, and the first audio features include content features of the sample text and acoustic features of the sample audio;

[0178] The prediction unit 602 is configured to perform noise prediction on the sample audio added with reference noise based on the first audio features through the diffusion network to obtain predicted noise;

[0179] The first determination unit 603 is configured to determine a first training loss based on the reference noise and the predicted noise, the first training loss is used to represent the difference between the reference noise and the predicted noise;

[0180] The updating unit 604 is configured to update parameters of the language network and the diffusion network in the first speech synthesis model based on the first training loss to obtain a second speech synthesis model.

[0181] In some embodiments, the processing unit 601 includes:

[0182] The intercepting sub-unit is configured to intercept a sample audio segment from the sample audio based on the content of the sample text, the sample audio segment including sound of all or part of the content of the sample text.

[0183] The processing sub-unit is configured to process the sample text and the sample audio segment through the language network to obtain first audio features.

[0184] In some embodiments, the intercepting sub-unit is configured to obtain secondary content from the sample text in addition to key content, the key content being used to represent a theme of the sample text; and intercept the sample audio segment from the sample audio based on the secondary content, the content of the sample audio segment being the secondary content.

[0185] In some embodiments, the intercepting sub-unit is configured to obtain an emotional keyword from the content of the sample text, the emotional keyword being used to represent an emotion conveyed by the content of the sample text; and intercept the sample audio segment from the sample audio based on the emotional keyword, the sample audio segment including the emotional keyword.

[0186] In some embodiments, the sample audio includes speech of multiple objects.

[0187] The intercepting sub-unit is configured to, for any one of the multiple objects, intercept a sample audio segment corresponding to the object from the sample audio based on the content of the sample text and an acoustic feature of the object.

[0188] In some embodiments, the prediction unit 602 is configured to obtain second audio features corresponding to the sample audio and a reference noise feature, the reference noise feature being used to represent reference noise; add the reference noise feature to the second audio features through the diffusion network to obtain third audio features; and perform noise prediction on the third audio features through the diffusion network conditioned on the first audio features to obtain predicted noise features, the predicted noise features being used to represent predicted noise.

[0189] The first determining unit 603 is configured to determine the first training loss based on the reference noise feature and the predicted noise feature.

[0190] In some embodiments, the apparatus further includes:

[0191] The second determining unit is configured to determine a second training loss based on the first audio feature and a second audio feature corresponding to the sample audio, the second audio feature including a content feature and an acoustic feature in the sample audio, and the second training loss being used to represent a difference degree between the first audio feature and the second audio feature.

[0192] The updating unit 604 is configured to update parameters of the language network and the diffusion network in the first speech synthesis model based on the first training loss and the second training loss to obtain a second speech synthesis model.

[0193] In some embodiments, the apparatus further includes:

[0194] The third determining unit is configured to remove the predicted noise from the sample audio to which the reference noise is added to obtain target audio, and determine a third training loss based on the sample audio and the target audio, the third training loss being used to represent a difference degree between the sample audio and the target audio.

[0195] The updating unit 604 is configured to update parameters of the language network and the diffusion network in the first speech synthesis model based on the first training loss and the third training loss to obtain a second speech synthesis model.

[0196] In some embodiments, the processing unit 601 includes:

[0197] The obtaining sub-unit is configured to obtain a phoneme sequence of the sample text and an acoustic feature of the sample audio.

[0198] The processing sub-unit is configured to process the phoneme sequence and the acoustic feature through the language network to obtain the first audio feature.

[0199] In some embodiments, the obtaining sub-unit is configured to perform preprocessing on the sample text, the preprocessing including at least one of word segmentation, noise removal, text normalization, and multi-pronunciation word pronunciation annotation, and convert the preprocessed sample text into the phoneme sequence based on a corresponding relationship between characters and phonemes.

[0200] The training device of the speech synthesis model provided in the embodiments of the present disclosure adopts the content-consistent sample text and sample audio to jointly train the language network and diffusion network in the speech synthesis model, so that the speech synthesis model can learn as a whole to convert the text into audio containing acoustic features, the purpose of end-to-end training of the speech synthesis model is achieved, the training steps are simplified, and the training efficiency of the model is improved; and the diffusion network is used to learn to reconstruct the audio, the speech synthesis model is trained through the first training loss, so that the difference between the reference noise and the predicted noise becomes smaller and smaller, that is, the diffusion network can accurately learn the noise in the sample audio, so that the diffusion network can reconstruct more accurate audio, which is beneficial to improve the accuracy of the speech synthesis model, so as to generate audio with higher expressiveness and naturalness and higher quality.

[0201] In addition, compared with the scheme of "first learning semantic information of the text, and then converting the semantic information into audio containing acoustic features" in the related art, the present scheme also does not need to encode the semantic information of the text, thereby avoiding the dependence of audio reconstruction on the accuracy of semantic encoding, and being beneficial to guarantee the reliability and stability of the speech synthesis model.

[0202] It should be noted that the training device of the speech synthesis model provided in the above embodiments is only exemplified by the division of the above functional units when training the speech synthesis model, and in actual application, the above functions can be completed by different functional units according to needs, that is, the internal structure of the electronic device is divided into different functional units to complete all or part of the functions described above. In addition, the training device of the speech synthesis model provided in the above embodiments and the training method of the speech synthesis model embodiment belong to the same concept, and the specific implementation process is described in detail in the method embodiment, which will not be described here.

[0203] As to the device in the above embodiments, the specific manner in which each module performs operations has been described in detail in the embodiments relating to the method, and will not be described in detail here.

[0204] Figure 7 is a block diagram of a speech synthesis device according to an example embodiment. Referring to Figure 7 , the device includes:

[0205] The acquisition unit 701 is configured to perform acquisition of input text and prompt audio, the prompt audio being used to indicate acoustic information for speech synthesis;

[0206] The first processing unit 702 is configured to perform processing of the input text and the prompt audio through the language network in the second speech synthesis model to obtain intermediate audio features corresponding to the input text, the second speech synthesis model being trained based on the training method of the speech synthesis model described above;

[0207] The second processing unit 703 is configured to perform denoising processing on the intermediate audio feature through a diffusion network in the second speech synthesis model to obtain a target audio feature, and generate output audio based on the target audio feature.

[0208] The speech synthesis apparatus provided in the embodiments of the present disclosure can synthesize speech for any input text through the second speech synthesis model to obtain output audio consistent with the content of the input text. Since the second speech synthesis model is trained based on the training method of the speech synthesis model, the output audio consistent with the content of the input text can be accurately obtained through the second speech synthesis model, and the accuracy of speech synthesis is improved.

[0209] It should be noted that the speech synthesis apparatus provided in the above embodiments is only used as an example to illustrate the division of the above functional units when generating audio based on text. In actual applications, the above functions can be completed by different functional units according to needs, that is, the internal structure of the electronic device is divided into different functional units to complete all or part of the functions described above. In addition, the speech synthesis apparatus and the speech synthesis method provided in the above embodiments belong to the same concept, and the specific implementation process is described in detail in the method embodiments, which will not be described here.

[0210] As for the apparatus in the above embodiments, the specific manner in which each module performs operations has been described in detail in the embodiments related to the method, and will not be described in detail here.

[0211] When the electronic device is provided as a terminal, Figure 8 is a block diagram of a terminal 800 according to an example embodiment. The terminal Figure 8 A structural block diagram of a terminal 800 provided by an example embodiment of the present disclosure is shown. The terminal 800 can be a smartphone, a tablet computer, an MP3 (Moving Picture Experts Group Audio Layer III) player, an MP4 (Moving Picture Experts Group Audio Layer IV) player, a notebook computer, or a desktop computer. The terminal 800 can also be referred to as a user equipment, a portable terminal, a laptop terminal, a desktop terminal, or other names.

[0212] Generally, the terminal 800 includes a processor 801 and a memory 802.

[0213] The processor 801 can include one or more processing cores, such as a 4-core processor, an 8-core processor, and the like. The processor 801 can be implemented in at least one of a hardware form of a DSP (Digital Signal Processing), an FPGA (Field-Programmable Gate Array), a PLA (Programmable Logic Array). The processor 801 can also include a main processor and a coprocessor, the main processor being a processor for processing data in an awake state, also known as a CPU (Central Processing Unit), and the coprocessor being a low-power processor for processing data in a standby state. In some embodiments, the processor 801 can be integrated with a GPU (Graphics Processing Unit) for rendering and drawing content required to be displayed by the display screen. In some embodiments, the processor 801 can further include an AI (Artificial Intelligence) processor for processing machine learning related computing operations.

[0214] The memory 802 can include one or more computer-readable storage media that can be non-transitory. The memory 802 can also include high-speed random access memory and nonvolatile, computer-readable storage media such as one or more magnetic disk storage devices, flash memory devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 802 is used to store at least one computer program for being executed by the processor 801 to implement the training method of the speech synthesis model or the speech synthesis method provided by the method embodiments in the present application.

[0215] In some embodiments, the terminal 800 can also optionally include a peripheral device interface 803 and at least one peripheral device. The processor 801, the memory 802, and the peripheral device interface 803 can be connected through a bus or a signal line. Each peripheral device can be connected to the peripheral device interface 803 through a bus, a signal line, or a circuit board. Specifically, the peripheral device includes at least one of a radio frequency circuit 804, a display screen 805, a camera assembly 806, an audio circuit 807, and a power supply 808.

[0216] The peripheral interface 803 can be used to connect at least one I / O (Input / Output) related peripheral device to the processor 801 and the memory 802. In some embodiments, the processor 801, the memory 802 and the peripheral interface 803 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 801, the memory 802 and the peripheral interface 803 can be implemented on a separate chip or circuit board, and the present embodiments are not limited in this regard.

[0217] The radio frequency circuit 804 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 804 communicates with a communication network and other communication devices through electromagnetic signals. The radio frequency circuit 804 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals into electrical signals. In some embodiments, the radio frequency circuit 804 includes an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a subscriber identity module card, and the like. The radio frequency circuit 804 can communicate with other terminals through at least one wireless communication protocol. The wireless communication protocol includes but is not limited to: the World Wide Web, a metropolitan area network, an intranet, various generations of mobile communication networks (2G, 3G, 4G and 5G), a wireless local area network and / or a WiFi (Wireless Fidelity) network. In some embodiments, the radio frequency circuit 804 can also include NFC (Near Field Communication) related circuitry, and the present application is not limited in this regard.

[0218] The display screen 805 is configured to display a UI (User Interface). The UI can include graphics, text, icons, video, and any combination thereof. When the display screen 805 is a touch display screen, the display screen 805 is further configured to capture touch signals on or above the surface of the display screen 805. The touch signals can be input to the processor 801 as control signals for processing. In this case, the display screen 805 can also be configured to provide virtual buttons and / or virtual keyboard, also known as soft buttons and / or soft keyboard. In some embodiments, the display screen 805 can be one, disposed on the front panel of the terminal 800; in other embodiments, the display screen 805 can be at least two, respectively disposed on different surfaces of the terminal 800 or in a folding design; in other embodiments, the display screen 805 can be a flexible display screen, disposed on a curved surface or a folding surface of the terminal 800. Even, the display screen 805 can also be disposed in an irregular shape other than a rectangle, i.e., a special-shaped screen. The display screen 805 can be made of LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode), etc.

[0219] The camera assembly 806 is configured to capture images or videos. In some embodiments, the camera assembly 806 includes a front camera and a rear camera. Generally, the front camera is disposed on the front panel of the terminal, and the rear camera is disposed on the back of the terminal. In some embodiments, the rear camera is at least two, which are any one of a main camera, a depth-of-field camera, a wide-angle camera, and a telephoto camera, to realize the background blur function by fusing the main camera and the depth-of-field camera, the panoramic shooting and VR (Virtual Reality) shooting function by fusing the main camera and the wide-angle camera, or other fusion shooting functions. In some embodiments, the camera assembly 806 can further include a flash. The flash can be a single-color-temperature flash or a dual-color-temperature flash. The dual-color-temperature flash refers to the combination of a warm light flash and a cold light flash, which can be used for light compensation under different color temperatures.

[0220] The audio circuit 807 can include a microphone and a speaker. The microphone is used to collect sound waves of a user and an environment, and convert the sound waves into an electrical signal input to the processor 801 for processing, or input to the radio frequency circuit 804 to realize voice communication. For the purpose of stereo sound collection or noise reduction, the microphone can be multiple, respectively arranged at different parts of the terminal 800. The microphone can also be an array microphone or an omnidirectional collection type microphone. The speaker is used to convert an electrical signal from the processor 801 or the radio frequency circuit 804 into sound waves. The speaker can be a conventional diaphragm speaker, or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, not only can it convert electrical signals into sound waves that humans can hear, but it can also convert electrical signals into sound waves that humans cannot hear for ranging purposes. In some embodiments, the audio circuit 807 can also include a headphone jack.

[0221] The power supply 808 is used to supply power to each component in the terminal 800. The power supply 808 can be alternating current, direct current, disposable battery or rechargeable battery. When the power supply 808 includes a rechargeable battery, the rechargeable battery can be a wired charging battery or a wireless charging battery. The wired charging battery is a battery charged through a wired line, and the wireless charging battery is a battery charged through a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0222] Those skilled in the art can understand that the structure shown in the above embodiments is not a limitation on the terminal 800, and the terminal 800 can include more or fewer components than shown in the figure, or combine certain components, or use different component arrangements. Figure 8 Those skilled in the art can understand that the structure shown in the above embodiments is not a limitation on the terminal 800, and the terminal 800 can include more or fewer components than shown in the figure, or combine certain components, or use different component arrangements.

[0223] When the electronic device is provided as a server, Figure 9 Fig. 9 is a block diagram of a server 900 according to an example embodiment. The server 900 can have a large difference in configuration or performance, and can include one or more processors (Central Processing Units, CPUs) 901 and one or more memories 902, wherein the memory 902 stores at least one program code, which is loaded and executed by the processor 901 to implement the training method of the speech synthesis model or the speech synthesis method provided by each of the above methods. Of course, the server can also have a wired or wireless network interface, a keyboard, and an input and output interface, etc. to perform input and output, and the server 900 can also include other components for realizing the functions of the device, which are not described here.

[0224] In an example embodiment, a computer readable storage medium including instructions, such as the memory 802 or the memory 902 including instructions, is also provided, which can be executed by the processor 801 of the terminal 800 or the processor 901 of the server 900 to complete the above voice synthesis model training method or voice synthesis method. Optionally, the computer readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, and an optical data storage device, etc.

[0225] A computer program product including computer programs / instructions, which, when executed by a processor, implement the above voice synthesis model training method or voice synthesis method.

[0226] Other embodiments of the present disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the features disclosed herein. The present disclosure is intended to cover any variations, uses, or adaptations of the present disclosure following the general principles thereof and including such departures from the present disclosure that come within known, accepted, and / or customary practice in the art to which the present disclosure pertains. The specification and examples are to be regarded as illustrative only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0227] It should be understood that the present disclosure is not limited to the precise structures herein described and illustrated in the drawings, and that various modifications and changes can be made without departing from the scope thereof. The scope of the present disclosure is indicated by the appended claims, only.

Claims

1. A training method for a speech synthesis model, characterized in that, The first speech synthesis model includes a language network and a diffusion network, wherein the language network is used to predict the audio features of the text, and the diffusion network is used to reconstruct the audio based on the audio features of the text; the method includes: The sample text and sample audio are processed by the language network to obtain a first audio feature, wherein the sample text and the sample audio have the same content, and the first audio feature includes the content features of the sample text and the acoustic features of the sample audio. Based on the first audio feature, noise prediction is performed on the sample audio with added reference noise through the diffusion network to obtain the predicted noise. Based on the reference noise and the predicted noise, a first training loss is determined, which represents the degree of difference between the reference noise and the predicted noise. Based on the first training loss, the parameters of the language network and the diffusion network in the first speech synthesis model are updated to obtain the second speech synthesis model.

2. The training method for the speech synthesis model according to claim 1, characterized in that, The process of processing sample text and sample audio through the language network to obtain the first audio feature includes: Based on the content of the sample text, a sample audio segment is extracted from the sample audio, and the sample audio segment includes the sound of all or part of the content of the sample text; The first audio feature is obtained by processing the sample text and the sample audio segment through the language network.

3. The training method for the speech synthesis model according to claim 2, characterized in that, The step of extracting sample audio segments from the sample audio based on the content of the sample text includes: Secondary content, excluding key content, is extracted from the sample text, where the key content is used to represent the theme of the sample text; Based on the secondary content, the sample audio segment is extracted from the sample audio, and the content of the sample audio segment is the secondary content.

4. The training method for the speech synthesis model according to claim 2, characterized in that, The step of extracting sample audio segments from the sample audio based on the content of the sample text includes: Emotional keywords are extracted from the content of the sample text, and the emotional keywords are used to represent the emotions conveyed by the content of the sample text. Based on the emotional keywords, a sample audio segment is extracted from the sample audio, and the sample audio segment includes the emotional keywords.

5. The training method for the speech synthesis model according to claim 2, characterized in that, The sample audio includes the speech of multiple objects; The step of extracting sample audio segments from the sample audio based on the content of the sample text includes: For any one of the plurality of objects, based on the content of the sample text and the acoustic features of the object, a sample audio segment corresponding to the object is extracted from the sample audio.

6. The training method for the speech synthesis model according to claim 1, characterized in that, The step of performing noise prediction on the sample audio with added reference noise based on the first audio feature through the diffusion network to obtain predicted noise includes: Obtain the second audio feature and reference noise feature corresponding to the sample audio, wherein the reference noise feature is used to represent the reference noise; The reference noise feature is added to the second audio feature through the diffusion network to obtain the third audio feature; Through the diffusion network, noise prediction is performed on the third audio feature using the first audio feature as a condition to obtain a predicted noise feature, which is used to represent the predicted noise. Determining the first training loss based on the reference noise and the predicted noise includes: The first training loss is determined based on the reference noise features and the predicted noise features.

7. The training method for the speech synthesis model according to claim 1, characterized in that, The method further includes: Based on the first audio feature and the second audio feature corresponding to the sample audio, a second training loss is determined. The second audio feature includes content features and acoustic features in the sample audio. The second training loss is used to represent the degree of difference between the first audio feature and the second audio feature. The step of updating the parameters of the language network and the diffusion network in the first speech synthesis model based on the first training loss includes: Based on the first training loss and the second training loss, the parameters of the language network and the diffusion network in the first speech synthesis model are updated to obtain the second speech synthesis model.

8. The training method for the speech synthesis model according to claim 1, characterized in that, The method further includes: The predicted noise is removed from the sample audio with the added reference noise to obtain the target audio; Based on the sample audio and the target audio, a third training loss is determined, which is used to represent the degree of difference between the sample audio and the target audio; The step of updating the parameters of the language network and the diffusion network in the first speech synthesis model based on the first training loss to obtain the second speech synthesis model includes: Based on the first training loss and the third training loss, the parameters of the language network and the diffusion network in the first speech synthesis model are updated to obtain the second speech synthesis model.

9. The training method for the speech synthesis model according to claim 1, characterized in that, The process of processing sample text and sample audio through the language network to obtain the first audio feature includes: Obtain the phoneme sequence of the sample text and the acoustic features of the sample audio; The first audio feature is obtained by processing the phoneme sequence and the acoustic features through the language network.

10. The training method for the speech synthesis model according to claim 9, characterized in that, The process of obtaining the phoneme sequence of the sample text includes: The sample text is preprocessed, and the preprocessing includes at least one of word segmentation, noise reduction, text normalization, and pronunciation annotation of polyphonic characters. Based on the correspondence between characters and phonemes, the preprocessed sample text is converted into the phoneme sequence.

11. A speech synthesis method, characterized in that, The method includes: Acquire input text and prompt audio, wherein the prompt audio is used to indicate the acoustic information used in speech synthesis; The input text and the prompt audio are processed through the language network in the second speech synthesis model to obtain the intermediate audio features corresponding to the input text. The second speech synthesis model is trained based on the training method of the speech synthesis model according to any one of claims 1-10. The intermediate audio features are denoised using the diffusion network in the second speech synthesis model to obtain target audio features, and output audio is generated based on the target audio features.

12. A training device for a speech synthesis model, characterized in that, The first speech synthesis model includes a language network and a diffusion network, wherein the language network is used to predict the audio features of the text, and the diffusion network is used to reconstruct audio based on the audio features of the text; the device includes: The processing unit is configured to perform processing on sample text and sample audio through the language network to obtain a first audio feature, wherein the sample text and the sample audio have the same content, and the first audio feature includes the content features of the sample text and the acoustic features of the sample audio. The prediction unit is configured to perform noise prediction on the sample audio with added reference noise based on the first audio feature through the diffusion network to obtain predicted noise; The first determining unit is configured to perform the task of determining a first training loss based on the reference noise and the predicted noise, wherein the first training loss is used to represent the degree of difference between the reference noise and the predicted noise; The update unit is configured to update the parameters of the language network and the diffusion network in the first speech synthesis model based on the first training loss, so as to obtain a second speech synthesis model.

13. A speech synthesis device, characterized in that, The device includes: The acquisition unit is configured to acquire input text and prompt audio, the prompt audio being used to indicate acoustic information used in speech synthesis; The first processing unit is configured to process the input text and the prompt audio through the language network in the second speech synthesis model to obtain the intermediate audio features corresponding to the input text. The second speech synthesis model is trained based on the training method of the speech synthesis model according to any one of claims 1-10. The second processing unit is configured to perform denoising processing on the intermediate audio features through the diffusion network in the second speech synthesis model to obtain target audio features, and generate output audio based on the target audio features.

14. An electronic device, characterized in that, The electronic device includes: One or more processors; Memory used to store the executable program code of the processor; The processor is configured to execute the program code to implement the training method of the speech synthesis model as described in any one of claims 1 to 10 or the speech synthesis method as described in claim 11.

15. A computer-readable storage medium, characterized in that, When the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device is able to perform the training method of the speech synthesis model as described in any one of claims 1 to 10 or the speech synthesis method as described in claim 11.

16. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the training method of the speech synthesis model according to any one of claims 1 to 10 or the speech synthesis method according to claim 11.

Images