Speech synthesis model training method, speech synthesis method, device and equipment

Abstract

The invention provides a speech synthesis model training method, a speech synthesis method, a speech synthesis device and speech synthesis equipment, and belongs to the technical field of computers. The first speech synthesis model comprises a language network and a diffusion network; the method comprises: processing a sample text and a sample audio through a language network to obtain a first audio feature, the sample text and the sample audio being consistent in content, the first audio feature comprising a content feature of the sample text and an acoustic feature of the sample audio; performing noise prediction on the sample audio added with the reference noise based on the first audio feature through a diffusion network to obtain predicted noise; determining a first training loss based on the reference noise and the predicted noise; and updating parameters of a language network and a diffusion network in the first speech synthesis model based on the first training loss to obtain a second speech synthesis model. According to the technical scheme, the purpose of end-to-end training of the speech synthesis model is achieved, the training steps are simplified, and the training efficiency of the model is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a training method for a speech synthesis model, a speech synthesis method, a device and equipment. Background Art

[0002] With the rapid development of computer technology, speech synthesis technology has been widely used in people's daily life. For example, voice assistants, navigation systems, smart home controls, reading software, etc. It provides users with a more natural and convenient way of interaction by converting text information into audible speech.

[0003] At present, speech synthesis technology is mainly implemented based on network models. Network models usually divide the text-to-speech process into two stages: text-to-semantics and semantics-to-acoustics. First, the encoder in the network model is trained in a supervised manner to learn the semantic information of the text; then the language model in the network model is trained in an unsupervised manner to learn to convert the semantic information into acoustic features; finally, speech is generated based on the acoustic features.

[0004] However, in the above technical solution, text-to-semantics and semantics-to-acoustics need to be trained separately, which affects the training efficiency of the model. Summary of the invention

[0005] The present disclosure provides a training method, a speech synthesis method, a device and an apparatus for a speech synthesis model, which achieves the purpose of end-to-end training of the speech synthesis model, simplifies the training steps and helps improve the training efficiency of the model. The technical solution of the present disclosure is as follows:

[0006] According to one aspect of an embodiment of the present disclosure, a method for training a speech synthesis model is provided, wherein a first speech synthesis model includes a language network and a diffusion network, wherein the language network is used to predict audio features of a text, and the diffusion network is used to reconstruct audio based on the audio features of the text; the method includes:

[0007] Processing the sample text and the sample audio through the language network to obtain a first audio feature, wherein the sample text and the sample audio have the same content, and the first audio feature includes a content feature of the sample text and an acoustic feature of the sample audio;

[0008] By using the diffusion network, based on the first audio feature, noise prediction is performed on the sample audio to which reference noise is added to obtain predicted noise;

[0009] Based on the reference noise and the predicted noise, determining a first training loss, wherein the first training loss is used to represent a degree of difference between the reference noise and the predicted noise;

[0010] Based on the first training loss, parameters of the language network and the diffusion network in the first speech synthesis model are updated to obtain a second speech synthesis model.

[0011] According to another aspect of the present disclosure, a speech synthesis method is provided, the method comprising:

[0012] Obtaining input text and prompt audio, where the prompt audio is used to indicate acoustic information used for speech synthesis;

[0013] The input text and the prompt audio are processed by a language network in a second speech synthesis model to obtain intermediate audio features corresponding to the input text, wherein the second speech synthesis model is trained based on the training method of the speech synthesis model described above;

[0014] The intermediate audio features are denoised by the diffusion network in the second speech synthesis model to obtain target audio features, and output audio is generated based on the target audio features.

[0015] According to another aspect of an embodiment of the present disclosure, a training device for a speech synthesis model is provided, wherein a first speech synthesis model includes a language network and a diffusion network, wherein the language network is used to predict audio features of a text, and the diffusion network is used to reconstruct audio based on the audio features of the text; the device includes:

[0016] A processing unit is configured to process the sample text and the sample audio through the language network to obtain a first audio feature, wherein the sample text and the sample audio have the same content, and the first audio feature includes a content feature of the sample text and an acoustic feature of the sample audio;

[0017] A prediction unit is configured to perform noise prediction on the sample audio to which reference noise is added based on the first audio feature through the diffusion network to obtain predicted noise;

[0018] A first determining unit is configured to determine a first training loss based on the reference noise and the predicted noise, where the first training loss is used to represent a degree of difference between the reference noise and the predicted noise;

[0019] An updating unit is configured to update parameters of the language network and the diffusion network in the first speech synthesis model based on the first training loss to obtain a second speech synthesis model.

[0020] In some embodiments, the processing unit includes:

[0021] An interception subunit is configured to intercept a sample audio segment from the sample audio based on the content of the sample text, wherein the sample audio segment includes the sound of all or part of the content in the sample text;

[0022] The processing subunit is configured to process the sample text and the sample audio segment through the language network to obtain the first audio feature.

[0023] In some embodiments, the interception sub-unit is configured to obtain secondary content other than key content from the sample text, wherein the key content is used to represent the theme of the sample text; based on the secondary content, the sample audio segment is intercepted from the sample audio, and the content of the sample audio segment is the secondary content.

[0024] In some embodiments, the interception sub-unit is configured to obtain emotion keywords from the content of the sample text, and the emotion keywords are used to represent the emotions conveyed by the content of the sample text; based on the emotion keywords, intercept the sample audio segment from the sample audio, and the sample audio segment includes the emotion keywords.

[0025] In some embodiments, the sample audio includes speech of multiple subjects;

[0026] The interception subunit is configured to execute, for any object among the multiple objects, intercepting a sample audio segment corresponding to the object from the sample audio based on the content of the sample text and the acoustic features of the object.

[0027] In some embodiments, the prediction unit is configured to execute the acquisition of a second audio feature and a reference noise feature corresponding to the sample audio, wherein the reference noise feature is used to represent the reference noise; adding the reference noise feature to the second audio feature through the diffusion network to obtain a third audio feature; and performing noise prediction on the third audio feature through the diffusion network with the first audio feature as a condition to obtain a predicted noise feature, wherein the predicted noise feature is used to represent the predicted noise;

[0028] The first determining unit is configured to determine the first training loss based on the reference noise feature and the predicted noise feature.

[0029] In some embodiments, the apparatus further comprises:

[0030] A second determining unit is configured to determine a second training loss based on the first audio feature and a second audio feature corresponding to the sample audio, wherein the second audio feature includes a content feature and an acoustic feature in the sample audio, and the second training loss is used to represent a degree of difference between the first audio feature and the second audio feature;

[0031] The updating unit is configured to update parameters of the language network and the diffusion network in the first speech synthesis model based on the first training loss and the second training loss to obtain the second speech synthesis model.

[0032] In some embodiments, the apparatus further comprises:

[0033] A third determining unit is configured to remove the predicted noise from the sample audio to which the reference noise is added to obtain a target audio; based on the sample audio and the target audio, determine a third training loss, wherein the third training loss is used to represent a degree of difference between the sample audio and the target audio;

[0034] The updating unit is configured to update parameters of the language network and the diffusion network in the first speech synthesis model based on the first training loss and the third training loss to obtain the second speech synthesis model.

[0035] In some embodiments, the processing unit includes:

[0036] An acquisition subunit, configured to acquire a phoneme sequence of the sample text and an acoustic feature of the sample audio;

[0037] The processing subunit is configured to process the phoneme sequence and the acoustic feature through the language network to obtain the first audio feature.

[0038] In some embodiments, the acquisition subunit is configured to perform preprocessing on the sample text, wherein the preprocessing includes at least one of word segmentation, denoising, text normalization, and pronunciation annotation of polyphones; and based on the correspondence between characters and phonemes, convert the preprocessed sample text into the phoneme sequence.

[0039] According to another aspect of an embodiment of the present disclosure, a speech synthesis device is provided, the device comprising:

[0040] An acquisition unit is configured to acquire input text and prompt audio, where the prompt audio is used to indicate acoustic information used for speech synthesis;

[0041] A first processing unit is configured to process the input text and the prompt audio through a language network in a second speech synthesis model to obtain intermediate audio features corresponding to the input text, wherein the second speech synthesis model is trained based on the training method of the speech synthesis model;

[0042] The second processing unit is configured to perform denoising processing on the intermediate audio features through a diffusion network in the second speech synthesis model to obtain target audio features, and generate output audio based on the target audio features.

[0043] According to another aspect of an embodiment of the present disclosure, an electronic device is provided, the electronic device comprising:

[0044] one or more processors;

[0045] A memory for storing program codes executable by the processor;

[0046] Among them, the processor is configured to execute the program code to implement the above-mentioned speech synthesis model training method or speech synthesis method.

[0047] According to another aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided. When the program code in the computer-readable storage medium is executed by a processor of an electronic device, the electronic device can execute the training method of the speech synthesis model or the speech synthesis method of the above-mentioned method.

[0048] According to another aspect of an embodiment of the present disclosure, a computer program product is provided, including a computer program / instruction, which implements the above-mentioned speech synthesis model training method or speech synthesis method when executed by a processor.

[0049] The disclosed embodiment provides a method for training a speech synthesis model, which uses sample text and sample audio with consistent content to jointly train the language network and diffusion network in the speech synthesis model, so that the speech synthesis model can learn to convert text into audio containing acoustic features as a whole, thereby achieving the purpose of end-to-end training of the speech synthesis model, simplifying the training steps, and helping to improve the training efficiency of the model; and, this solution uses a diffusion network to learn to reconstruct audio, and trains the speech synthesis model through a first training loss, so that the difference between reference noise and predicted noise becomes smaller and smaller, that is, the diffusion network can accurately learn the noise in the sample audio, ensuring that the diffusion network can reconstruct more accurate audio, that is, helping to improve the accuracy of the speech synthesis model, so as to generate higher-quality audio with higher expressiveness and naturalness.

[0050] In addition, compared with the solution in the related technology of "first learning the semantic information of the text, and then converting the semantic information into audio containing acoustic features", this solution does not require semantic encoding of the text, thereby avoiding the dependence of audio reconstruction on the accuracy of semantic encoding, which is conducive to ensuring the reliability and stability of the speech synthesis model.

[0051] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0053] Figure 1 It is a schematic diagram of an implementation environment of a method for training a speech synthesis model according to an exemplary embodiment.

[0054] Figure 2 The present invention is a flowchart of a method for training a speech synthesis model according to an exemplary embodiment.

[0055] Figure 3 The present invention is a flowchart of another method for training a speech synthesis model according to an exemplary embodiment.

[0056] Figure 4 It is a schematic diagram of a training process of a first speech synthesis model according to an exemplary embodiment.

[0057] Figure 5 The figure is a flow chart of a speech synthesis method according to an exemplary embodiment.

[0058] Figure 6 It is a block diagram of a training device for a speech synthesis model according to an exemplary embodiment.

[0059] Figure 7 The invention is a block diagram of a speech synthesis device according to an exemplary embodiment.

[0060] Figure 8 It is a block diagram of a terminal according to an exemplary embodiment.

[0061] Fig. 9 It is a block diagram of a server according to an exemplary embodiment. DETAILED DESCRIPTION

[0062] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0063] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0064] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in this disclosure are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions. For example, the sample text, sample audio, input text and prompt audio involved in this disclosure are all obtained with full authorization.

[0065] Figure 1 FIG. 1 is a schematic diagram of an implementation environment of a method for training a speech synthesis model according to an exemplary embodiment. Figure 1 The implementation environment specifically includes: a terminal 101 and a server 102. The terminal 101 can be connected to the server 102 via a wireless network or a wired network.

[0066] Terminal 101 is at least one of a smart phone, a smart watch, a desktop computer, a laptop, an MP3 player, an MP4 player, and a laptop computer. An application that supports speech synthesis is installed and run on terminal 101. The application may be a voice assistant, a social application, a navigation application, or a video application, etc., which is not limited in the embodiments of the present disclosure. The user can log in to the application through terminal 101 to obtain the services provided by the application. Taking the application as a navigation application as an example, the user can interact with the navigation application through terminal 101. Accordingly, the navigation application can understand the address entered by the user, generate the route text corresponding to the address, and convert the route text into navigation voice through speech synthesis technology to play the navigation voice to the user. The application is connected to server 102, and server 102 provides background services.

[0067] The terminal 101 generally refers to one of a plurality of terminals, and the present embodiment is described by taking the terminal 101 as an example. Those skilled in the art may know that the number of the above terminals may be more or less. For example, the above terminals may be several, or the above terminals may be dozens or hundreds, or a greater number. The present embodiment does not limit the number and device type of the terminals.

[0068] In some embodiments, the server 102 is at least one of a single server, multiple servers, a cloud computing platform, and a virtualization center. Optionally, the number of the above servers may be more or less, which is not limited in the embodiments of the present disclosure. Of course, the server 102 may also include other functional servers to provide more comprehensive and diversified services. In some embodiments, the server 102 undertakes the main computing work, and the terminal 101 undertakes the secondary computing work; or, the server 102 undertakes the secondary computing work, and the terminal 101 undertakes the main computing work; or, the server 102 and the terminal 101 use a distributed computing architecture for collaborative computing. The server 102 can be connected to the terminal 101 and other terminals via a wireless network or a wired network. Optionally, the number of the above servers may be more or less, which is not limited in the embodiments of the present disclosure.

[0069] Figure 2 is a flowchart of a method for training a speech synthesis model according to an exemplary embodiment. Figure 2 As shown, the training method of the speech synthesis model is applied to an electronic device, comprising the following steps:

[0070] In step 201, the electronic device processes the sample text and the sample audio through the language network in the first speech synthesis model to obtain a first audio feature, the sample text and the sample audio have consistent content, and the first audio feature includes the content feature of the sample text and the acoustic feature of the sample audio.

[0071] In the embodiment of the present disclosure, the first speech synthesis model is a speech synthesis model to be trained. The first speech synthesis model includes a language network and a diffusion network. The language network is used to predict the audio features of the text. The diffusion network is used to reconstruct the audio based on the audio features of the text.

[0072] The electronic device inputs the sample text and the sample audio into the language network in the first speech synthesis model, processes the sample text and the sample audio through the language network, extracts the content features of the sample text and the acoustic features of the sample audio, and obtains the first audio features corresponding to the sample text. The first audio features are the audio features of the sample text predicted by the language network.

[0073] Among them, the sample text and the sample audio content are consistent. That is, the language network can use the sample audio as a prompt to learn the acoustic features to be possessed in the audio features to be predicted, and then combine the content features of the sample text with the acoustic features to obtain the first audio features. Acoustic features are used to indicate the characteristics of acoustic performance such as timbre, sound intensity, pitch, and sound length in the sample audio. The language network can be any autoregressive language model (Auto Regressive Language Model, ARLM), for example, GPT (Generative Pre-Trained Transformer, generative pre-trained Transformer model), etc. The embodiment of the present disclosure does not limit the architecture of the language network.

[0074] In step 202, the electronic device performs noise prediction on the sample audio to which the reference noise is added based on the first audio feature through the diffusion network in the first speech synthesis model to obtain predicted noise.

[0075] In the disclosed embodiment, after the language network outputs the first audio feature, the electronic device inputs the first audio feature into the diffusion network in the first speech synthesis model, and through the diffusion network, the sample audio with the reference noise added is subjected to noise prediction based on the first audio feature to obtain the predicted noise. Then, the electronic device removes the predicted noise from the sample audio with the reference noise added through the prediction network to obtain the predicted audio. The predicted audio is the audio reconstructed by the diffusion network based on the audio feature of the text.

[0076] Among them, the predicted audio is the audio finally synthesized by the first speech synthesis model based on the sample text. Since the predicted audio is obtained by removing the predicted noise from the sample audio with reference noise added, the sample audio and the reference noise are known, and the accuracy of the predicted audio depends on the accuracy of the predicted noise. Accordingly, the performance of the diffusion network in determining the predicted noise can reflect the performance of the predicted audio output by the first speech synthesis model (i.e., the overall performance of the model). Therefore, the electronic device can determine the loss of the first speech synthesis model based on the error of the diffusion network predicted noise, thereby performing model training. Please refer to the subsequent steps for details.

[0077] In step 203, the electronic device determines a first training loss based on the reference noise and the predicted noise, where the first training loss is used to represent the degree of difference between the reference noise and the predicted noise.

[0078] In an embodiment of the present disclosure, the electronic device can determine the first training loss of the first speech synthesis model according to the difference between the reference noise and the predicted noise. The larger the first training loss, the greater the difference between the reference noise and the predicted noise; the smaller the first training loss, the smaller the difference between the reference noise and the predicted noise.

[0079] Among them, in calculating the first training loss, the electronic device can use the mean square error calculation method, the mean absolute error calculation method, the cross entropy loss calculation method, etc. to perform calculation. The embodiment of the present disclosure does not limit the calculation method of the first training loss.

[0080] In step 204, the electronic device updates parameters of the language network and the diffusion network in the first speech synthesis model based on the first training loss to obtain a second speech synthesis model.

[0081] In the disclosed embodiment, the electronic device updates the parameters of the language network and the diffusion network in the first speech synthesis model with the goal of minimizing the first training loss until the second speech synthesis model obtained after the parameter update meets the training stop condition. The training stop condition may be that the first training loss is lower than the first loss threshold, or that the number of parameter updates reaches a preset number, etc., which is not limited in the disclosed embodiment.

[0082] The disclosed embodiment provides a method for training a speech synthesis model, which uses sample text and sample audio with consistent content to jointly train the language network and diffusion network in the speech synthesis model, so that the speech synthesis model can learn to convert text into audio containing acoustic features as a whole, thereby achieving the purpose of end-to-end training of the speech synthesis model, simplifying the training steps, and helping to improve the training efficiency of the model; and, this solution uses a diffusion network to learn to reconstruct audio, and trains the speech synthesis model through a first training loss, so that the difference between reference noise and predicted noise becomes smaller and smaller, that is, the diffusion network can accurately learn the noise in the sample audio, ensuring that the diffusion network can reconstruct more accurate audio, that is, helping to improve the accuracy of the speech synthesis model, so as to generate higher-quality audio with higher expressiveness and naturalness.

[0083] In addition, compared with the solution in the related technology of "first learning the semantic information of the text, and then converting the semantic information into audio containing acoustic features", this solution does not require semantic encoding of the text, thereby avoiding the dependence of audio reconstruction on the accuracy of semantic encoding, which is conducive to ensuring the reliability and stability of the speech synthesis model.

[0084] In some embodiments, the sample text and the sample audio are processed by a language network to obtain a first audio feature, including:

[0085] Based on the content of the sample text, a sample audio segment is intercepted from the sample audio, where the sample audio segment includes the sound of all or part of the content in the sample text;

[0086] The sample text and the sample audio segment are processed through a language network to obtain a first audio feature.

[0087] The solution provided by the embodiment of the present disclosure is to extract a sample audio segment from the sample audio according to the content of the sample text, and then process the sample text and the sample audio segment through the language network to obtain the first audio feature. Since the sample audio segment has a smaller data volume than the sample audio and contains the sound of all or part of the content in the sample text, it can not only improve the processing efficiency of the language network, but also accurately learn the acoustic features used for speech synthesis, which is conducive to improving the accuracy of the language network generating audio features. Moreover, in the case where the sample audio segment only includes the sound of part of the content in the sample text, it is possible to avoid the language network from learning the content of the sample text from the sample audio segment, that is, to try to ensure that the content learned by the language network comes from the sample text rather than the sample audio, which is conducive to improving the language network's accurate learning of the content of the sample text, thereby facilitating the improvement of the accuracy and training efficiency of the speech synthesis model.

[0088] In some embodiments, based on the content of the sample text, extracting a sample audio segment from the sample audio includes:

[0089] Obtain secondary content from the sample text except for the key content, the key content is used to indicate the theme of the sample text;

[0090] Based on the secondary content, a sample audio segment is extracted from the sample audio, and the content of the sample audio segment is the secondary content.

[0091] The solution provided by the embodiment of the present disclosure extracts a sample audio segment from the sample audio based on the minor content in the sample text. Since the sample audio segment has a smaller data volume than the sample audio and contains the sound of the minor content in the sample text, it can not only improve the processing efficiency of the language network, but also accurately learn the acoustic features used for speech synthesis, which is conducive to improving the accuracy of the language network generating audio features. In addition, since the sample audio segment only contains the minor content in the sample text, it can avoid the language network from learning the key content of the sample text from the sample audio segment as much as possible, that is, it is to ensure that the content learned by the language network comes from the sample text rather than the sample audio as much as possible, which is conducive to improving the language network's accurate learning of the content of the sample text, thereby helping to improve the accuracy and training efficiency of the speech synthesis model.

[0092] In some embodiments, based on the content of the sample text, extracting a sample audio segment from the sample audio includes:

[0093] Obtaining emotional keywords from the content of the sample text, where the emotional keywords are used to represent the emotions conveyed by the content of the sample text;

[0094] Based on the emotion keywords, a sample audio segment is extracted from the sample audio, where the sample audio segment includes the emotion keywords.

[0095] The solution provided by the embodiment of the present disclosure is to extract a sample audio segment from the sample audio based on the emotional keywords in the sample text. Since the sample audio segment has a smaller data volume than the sample audio and contains the sound of part of the content in the sample text, it can not only improve the processing efficiency of the language network, but also accurately learn the acoustic features used for speech synthesis, which is conducive to improving the accuracy of the language network generating audio features. In addition, since the sample audio segment contains the emotional keywords in the sample text, it can not only avoid the language network from learning the main content of the sample text from the sample audio segment, that is, to ensure that the content learned by the language network comes from the sample text rather than the sample audio, which is conducive to improving the language network's accurate learning of the content of the sample text, but also can learn the emotions required by the audio of the sample text from the sample audio segment, which is conducive to generating audio with more accurate and natural emotions and higher quality.

[0096] In some embodiments, the sample audio includes speech of multiple subjects;

[0097] Based on the content of the sample text, a sample audio segment is extracted from the sample audio, including:

[0098] For any object among the multiple objects, a sample audio segment corresponding to the object is extracted from the sample audio based on the content of the sample text and the acoustic features of the object.

[0099] The solution provided by the embodiments of the present disclosure includes the speech of multiple objects in the sample audio, and a sample audio segment of any object can be intercepted from the sample audio according to the acoustic features of the object, so as to subsequently generate audio based on the acoustic features of the object. That is, the speech synthesis model can learn the acoustic features of any object to generate audio, thereby ensuring the flexibility of model training, that is, the user can select an audio segment of an object for model training according to his or her own needs, so that the speech synthesis model can fully learn the acoustic features of the object, which is conducive to the subsequent generation of audio that conforms to the object.

[0100] In some embodiments, performing noise prediction on a sample audio to which reference noise is added based on a first audio feature through a diffusion network to obtain predicted noise includes:

[0101] Acquire a second audio feature and a reference noise feature corresponding to the sample audio, where the reference noise feature is used to represent reference noise;

[0102] By using a diffusion network, a reference noise feature is added to the second audio feature to obtain a third audio feature;

[0103] By using a diffusion network, taking the first audio feature as a condition, performing noise prediction on the third audio feature to obtain a predicted noise feature, where the predicted noise feature is used to represent the predicted noise;

[0104] Based on the reference noise and the predicted noise, the first training loss is determined, including:

[0105] Based on the reference noise feature and the predicted noise feature, a first training loss is determined.

[0106] The solution provided by the embodiment of the present disclosure adopts a diffusion network to learn to reconstruct audio, that is, noise is first added to the sample audio through the diffusion network, and then the added noise is predicted based on the first audio feature predicted by the language network, so as to reconstruct the audio by removing the predicted noise. Since the sample audio and the reference noise are known, the accuracy of the predicted audio depends on the accuracy of the predicted noise, that is, the performance of the diffusion network in determining the predicted noise can reflect the performance of the predicted audio output by the first speech synthesis model (that is, the overall performance of the model). Accordingly, the speech synthesis model is trained through the first training loss, so that the difference between the reference noise and the predicted noise becomes smaller and smaller, that is, the diffusion network can accurately learn the noise in the sample audio, which ensures that the diffusion network can reconstruct more accurate audio, that is, it is beneficial to improve the accuracy of the speech synthesis model, so as to generate higher-quality audio with higher expressiveness and naturalness.

[0107] In some embodiments, the method further comprises:

[0108] Determine a second training loss based on the first audio feature and a second audio feature corresponding to the sample audio, where the second audio feature includes a content feature and an acoustic feature in the sample audio, and the second training loss is used to represent the degree of difference between the first audio feature and the second audio feature;

[0109] Based on the first training loss, updating parameters of the language network and the diffusion network in the first speech synthesis model includes:

[0110] Based on the first training loss and the second training loss, parameters of the language network and the diffusion network in the first speech synthesis model are updated to obtain a second speech synthesis model.

[0111] In the solution provided by the embodiments of the present disclosure, since the first audio feature is the feature of the audio of the sample text predicted by the language network, and the second audio feature is the feature of the real audio, the second training loss can accurately reflect the loss of the audio predicted by the language network. The speech synthesis model is trained by the first training loss and the second training loss, and the speech synthesis model is trained from two perspectives: the diffusion network predicts noise and the language network predicts audio, which is beneficial to improving the accuracy of the speech synthesis model so as to generate higher-quality audio with higher expressiveness and naturalness.

[0112] In some embodiments, the method further comprises:

[0113] Removing the prediction noise from the sample audio to which the reference noise is added to obtain the target audio;

[0114] Based on the sample audio and the target audio, determine a third training loss, where the third training loss is used to represent the degree of difference between the sample audio and the target audio;

[0115] Based on the first training loss, updating parameters of the language network and the diffusion network in the first speech synthesis model to obtain a second speech synthesis model includes:

[0116] Based on the first training loss and the third training loss, the parameters of the language network and the diffusion network in the first speech synthesis model are updated to obtain the second speech synthesis model.

[0117] In the solution provided by the embodiments of the present disclosure, since the target audio is the audio predicted by the speech synthesis model as a whole and the sample audio is real audio, the third training loss can accurately reflect the loss of the audio generated by the speech synthesis model as a whole. The speech synthesis model is trained by the first training loss and the third training loss, and the speech synthesis model is trained by predicting the audio from the speech synthesis model as a whole. This is beneficial to improving the accuracy of the speech synthesis model so that higher-quality audio with higher expressiveness and naturalness can be generated.

[0118] In some embodiments, the sample text and the sample audio are processed by a language network to obtain a first audio feature, including:

[0119] Obtaining the phoneme sequence of the sample text and the acoustic features of the sample audio;

[0120] The phoneme sequence and the acoustic feature are processed through a language network to obtain a first audio feature.

[0121] The solution provided by the embodiments of the present disclosure can train the language network's ability to predict audio features based on phoneme sequences and acoustic features, thereby improving the accuracy of the language network in predicting audio features, by inputting the phoneme sequence of sample text and the acoustic features of sample audio into a language network for processing.

[0122] In some embodiments, the process of obtaining a phoneme sequence of a sample text includes:

[0123] Preprocessing the sample text, the preprocessing including at least one of word segmentation, denoising, text normalization, and polyphonetic pronunciation annotation;

[0124] Based on the correspondence between characters and phonemes, the preprocessed sample text is converted into a phoneme sequence.

[0125] The solution provided by the embodiment of the present disclosure can standardize at least one of the content and pronunciation of the sample text by preprocessing the sample text, so that the sample text can be converted into a phoneme sequence more accurately, thereby improving the accuracy of the phoneme sequence.

[0126] Above Figure 2 The following is only a basic process of the present disclosure. The solution provided by the present disclosure is further described based on a specific implementation method. Figure 3 FIG. 1 is a flowchart of another method for training a speech synthesis model according to an exemplary embodiment. Figure 3 , the method comprising:

[0127] In step 301, the electronic device extracts a sample audio segment from the sample audio based on the content of the sample text, the sample text and the sample audio content are consistent, and the sample audio segment includes the sound of all or part of the content in the sample text.

[0128] In an embodiment of the present disclosure, the electronic device can directly train the first speech synthesis model based on the sample text and the sample audio. Alternatively, considering that the amount of sample audio data is large, and the corresponding amount of audio feature data to be processed by the model is large, the electronic device can also intercept part of the audio from the sample audio for training based on the sample text. Among them, the electronic device can randomly intercept sample audio segments from the sample audio, and can also intercept sample audio segments based on the content of the sample text. The embodiment of the present disclosure does not limit the method of intercepting sample audio segments. For example, intercepting the part of the sound that contains all the content in the sample text from the sample audio is equivalent to removing the part of the sample audio that is left blank or only has background sound. The following exemplifies three ways of intercepting sample audio segments, but is by no means limited to this.

[0129] In the first way, the electronic device can intercept the sample audio segment according to the secondary content in the sample text. Accordingly, the process of the electronic device intercepting the sample audio segment from the sample audio includes: the electronic device obtains the secondary content except the key content from the sample text. Then, the electronic device intercepts the sample audio segment from the sample audio based on the secondary content, and the content of the sample audio segment is the secondary content. Among them, the key content is used to represent the theme of the sample text. The electronic device can determine the key content in the sample text by performing theme recognition on the sample text, so as to obtain the secondary content in the sample text. The secondary content can be auxiliary descriptions (such as certain adjectives), background information (such as the first sentence in the text), etc. in the sample text, which is not limited in the embodiments of the present disclosure.

[0130] The solution provided by the embodiment of the present disclosure extracts a sample audio segment from the sample audio based on the minor content in the sample text. Since the sample audio segment has a smaller data volume than the sample audio and contains the sound of the minor content in the sample text, it can not only improve the processing efficiency of the language network, but also accurately learn the acoustic features used for speech synthesis, which is conducive to improving the accuracy of the language network generating audio features. In addition, since the sample audio segment only contains the minor content in the sample text, it can avoid the language network from learning the key content of the sample text from the sample audio segment as much as possible, that is, it is to ensure that the content learned by the language network comes from the sample text rather than the sample audio as much as possible, which is conducive to improving the language network's accurate learning of the content of the sample text, thereby helping to improve the accuracy and training efficiency of the speech synthesis model.

[0131] In the second way, the electronic device can intercept the sample audio clip based on the emotional keywords in the sample text. Accordingly, the process of the electronic device intercepting the sample audio clip from the sample audio includes: the electronic device obtains the emotional keywords from the content of the sample text. Then, the electronic device intercepts the sample audio clip from the sample audio based on the emotional keywords. The sample audio clip includes emotional keywords. The emotional keywords are used to represent the emotions conveyed by the content of the sample text. The emotional keywords can be happy, angry, contemptuous, etc., which is not limited in the embodiments of the present disclosure. The electronic device can determine the emotional keywords in the sample text by performing emotion recognition on the sample text. The embodiments of the present disclosure do not limit the number of emotional keywords.

[0132] The solution provided by the embodiment of the present disclosure is to extract a sample audio segment from the sample audio based on the emotional keywords in the sample text. Since the sample audio segment has a smaller data volume than the sample audio and contains the sound of part of the content in the sample text, it can not only improve the processing efficiency of the language network, but also accurately learn the acoustic features used for speech synthesis, which is conducive to improving the accuracy of the language network generating audio features. In addition, since the sample audio segment contains the emotional keywords in the sample text, it can not only avoid the language network from learning the main content of the sample text from the sample audio segment, that is, to ensure that the content learned by the language network comes from the sample text rather than the sample audio, which is conducive to improving the language network's accurate learning of the content of the sample text, but also can learn the emotions required by the audio of the sample text from the sample audio segment, which is conducive to generating audio with more accurate and natural emotions and higher quality.

[0133] In the third way, the sample audio includes the speech of multiple objects, and the electronic device can intercept the speech of any object for model training. Accordingly, the process of the electronic device intercepting the sample audio segment from the sample audio includes: for any object among the multiple objects, the electronic device intercepts the sample audio segment corresponding to the object from the sample audio based on the content of the sample text and the acoustic features of the object. Among them, the object can be the object to which the acoustic features used for speech synthesis belong. That is, the electronic device can first determine the object to which the acoustic features used for speech synthesis belong, and then intercept the corresponding sample audio segment from the sample audio. The object can be specified by the user, and the embodiments of the present disclosure are not limited to this.

[0134] The solution provided by the embodiments of the present disclosure includes the speech of multiple objects in the sample audio, and a sample audio segment of any object can be intercepted from the sample audio according to the acoustic features of the object, so as to subsequently generate audio based on the acoustic features of the object. That is, the speech synthesis model can learn the acoustic features of any object to generate audio, thereby ensuring the flexibility of model training, that is, the user can select an audio segment of an object for model training according to his or her own needs, so that the speech synthesis model can fully learn the acoustic features of the object, which is conducive to the subsequent generation of audio that conforms to the object.

[0135] Before extracting the sample audio segment through the sample text, the electronic device may preprocess the sample text first, and then extract the sample audio segment according to the preprocessed sample text to improve the quality of the sample audio segment.

[0136] Among them, the preprocessing includes at least one of word segmentation, denoising, text normalization and polyphonic pronunciation annotation. Word segmentation refers to dividing the sample text into multiple characters. Denoising refers to removing meaningless symbols (such as punctuation marks, emoticons, etc.) in the sample text. Text normalization refers to converting English abbreviations, numbers and other contents in the sample text into standard forms. For example, "TTS" in the sample text is converted to "Text To Speech"; "857" in the sample text is converted to "eight hundred and fifty-seven". Polyphonic pronunciation annotation refers to annotating the pronunciation of polyphonic characters in the sample text. In the process of annotating the pronunciation, the electronic device can determine the pronunciation of the polyphonic character according to the phrase to which the polyphonic character belongs, and the embodiment of the present disclosure is not limited to this. For example, when the phrase "weight" is included in the sample text, the electronic device can annotate the pronunciation of "heavy" as "zhong-fourth tone". When the phrase "repeat" is included in the sample text, the electronic device can annotate the pronunciation of "heavy" as "chong-second tone".

[0137] In step 302, the electronic device processes the sample text and the sample audio segment through the language network in the first speech synthesis model to obtain a first audio feature, where the first audio feature includes the content feature of the sample text and the acoustic feature of the sample audio.

[0138] In the embodiments of the present disclosure, the electronic device may input the sample text and the sample audio segment into the language network in the first speech synthesis model, and process the sample text and the sample audio segment through the language network to extract the content feature of the sample text and the acoustic feature of the sample audio segment, so as to obtain the first audio feature corresponding to the sample text. Alternatively, the electronic device may also extract the content feature of the sample text and the acoustic feature of the sample audio segment through other network models outside the language network, and then input the content feature of the sample text and the acoustic feature of the sample audio segment into the language network for processing, so as to obtain the first audio feature corresponding to the sample text. The embodiments of the present disclosure do not limit this. The first audio feature is the audio feature of the sample text predicted by the language network.

[0139] In some embodiments, the language network can predict the first audio feature according to the phoneme sequence of the sample text and the acoustic feature of the sample audio. Correspondingly, the electronic device obtains the phoneme sequence of the sample text and the acoustic feature of the sample audio. Then, the electronic device processes the phoneme sequence and the acoustic feature through the language network to obtain the first audio feature. That is, the electronic device obtains the phoneme sequence of the sample text and the acoustic feature of the sample audio segment. Then, the electronic device processes the phoneme sequence and the acoustic feature through the language network to obtain the first audio feature.

[0140] Among them, the phoneme sequence includes the phonemes of multiple characters in the sample text. The phoneme sequence can represent the text content of the sample text. For example, when the sample text contains the character "今", the phoneme sequence includes the three phonemes "ji n" of this character. Then, the electronic device inputs the phoneme sequence of the sample text and the acoustic feature of the sample audio segment into the language network, and processes the phoneme sequence and the acoustic feature through the language network to predict the first audio feature. The solution provided by the embodiments of the present disclosure can train the ability of the language network to predict audio features according to the phoneme sequence and the acoustic feature by inputting the phoneme sequence of the sample text and the acoustic feature of the sample audio into the language network for processing, and further improve the accuracy of the language network in predicting audio features.

[0141] In the process of obtaining the phoneme sequence of the sample text, the electronic device can preprocess the sample text. The preprocessing includes at least one of word segmentation, denoising, text normalization, and polyphonetic pronunciation annotation. Then, the electronic device converts the preprocessed sample text into a phoneme sequence based on the correspondence between the characters and the phonemes. That is, the electronic device determines the factors of each character in the sample text according to the correspondence between the characters and the phonemes. Then, the electronic device sorts the audio of each character according to the arrangement order of each character in the sample text to obtain the phoneme sequence corresponding to the sample text. The solution provided by the embodiment of the present disclosure can standardize at least one of the content and pronunciation of the sample text by preprocessing the sample text, so that the sample text can be converted into a phoneme sequence more accurately, thereby improving the accuracy of the phoneme sequence.

[0142] In the process of obtaining the phoneme sequence, the electronic device can use the G2P (Grapheme-to-Phoneme) model to process the preprocessed sample text to obtain the phoneme sequence of the sample text. The present disclosure embodiment does not limit the method of obtaining the phoneme series. In the process of obtaining the acoustic features, the electronic device can use the prompt encoder to process the sample audio clip to obtain the acoustic features of the sample audio clip to reflect the acoustic information such as timbre, sound intensity, pitch, and sound length used for speech synthesis.

[0143] In step 303, the electronic device performs noise prediction on the sample audio to which the reference noise is added based on the first audio feature through the diffusion network in the first speech synthesis model to obtain predicted noise.

[0144] In the disclosed embodiment, the diffusion network includes a forward diffusion process and a reverse diffusion process. The forward diffusion process of the diffusion network is to add noise to the sample audio. The reverse diffusion process of the diffusion network is the process of predicting and removing noise from the sample audio to which noise is added. After the language network outputs the first audio feature, the first audio feature is input into the diffusion network in the first speech synthesis model. The electronic device uses the diffusion network to perform noise prediction on the sample audio to which reference noise is added, taking the first audio feature as a condition, to obtain predicted noise. Then, the electronic device uses the prediction network to remove the predicted noise from the sample audio to which reference noise is added, to obtain predicted audio. The predicted audio is the audio that the diffusion network uses to reconstruct the audio features based on the text.

[0145] Among them, in the forward diffusion process, the electronic device adds n-times noise (n-times noise is the reference noise) to the sample audio through the diffusion network, so that the sample audio changes from a clear state to a completely random state. This process can be regarded as the diffusion of data in noise. By gradually increasing the noise, the information of the original data is gradually obscured. In the reverse diffusion process, the electronic device gradually removes the predicted noise from the sample audio with the reference noise added through the diffusion network, based on the first audio feature (that is, learning the noise distribution in the forward diffusion process and removing the n-times noise), so that the predicted audio after removing the predicted noise meets the first audio feature.

[0146] In some embodiments, the above-mentioned processes of adding noise, predicting noise, and removing noise are all processed in the dimension of features. Accordingly, the electronic device obtains the second audio feature and the reference noise feature corresponding to the sample audio, and the reference noise feature is used to represent the reference noise. The electronic device adds the reference noise feature to the second audio feature through a diffusion network to obtain a third audio feature. Then, the electronic device performs noise prediction on the third audio feature through a diffusion network with the first audio feature as a condition to obtain a predicted noise feature. The predicted noise feature is used to represent the predicted noise. Subsequently, the electronic device can determine the first training loss based on the reference noise feature and the predicted noise feature.

[0147] The solution provided by the embodiment of the present disclosure adopts a diffusion network to learn to reconstruct audio, that is, noise is first added to the sample audio through the diffusion network, and then the added noise is predicted based on the first audio feature predicted by the language network, so as to reconstruct the audio by removing the predicted noise. Since the sample audio and the reference noise are known, the accuracy of the predicted audio depends on the accuracy of the predicted noise, that is, the performance of the diffusion network in determining the predicted noise can reflect the performance of the predicted audio output by the first speech synthesis model (that is, the overall performance of the model). Accordingly, the speech synthesis model is trained through the first training loss, so that the difference between the reference noise and the predicted noise becomes smaller and smaller, that is, the diffusion network can accurately learn the noise in the sample audio, which ensures that the diffusion network can reconstruct more accurate audio, that is, it is beneficial to improve the accuracy of the speech synthesis model, so as to generate higher-quality audio with higher expressiveness and naturalness.

[0148] The reference noise may be Gaussian noise randomly generated by a diffusion network, which is not limited in the embodiments of the present disclosure.

[0149] In step 304, the electronic device determines a first training loss based on the reference noise and the predicted noise, where the first training loss is used to represent the degree of difference between the reference noise and the predicted noise.

[0150] In an embodiment of the present disclosure, the electronic device can determine the first training loss of the first speech synthesis model according to the difference between the reference noise and the predicted noise. The larger the first training loss, the greater the difference between the reference noise and the predicted noise; the smaller the first training loss, the smaller the difference between the reference noise and the predicted noise.

[0151] In some embodiments, the first training loss is a mean square error loss. Accordingly, the electronic device calculates the mean square error loss between the reference noise feature and the predicted noise according to the mean square error loss function, and determines the mean square error loss as the first training loss. The reference noise feature is used to represent the reference noise; the predicted noise feature is used to represent the predicted noise. By determining the mean square error loss between the reference noise and the predicted noise as the second training loss of the speech synthesis model, the second training loss can accurately reflect the difference between the noise predicted by the speech synthesis model and the reference noise, thereby improving the accuracy and efficiency of training the first speech synthesis model through the first training loss.

[0152] In step 305, the electronic device updates parameters of the language network and the diffusion network in the first speech synthesis model based on the first training loss to obtain a second speech synthesis model.

[0153] In the disclosed embodiment, the electronic device can update the parameters of the language network and the diffusion network in the first speech synthesis model in a multi-round iteration manner with the goal of minimizing the first training loss, until the second speech synthesis model obtained after the parameter update meets the training stop condition. The training stop condition can be that the first training loss is lower than the first loss threshold, or the number of parameter updates reaches a preset number, etc., which is not limited in the disclosed embodiment.

[0154] In order to more clearly describe the training process of the first speech synthesis model, the training process is further described below with reference to the accompanying drawings. Figure 4 FIG. 1 is a schematic diagram showing a training process of a first speech synthesis model according to an exemplary embodiment. Figure 4As shown, the first speech synthesis model includes a language network and a diffusion network. The electronic device converts the sample text into a phoneme sequence (Phoneme embedding) through the G2P model. The electronic device extracts the acoustic features (Prompt embedding) of the sample audio through the Prompt Encoder. Then, the electronic device inputs the phoneme sequence and the acoustic features into the language network in the first speech synthesis model, processes the phoneme sequence and the acoustic features through the language network, and obtains the first audio feature (a series of hidden variables). Then, the electronic device performs noise prediction on the third audio feature (the second audio feature with the reference noise feature added) through the diffusion network in the first speech synthesis model, with the first audio feature as a condition, to obtain the predicted noise feature. Then, the electronic device determines the first training loss based on the reference noise feature and the predicted noise feature, trains the first speech synthesis model based on the first training loss, and obtains the second speech synthesis model.

[0155] In some embodiments, in addition to model training through the first training loss, the electronic device can also calculate the language network prediction audio feature loss to perform model training. Accordingly, the electronic device determines the second training loss based on the first audio feature and the second audio feature corresponding to the sample audio. The second audio feature includes content features and acoustic features in the sample audio. The second training loss is used to represent the degree of difference between the first audio feature and the second audio feature. Then, based on the first training loss and the second training loss, the electronic device updates the parameters of the language network and the diffusion network in the first speech synthesis model to obtain a second speech synthesis model.

[0156] The second training loss may be a cross entropy loss. The electronic device can use a cross entropy loss function to calculate the cross entropy loss between the first audio feature and the second audio feature corresponding to the sample audio, and use the cross entropy loss as the second training loss. By determining the cross entropy loss between the first audio feature and the second audio feature as the second training loss of the speech synthesis model, the second training loss can accurately reflect the difference between the audio features predicted by the language network and the actual audio features, thereby improving the accuracy and efficiency of training the speech synthesis model through the second training loss.

[0157] In the solution provided by the embodiments of the present disclosure, since the first audio feature is the feature of the audio of the sample text predicted by the language network, and the second audio feature is the feature of the real audio, the second training loss can accurately reflect the loss of the audio predicted by the language network. The speech synthesis model is trained by the first training loss and the second training loss, and the speech synthesis model is trained from two perspectives: the diffusion network predicts noise and the language network predicts audio, which is beneficial to improving the accuracy of the speech synthesis model so as to generate higher-quality audio with higher expressiveness and naturalness.

[0158] In some embodiments, in addition to model training through the first training loss, the electronic device can also calculate the loss of the overall predicted audio of the speech synthesis model to perform model training. Accordingly, the electronic device removes the predicted noise from the sample audio to which the reference noise is added to obtain the target audio. Then, the electronic device determines the third training loss based on the sample audio and the target audio. The third training loss is used to indicate the degree of difference between the sample audio and the target audio. Then, based on the first training loss and the third training loss, the electronic device updates the parameters of the language network and the diffusion network in the first speech synthesis model to obtain a second speech synthesis model.

[0159] Among them, the above-mentioned processes of removing noise and calculating the third training loss can be processed in the dimension of features. Correspondingly, the electronic device removes the predicted noise feature from the third audio feature (the second audio feature with the reference noise feature added) to obtain the target audio feature. Then, the electronic device determines the third training loss based on the second audio feature and the target audio feature corresponding to the sample audio. Alternatively, the electronic device can also calculate the third training loss in the dimension of audio after obtaining the target audio. Optionally, the electronic device can respectively extract the Mel spectrum graph of the sample audio and the Mel spectrum graph of the target audio, calculate the gap between the Mel spectrum graph of the sample audio and the Mel spectrum graph of the target audio, and use the gap as the third training loss. The embodiment of the present disclosure does not limit the calculation method of the third training loss. The third training loss can be a mean square error loss, a cross entropy loss, or a mean absolute error loss, etc., which is also not limited in the embodiment of the present disclosure.

[0160] In the solution provided by the embodiments of the present disclosure, since the target audio is the audio predicted by the speech synthesis model as a whole and the sample audio is real audio, the third training loss can accurately reflect the loss of the audio generated by the speech synthesis model as a whole. The speech synthesis model is trained by the first training loss and the third training loss, and the speech synthesis model is trained by predicting the audio from the speech synthesis model as a whole. This is beneficial to improving the accuracy of the speech synthesis model so that higher-quality audio with higher expressiveness and naturalness can be generated.

[0161] In some embodiments, the electronic device may also update parameters of the language network and the diffusion network in the first speech synthesis model based on the first training loss, the second training loss and the third training loss to obtain a second speech synthesis model, which will not be repeated here.

[0162] In some embodiments, during the above-mentioned model training process, the electronic device can use at least one optimization method such as gradient clipping, learning rate decay, regularization, etc. to optimize the performance of the first speech synthesis model. For example, the electronic device can adjust the learning rate during the training process through a learning rate scheduler so that the learning rates corresponding to different iteration rounds are not exactly the same. Alternatively, the electronic device can also scale or truncate the back-propagated gradient during the model training using the back-propagation algorithm to keep the gradient within a reasonable range, thereby avoiding situations such as gradient explosion or gradient disappearance. Optionally, the electronic device can perform gradient clipping according to the absolute value of the gradient or the norm of the gradient.

[0163] After the model training is completed, the electronic device can also evaluate the quality of the speech generated by the second speech synthesis model by at least one method such as Mel Cepstral Distortion (MCD) and PESQ (Perceptual evaluation of speech quality). If the quality of the generated speech does not meet the preset standard, the electronic device can continue to fine-tune the parameters of the language network and the diffusion network in the second speech synthesis model to improve the naturalness and expressiveness of the speech synthesis.

[0164] The disclosed embodiment provides a method for training a speech synthesis model, which uses sample text and sample audio with consistent content to jointly train the language network and diffusion network in the speech synthesis model, so that the speech synthesis model can learn to convert text into audio containing acoustic features as a whole, thereby achieving the purpose of end-to-end training of the speech synthesis model, simplifying the training steps, and helping to improve the training efficiency of the model; and, this solution uses a diffusion network to learn to reconstruct audio, and trains the speech synthesis model through a first training loss, so that the difference between reference noise and predicted noise becomes smaller and smaller, that is, the diffusion network can accurately learn the noise in the sample audio, ensuring that the diffusion network can reconstruct more accurate audio, that is, helping to improve the accuracy of the speech synthesis model, so as to generate higher-quality audio with higher expressiveness and naturalness.

[0165] In addition, compared with the solution in the related technology of "first learning the semantic information of the text, and then converting the semantic information into audio containing acoustic features", this solution does not require semantic encoding of the text, thereby avoiding the dependence of audio reconstruction on the accuracy of semantic encoding, which is conducive to ensuring the reliability and stability of the speech synthesis model.

[0166] The above embodiments mainly introduce the training process of the speech synthesis model. In some embodiments, the electronic device can implement the speech synthesis task through the trained second speech synthesis model. Figure 5 is a flowchart of a speech synthesis method according to an exemplary embodiment, the method is executed by an electronic device, see Figure 5 , the method comprising:

[0167] In step 501, the electronic device obtains input text and prompt audio, where the prompt audio is used to indicate acoustic information used for speech synthesis.

[0168] In the embodiment of the present disclosure, the input text is the text to be speech synthesized. For example, the input text is the navigation text generated by a navigation application, or the reply text of a voice assistant to a user, etc., which is not limited in the embodiment of the present disclosure. The prompt audio is used to provide acoustic features for speech synthesis. For example, the prompt audio can be a voice of a celebrity selected by the user. Before performing speech synthesis, the electronic device obtains the input text and the prompt audio. Before inputting into the second speech synthesis model, the electronic device can preprocess the input text. For details, please refer to the preprocessing process in step 301, which will not be repeated here.

[0169] In step 502, the electronic device processes the input text and the prompt audio through the language network in the second speech synthesis model to obtain the intermediate audio features corresponding to the input text.

[0170] In an embodiment of the present disclosure, the electronic device can input the input text and the prompt audio into the language network in the second speech synthesis model, process the input text and the prompt audio through the language network, extract the content features of the input text and the acoustic features of the sample audio segment, and obtain the intermediate audio features corresponding to the input text. Alternatively, the electronic device can also extract the content features of the input text and the acoustic features of the prompt audio through other network models other than the language network, and then input the content features of the input text and the acoustic features of the prompt audio into the language network for processing to obtain the intermediate audio features corresponding to the input text. This embodiment of the present disclosure is not limited to this. The intermediate audio features are the audio features of the input text predicted by the language network. For details, please refer to the process of obtaining the phoneme sequence and acoustic features in step 302, which will not be repeated here.

[0171] In step 503, the electronic device performs denoising processing on the intermediate audio features through the diffusion network in the second speech synthesis model to obtain target audio features, and generates output audio based on the target audio features.

[0172] In the disclosed embodiment, after the language network outputs the intermediate audio features, the intermediate audio features are input into the diffusion network in the second speech synthesis model. The electronic device performs denoising on the intermediate audio features through the diffusion network to obtain the target audio features. Then, the electronic device converts the target audio features into output audio. The content of the output audio is consistent with the content of the input text, thereby realizing speech synthesis of the input text.

[0173] The disclosed embodiment provides a speech synthesis method, which can perform speech synthesis on any input text through a second speech synthesis model to obtain output audio consistent with the content of the input text. Since the second speech synthesis model is trained based on the training method of the speech synthesis model, the second speech synthesis model can accurately obtain output audio consistent with the content of the input text, thereby improving the accuracy of speech synthesis.

[0174] Moreover, since the input text is preprocessed and converted into a phoneme sequence during the speech synthesis process, the content of the input text can be standardized, which is conducive to the speech synthesis model to generate more accurate output audio according to the standardized phoneme sequence, reducing the occurrence of missing words, omissions and pronunciation errors, and improving the accuracy of speech synthesis.

[0175] All the above optional technical solutions can be arbitrarily combined to form optional embodiments of the present disclosure, and will not be described in detail here.

[0176] Figure 6 FIG. 1 is a block diagram of a training device for a speech synthesis model according to an exemplary embodiment. Figure 6 The first speech synthesis model includes a language network and a diffusion network, the language network is used to predict the audio features of the text, and the diffusion network is used to reconstruct the audio based on the audio features of the text; the device includes:

[0177] The processing unit 601 is configured to process the sample text and the sample audio through a language network to obtain a first audio feature, wherein the sample text and the sample audio have the same content, and the first audio feature includes a content feature of the sample text and an acoustic feature of the sample audio;

[0178] The prediction unit 602 is configured to perform noise prediction on the sample audio to which the reference noise is added based on the first audio feature through a diffusion network to obtain predicted noise;

[0179] A first determining unit 603 is configured to determine a first training loss based on the reference noise and the predicted noise, where the first training loss is used to represent the degree of difference between the reference noise and the predicted noise;

[0180] The updating unit 604 is configured to update parameters of the language network and the diffusion network in the first speech synthesis model based on the first training loss to obtain a second speech synthesis model.

[0181] In some embodiments, the processing unit 601 includes:

[0182] The interception subunit is configured to execute the content based on the sample text, intercepting the sample audio segment from the sample audio, wherein the sample audio segment includes the sound of all or part of the content in the sample text;

[0183] The processing subunit is configured to process the sample text and the sample audio segment through a language network to obtain a first audio feature.

[0184] In some embodiments, the interception subunit is configured to obtain secondary content other than key content from the sample text, where the key content is used to represent the theme of the sample text; based on the secondary content, a sample audio segment is intercepted from the sample audio, and the content of the sample audio segment is the secondary content.

[0185] In some embodiments, the interception subunit is configured to obtain emotion keywords from the content of the sample text, where the emotion keywords are used to represent the emotions conveyed by the content of the sample text; based on the emotion keywords, a sample audio segment is intercepted from the sample audio, where the sample audio segment includes the emotion keywords.

[0186] In some embodiments, the sample audio includes speech of multiple subjects;

[0187] The interception subunit is configured to execute, for any object among the multiple objects, intercepting a sample audio segment corresponding to the object from the sample audio based on the content of the sample text and the acoustic features of the object.

[0188] In some embodiments, the prediction unit 602 is configured to execute the acquisition of a second audio feature and a reference noise feature corresponding to the sample audio, where the reference noise feature is used to represent the reference noise; adding the reference noise feature to the second audio feature through a diffusion network to obtain a third audio feature; performing noise prediction on the third audio feature through a diffusion network with the first audio feature as a condition to obtain a predicted noise feature, where the predicted noise feature is used to represent the predicted noise;

[0189] The first determining unit 603 is configured to determine a first training loss based on the reference noise feature and the predicted noise feature.

[0190] In some embodiments, the apparatus further comprises:

[0191] A second determining unit is configured to determine a second training loss based on the first audio feature and a second audio feature corresponding to the sample audio, where the second audio feature includes a content feature and an acoustic feature in the sample audio, and the second training loss is used to represent a degree of difference between the first audio feature and the second audio feature;

[0192] The updating unit 604 is configured to update parameters of the language network and the diffusion network in the first speech synthesis model based on the first training loss and the second training loss to obtain a second speech synthesis model.

[0193] In some embodiments, the apparatus further comprises:

[0194] A third determination unit is configured to remove the predicted noise from the sample audio to which the reference noise is added to obtain the target audio; based on the sample audio and the target audio, determine a third training loss, the third training loss is used to represent the degree of difference between the sample audio and the target audio;

[0195] The updating unit 604 is configured to update parameters of the language network and the diffusion network in the first speech synthesis model based on the first training loss and the third training loss to obtain a second speech synthesis model.

[0196] In some embodiments, the processing unit 601 includes:

[0197] An acquisition subunit, configured to acquire a phoneme sequence of a sample text and an acoustic feature of a sample audio;

[0198] The processing subunit is configured to process the phoneme sequence and the acoustic feature through a language network to obtain a first audio feature.

[0199] In some embodiments, the acquisition subunit is configured to perform preprocessing on the sample text, the preprocessing including at least one of word segmentation, denoising, text normalization, and pronunciation annotation of polyphones; based on the correspondence between characters and phonemes, the preprocessed sample text is converted into a phoneme sequence.

[0200] The disclosed embodiment provides a training device for a speech synthesis model, which uses sample text and sample audio with consistent content to jointly train the language network and the diffusion network in the speech synthesis model, so that the speech synthesis model can learn as a whole to convert text into audio containing acoustic features, thereby achieving the purpose of end-to-end training of the speech synthesis model, simplifying the training steps, and helping to improve the training efficiency of the model; and, this solution uses a diffusion network to learn to reconstruct audio, and trains the speech synthesis model through a first training loss, so that the difference between the reference noise and the predicted noise becomes smaller and smaller, that is, the diffusion network can accurately learn the noise in the sample audio, ensuring that the diffusion network can reconstruct more accurate audio, that is, helping to improve the accuracy of the speech synthesis model, so as to be able to generate higher-quality audio with higher expressiveness and naturalness.

[0201] In addition, compared with the solution in the related technology of "first learning the semantic information of the text, and then converting the semantic information into audio containing acoustic features", this solution does not require semantic encoding of the text, thereby avoiding the dependence of audio reconstruction on the accuracy of semantic encoding, which is conducive to ensuring the reliability and stability of the speech synthesis model.

[0202] It should be noted that the training device for the speech synthesis model provided in the above embodiment only uses the division of the above functional units as an example when training the speech synthesis model. In actual applications, the above functions can be assigned to different functional units as needed, that is, the internal structure of the electronic device is divided into different functional units to complete all or part of the functions described above. In addition, the training device for the speech synthesis model provided in the above embodiment and the training method embodiment of the speech synthesis model belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0203] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0204] Figure 7 is a block diagram of a speech synthesis device according to an exemplary embodiment. Figure 7 , the device comprises:

[0205] An acquisition unit 701 is configured to acquire input text and prompt audio, where the prompt audio is used to indicate acoustic information used for speech synthesis;

[0206] The first processing unit 702 is configured to execute processing of the input text and the prompt audio through the language network in the second speech synthesis model to obtain the intermediate audio features corresponding to the input text, and the second speech synthesis model is trained based on the training method of the speech synthesis model;

[0207] The second processing unit 703 is configured to perform denoising processing on the intermediate audio features through the diffusion network in the second speech synthesis model to obtain target audio features, and generate output audio based on the target audio features.

[0208] The disclosed embodiment provides a speech synthesis device, which can perform speech synthesis on any input text through a second speech synthesis model to obtain output audio consistent with the content of the input text. Since the second speech synthesis model is trained based on the training method of the speech synthesis model, the second speech synthesis model can accurately obtain output audio consistent with the content of the input text, thereby improving the accuracy of speech synthesis.

[0209] It should be noted that the speech synthesis device provided in the above embodiment only uses the division of the above functional units as an example when generating audio based on text. In actual applications, the above functions can be assigned to different functional units as needed, that is, the internal structure of the electronic device is divided into different functional units to complete all or part of the functions described above. In addition, the speech synthesis device provided in the above embodiment and the speech synthesis method embodiment belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0210] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0211] When an electronic device is provided as a terminal, Figure 8 FIG. 8 is a block diagram of a terminal 800 according to an exemplary embodiment. Figure 8 The structure block diagram of a terminal 800 provided by an exemplary embodiment of the present disclosure is shown. The terminal 800 may be: a smart phone, a tablet computer, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 (Moving Picture Experts Group Audio Layer IV), a laptop computer or a desktop computer. The terminal 800 may also be called a user device, a portable terminal, a laptop terminal, a desktop terminal or other names.

[0212] Typically, the terminal 800 includes a processor 801 and a memory 802 .

[0213] The processor 801 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 801 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 801 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 801 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 801 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0214] The memory 802 may include one or more computer-readable storage media, which may be non-transitory. The memory 802 may also include a high-speed random access memory and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 802 is used to store at least one computer program, which is used to be executed by the processor 801 to implement the training method or speech synthesis method of the speech synthesis model provided in the method embodiment of the present application.

[0215] In some embodiments, the terminal 800 may further optionally include: a peripheral device interface 803 and at least one peripheral device. The processor 801, the memory 802 and the peripheral device interface 803 may be connected via a bus or a signal line. Each peripheral device may be connected to the peripheral device interface 803 via a bus, a signal line or a circuit board. Specifically, the peripheral device includes: at least one of a radio frequency circuit 804, a display screen 805, a camera assembly 806, an audio circuit 807 and a power supply 808.

[0216] The peripheral device interface 803 may be used to connect at least one peripheral device related to I / O (Input / Output) to the processor 801 and the memory 802. In some embodiments, the processor 801, the memory 802, and the peripheral device interface 803 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 801, the memory 802, and the peripheral device interface 803 may be implemented on a separate chip or circuit board, which is not limited in this embodiment.

[0217] The radio frequency circuit 804 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 804 communicates with the communication network and other communication devices through electromagnetic signals. The radio frequency circuit 804 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals into electrical signals. In some embodiments, the radio frequency circuit 804 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, and the like. The radio frequency circuit 804 can communicate with other terminals through at least one wireless communication protocol. The wireless communication protocol includes, but is not limited to: the World Wide Web, a metropolitan area network, an intranet, various generations of mobile communication networks (2G, 3G, 4G and 5G), a wireless local area network and / or a WiFi (Wireless Fidelity) network. In some embodiments, the radio frequency circuit 804 may also include circuits related to NFC (Near Field Communication), which is not limited in this application.

[0218] The display screen 805 is used to display the UI (User Interface). The UI may include graphics, text, icons, videos, and any combination thereof. When the display screen 805 is a touch display screen, the display screen 805 also has the ability to collect touch signals on the surface or above the surface of the display screen 805. The touch signal can be input to the processor 801 as a control signal for processing. At this time, the display screen 805 can also be used to provide virtual buttons and / or virtual keyboards, also known as soft buttons and / or soft keyboards. In some embodiments, the display screen 805 can be one, set on the front panel of the terminal 800; in other embodiments, the display screen 805 can be at least two, respectively set on different surfaces of the terminal 800 or in a folding design; in other embodiments, the display screen 805 can be a flexible display screen, set on the curved surface or folding surface of the terminal 800. Even, the display screen 805 can also be set to a non-rectangular irregular shape, that is, a special-shaped screen. The display screen 805 can be made of materials such as LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode, organic light-emitting diode).

[0219] The camera assembly 806 is used to capture images or videos. In some embodiments, the camera assembly 806 includes a front camera and a rear camera. Typically, the front camera is disposed on the front panel of the terminal, and the rear camera is disposed on the back of the terminal. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth of field camera, a wide-angle camera, and a telephoto camera, so as to realize the fusion of the main camera and the depth of field camera to realize the background blur function, the fusion of the main camera and the wide-angle camera to realize panoramic shooting and VR (Virtual Reality) shooting function or other fusion shooting functions. In some embodiments, the camera assembly 806 may also include a flash. The flash may be a monochrome temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cold light flash, which can be used for light compensation at different color temperatures.

[0220] The audio circuit 807 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals and input them into the processor 801 for processing, or input them into the radio frequency circuit 804 to achieve voice communication. For the purpose of stereo acquisition or noise reduction, there may be multiple microphones, which are respectively arranged at different parts of the terminal 800. The microphone may also be an array microphone or an omnidirectional acquisition microphone. The speaker is used to convert the electrical signal from the processor 801 or the radio frequency circuit 804 into sound waves. The speaker may be a traditional film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert the electrical signal into sound waves audible to humans, but also convert the electrical signal into sound waves inaudible to humans for purposes such as ranging. In some embodiments, the audio circuit 807 may also include a headphone jack.

[0221] The power supply 808 is used to power various components in the terminal 800. The power supply 808 can be an alternating current, a direct current, a disposable battery, or a rechargeable battery. When the power supply 808 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery charged through a wired line, and a wireless rechargeable battery is a battery charged through a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0222] Those skilled in the art will understand that Figure 8 The structure shown in the figure does not constitute a limitation on the terminal 800, and the terminal 800 may include more or less components than those shown in the figure, or combine some components, or adopt a different component arrangement.

[0223] When the electronic device is provided as a server, Fig. 9 It is a block diagram of a server 900 according to an exemplary embodiment. The server 900 may have relatively large differences due to different configurations or performances, and may include one or more processors (Central Processing Units, CPU) 901 and one or more memories 902, wherein the memory 902 stores at least one program code, and the at least one program code is loaded and executed by the processor 901 to implement the training method of the speech synthesis model or the speech synthesis method provided by the above-mentioned various method embodiments. Of course, the server may also have components such as a wired or wireless network interface, a keyboard, and an input and output interface for input and output. The server 900 may also include other components for implementing device functions, which will not be described in detail here.

[0224] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory 802 or a memory 902 including instructions, and the instructions can be executed by the processor 801 of the terminal 800 or the processor 901 of the server 900 to complete the training method of the speech synthesis model or the speech synthesis method. Optionally, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0225] A computer program product includes a computer program / instruction, which, when executed by a processor, implements the training method of the speech synthesis model or the speech synthesis method.

[0226] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0227] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A method for training a speech synthesis model, characterized in that: The first speech synthesis model includes a language network and a diffusion network, wherein the language network is used to predict audio features of the text, and the diffusion network is used to reconstruct the audio based on the audio features of the text; the method includes: Processing the sample text and the sample audio through the language network to obtain a first audio feature, wherein the sample text and the sample audio have the same content, and the first audio feature includes a content feature of the sample text and an acoustic feature of the sample audio; By using the diffusion network, based on the first audio feature, noise prediction is performed on the sample audio to which reference noise is added to obtain predicted noise; Based on the reference noise and the predicted noise, determining a first training loss, wherein the first training loss is used to represent a degree of difference between the reference noise and the predicted noise; Based on the first training loss, parameters of the language network and the diffusion network in the first speech synthesis model are updated to obtain a second speech synthesis model.

2. The method for training a speech synthesis model according to claim 1, characterized in that: The processing of the sample text and the sample audio by the language network to obtain the first audio feature includes: Based on the content of the sample text, extract a sample audio segment from the sample audio, wherein the sample audio segment includes the sound of all or part of the content in the sample text; The sample text and the sample audio segment are processed by the language network to obtain the first audio feature.

3. The method for training a speech synthesis model according to claim 2, characterized in that: The extracting a sample audio segment from the sample audio based on the content of the sample text comprises: Acquire secondary contents other than key contents from the sample text, wherein the key contents are used to represent the subject of the sample text; Based on the secondary content, the sample audio segment is extracted from the sample audio, and the content of the sample audio segment is the secondary content.

4. The method for training a speech synthesis model according to claim 2, characterized in that: The extracting a sample audio segment from the sample audio based on the content of the sample text comprises: Acquire emotional keywords from the content of the sample text, wherein the emotional keywords are used to represent the emotions conveyed by the content of the sample text; Based on the emotion keyword, the sample audio segment is extracted from the sample audio, and the sample audio segment includes the emotion keyword.

5. The method for training a speech synthesis model according to claim 2, characterized in that: The sample audio includes the voices of multiple objects; The extracting a sample audio segment from the sample audio based on the content of the sample text comprises: For any object among the multiple objects, based on the content of the sample text and the acoustic features of the object, a sample audio segment corresponding to the object is intercepted from the sample audio.

6. The method for training a speech synthesis model according to claim 1, characterized in that: The step of performing noise prediction on the sample audio to which reference noise is added based on the first audio feature through the diffusion network to obtain predicted noise includes: Acquire a second audio feature and a reference noise feature corresponding to the sample audio, where the reference noise feature is used to represent the reference noise; adding the reference noise feature to the second audio feature through the diffusion network to obtain a third audio feature; By using the diffusion network, taking the first audio feature as a condition, performing noise prediction on the third audio feature to obtain a predicted noise feature, wherein the predicted noise feature is used to represent the predicted noise; The determining a first training loss based on the reference noise and the predicted noise comprises: The first training loss is determined based on the reference noise feature and the predicted noise feature.

7. The method for training a speech synthesis model according to claim 1, characterized in that: The method further comprises: Determining a second training loss based on the first audio feature and a second audio feature corresponding to the sample audio, wherein the second audio feature includes a content feature and an acoustic feature in the sample audio, and the second training loss is used to represent a degree of difference between the first audio feature and the second audio feature; The updating of parameters of the language network and the diffusion network in the first speech synthesis model based on the first training loss includes: Based on the first training loss and the second training loss, parameters of the language network and the diffusion network in the first speech synthesis model are updated to obtain the second speech synthesis model.

8. The method for training a speech synthesis model according to claim 1, characterized in that: The method further comprises: Removing the predicted noise from the sample audio to which the reference noise is added to obtain a target audio; Based on the sample audio and the target audio, determine a third training loss, where the third training loss is used to represent the degree of difference between the sample audio and the target audio; The updating of parameters of the language network and the diffusion network in the first speech synthesis model based on the first training loss to obtain a second speech synthesis model includes: Based on the first training loss and the third training loss, parameters of the language network and the diffusion network in the first speech synthesis model are updated to obtain the second speech synthesis model.

9. The method for training a speech synthesis model according to claim 1, characterized in that: The processing of the sample text and the sample audio by the language network to obtain the first audio feature includes: Acquiring a phoneme sequence of the sample text and an acoustic feature of the sample audio; The phoneme sequence and the acoustic feature are processed through the language network to obtain the first audio feature.

10. The method for training a speech synthesis model according to claim 9, characterized in that: The process of obtaining the phoneme sequence of the sample text includes: Preprocessing the sample text, wherein the preprocessing includes at least one of word segmentation, denoising, text normalization, and polyphonetic pronunciation annotation; Based on the correspondence between characters and phonemes, the preprocessed sample text is converted into the phoneme sequence.

11. A speech synthesis method, characterized in that: The method comprises: Obtaining input text and prompt audio, where the prompt audio is used to indicate acoustic information used for speech synthesis; The input text and the prompt audio are processed by a language network in a second speech synthesis model to obtain intermediate audio features corresponding to the input text, wherein the second speech synthesis model is trained based on the training method for the speech synthesis model according to any one of claims 1 to 10; The intermediate audio features are denoised by the diffusion network in the second speech synthesis model to obtain target audio features, and output audio is generated based on the target audio features.

12. A training device for a speech synthesis model, characterized in that: The first speech synthesis model includes a language network and a diffusion network, wherein the language network is used to predict the audio features of the text, and the diffusion network is used to reconstruct the audio based on the audio features of the text; the device includes: A processing unit is configured to process the sample text and the sample audio through the language network to obtain a first audio feature, wherein the sample text and the sample audio have the same content, and the first audio feature includes a content feature of the sample text and an acoustic feature of the sample audio; A prediction unit is configured to perform noise prediction on the sample audio to which reference noise is added based on the first audio feature through the diffusion network to obtain predicted noise; A first determining unit is configured to determine a first training loss based on the reference noise and the predicted noise, where the first training loss is used to represent a degree of difference between the reference noise and the predicted noise; An updating unit is configured to update parameters of the language network and the diffusion network in the first speech synthesis model based on the first training loss to obtain a second speech synthesis model.

13. A speech synthesis device, characterized in that: The device comprises: An acquisition unit is configured to acquire input text and prompt audio, where the prompt audio is used to indicate acoustic information used for speech synthesis; A first processing unit is configured to process the input text and the prompt audio through a language network in a second speech synthesis model to obtain intermediate audio features corresponding to the input text, wherein the second speech synthesis model is trained based on the training method for the speech synthesis model according to any one of claims 1 to 10; The second processing unit is configured to perform denoising processing on the intermediate audio features through a diffusion network in the second speech synthesis model to obtain target audio features, and generate output audio based on the target audio features.

14. An electronic device, characterized in that: The electronic device comprises: one or more processors; a memory for storing program code executable by the processor; Wherein, the processor is configured to execute the program code to implement the training method of the speech synthesis model as described in any one of claims 1 to 10 or the speech synthesis method as described in claim 11.

15. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the training method of the speech synthesis model as described in any one of claims 1 to 10 or the speech synthesis method as described in claim 11.

16. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, it implements the training method of the speech synthesis model described in any one of claims 1 to 10 or the speech synthesis method described in claim 11.

Images