Deep learning model training method, speech synthesis method, and device

Abstract

The present disclosure provides a training method for a deep learning model, and relates to the field of artificial intelligence technology, and in particular to large models, large language models, generative models, deep learning, and speech processing technology. The specific implementation scheme is: determining the baseline speech features of the sample speech, and associating the baseline speech features with the rhythmic features of the sample speech; using the sample text corresponding to the sample speech to search the speech library to obtain the pronunciation expression features of the sample text; inputting the pronunciation expression features into the deep learning model to obtain the output speech features; determining the loss of the deep learning model based on the baseline speech features and the output speech features; and adjusting the parameters of the deep learning model based on the loss. The present disclosure also provides a speech synthesis method, another training method for a deep learning model, a device, an electronic device, and a storage medium.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to the field of large models, large language models, generative models, deep learning, and speech processing technology. More specifically, the present disclosure provides a training method for a deep learning model, a speech synthesis method, a training method for another deep learning model, an apparatus, an electronic device, a storage medium, and a computer program product. Background Art

[0002] Speech synthesis refers to the process of converting text into speech. After speech synthesis entered the era of deep learning, the effect of speech synthesis has been significantly improved. However, the current end-to-end speech synthesis model training has reached its upper limit on data utilization. For example, the amount of synthesis data is still limited to hundreds or thousands of hours. The addition of more training data does not bring any improvement to the model. Summary of the invention

[0003] The present disclosure provides a deep learning model training method, a speech synthesis method, another deep learning model training method, an apparatus, an electronic device, a storage medium, and a computer program product.

[0004] According to a first aspect, a training method for a deep learning model is provided, the method comprising: determining a baseline speech feature of a sample speech, the baseline speech feature being associated with a prosodic feature of the sample speech; searching a speech library using a sample text corresponding to the sample speech to obtain a pronunciation expression feature of the sample text; inputting the pronunciation expression feature into a deep learning model to obtain an output speech feature; determining a loss of the deep learning model based on the baseline speech feature and the output speech feature; and adjusting parameters of the deep learning model based on the loss.

[0005] According to a second aspect, a speech synthesis method is provided, which includes: using the text to be synthesized to retrieve pronunciation expression features of the text to be synthesized from a speech library; inputting the pronunciation expression features into a deep learning model to obtain output speech features, wherein the deep learning model is trained according to the training method of the above-mentioned deep learning model; and generating synthesized speech of the text to be synthesized based on the output speech features.

[0006] According to a third aspect, a training method for a deep learning model is provided, the method comprising: determining a sample speech and a sample text corresponding to the sample speech from a speech library; using the deep learning model to retrieve the pronunciation expression features of the sample text from other speech in the speech library except the sample speech based on the sample text; determining the loss of the deep learning model based on the speech features of the sample speech and the pronunciation expression features of the sample text; and adjusting the parameters of the deep learning model based on the loss.

[0007] According to a fourth aspect, a training device for a deep learning model is provided, the device comprising: a baseline speech feature determination module, used to determine the baseline speech feature of a sample speech, the baseline speech feature being associated with the prosodic feature of the sample speech; a first retrieval module, used to search a speech library using a sample text corresponding to the sample speech to obtain the pronunciation expression feature of the sample text; an output speech determination module, used to input the pronunciation expression feature into a deep learning model to obtain the output speech feature; a first loss determination module, used to determine the loss of the deep learning model based on the baseline speech feature and the output speech feature; and a first adjustment module, used to adjust the parameters of the deep learning model based on the loss.

[0008] According to the fifth aspect, a speech synthesis device is provided, which includes: a second retrieval module, used to use the text to be synthesized to retrieve the pronunciation expression features of the text to be synthesized from a speech library; a feature processing module, used to input the pronunciation expression features into a deep learning model to obtain output speech features, wherein the deep learning model is trained according to the training device of the above-mentioned deep learning model; and a speech synthesis module, used to generate synthesized speech of the text to be synthesized according to the output speech features.

[0009] According to the sixth aspect, a training device for a deep learning model is provided, which includes: a sample determination module for determining a sample speech and a sample text corresponding to the sample speech from a speech library; a third retrieval module for using the deep learning model to retrieve the pronunciation expression features of the sample text from other speech in the speech library except the sample speech based on the sample text; a second loss determination module for determining the loss of the deep learning model based on the speech features of the sample speech and the pronunciation expression features of the sample text; and a second adjustment module for adjusting the parameters of the deep learning model based on the loss.

[0010] According to the seventh aspect, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method provided according to the present disclosure.

[0011] According to an eighth aspect, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause a computer to execute the method provided according to the present disclosure.

[0012] According to a ninth aspect, a computer program product is provided, comprising a computer program, wherein the computer program is stored on at least one of a readable storage medium and an electronic device, and wherein the computer program implements the method provided according to the present disclosure when executed by a processor.

[0013] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0015] Figure 1 is a schematic diagram of an exemplary system architecture to which at least one of a training method for a deep learning model, a speech synthesis method, and a training method for another deep learning model can be applied according to an embodiment of the present disclosure;

[0016] Figure 2 is a flowchart of a method for training a deep learning model according to an embodiment of the present disclosure;

[0017] Figure 3 is a schematic diagram of a training method for a deep learning model according to an embodiment of the present disclosure;

[0018] Figure 4 is a schematic diagram of a method for determining a reference speech feature using a speech synthesis model according to an embodiment of the present disclosure;

[0019] Figure 5 is a schematic diagram of a method for retrieving pronunciation expression features using a speech retrieval model according to an embodiment of the present disclosure;

[0020] Figure 6 is a schematic diagram of a training method for a deep learning model according to an embodiment of the present disclosure;

[0021] Figure 7 is a flow chart of a speech synthesis method according to an embodiment of the present disclosure;

[0022] Figure 8 is a flowchart of a method for training a deep learning model according to another embodiment of the present disclosure;

[0023] Fig. 9 is a schematic diagram of a speech synthesis method according to an embodiment of the present disclosure;

[0024] Fig.10 is a block diagram of a training apparatus for a deep learning model according to an embodiment of the present disclosure;

[0025] Fig.11 is a block diagram of a speech synthesis device according to an embodiment of the present disclosure;

[0026] Fig.12 is a block diagram of a training apparatus for a deep learning model according to another embodiment of the present disclosure;

[0027] Fig.13 It is a block diagram of an electronic device according to at least one of a deep learning model training method, a speech synthesis method, and another deep learning model training method according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0028] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0029] With the rapid development of large models, speech synthesis technology has also entered the era of large models. Speech synthesis based on large models has put forward higher requirements for synthesis effects, such as requiring it to be comparable to or even better than real people.

[0030] Currently, a typical solution for large-model speech synthesis technology is to discretize speech features into multiple codebooks through pre-training of a large language model, and then model these discrete codebooks based on the large language model modeling technology. After expanding the data to the level of 10,000 hours, the synthesis effect has been greatly improved.

[0031] However, the current large-model speech synthesis technology has introduced some problems while using big data modeling to improve the synthesis effect. The main problems include: 1. The synthesis effect is unstable, and it is easy to have problems such as missing words, repetitions, mute sounds, and abnormal rhythm. 2. It is impossible to continuously increase data to improve the effect, because the training computing power of big data and large models is relatively large, and it is impossible to continuously improve the effect through continuous training. When new data arrives, retraining will bring about a large training cost. 3. In actual synthesis applications, when it is necessary to synthesize the voice of a specific person, it is necessary to provide the voice of a specific person as a prompt voice. The prompt voice may have high quality requirements. Therefore, the number of specific people's voices that can be used as prompt voices is limited, generally a few to dozens of sentences, and it is impossible to support the generation of synthesis effects with the characteristics of the speaker.

[0032] Therefore, the synthesis effect of the current large-model speech synthesis technology is unstable, especially in terms of rhythmic fluctuations, there is still a large gap compared with real people.

[0033] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0034] In the technical solution of the present disclosure, the user's authorization or consent is obtained before obtaining or collecting the user's personal information.

[0035] Figure 1 is a schematic diagram of an exemplary system architecture to which at least one of a training method for a deep learning model and a speech synthesis method can be applied according to an embodiment of the present disclosure. It should be noted that: Figure 1 What is shown is merely an example of a system architecture to which the embodiments of the present disclosure can be applied, in order to help those skilled in the art understand the technical content of the present disclosure, but it does not mean that the embodiments of the present disclosure cannot be used in other devices, systems, environments or scenarios.

[0036] like Figure 1 As shown, the system architecture 100 according to this embodiment may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0037] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Terminal devices 101, 102, 103 can be various electronic devices, including but not limited to smart phones, tablet computers, laptops, etc.

[0038] The training method of the deep learning model provided in the embodiment of the present disclosure can generally be executed by the server 105. Accordingly, the training device of the deep learning model provided in the embodiment of the present disclosure can generally be set in the server 105.

[0039] The speech synthesis method provided in the embodiment of the present disclosure can generally be executed by the terminal devices 101, 102, and 103. Accordingly, the training device of the deep learning model provided in the embodiment of the present disclosure can generally be set in the terminal devices 101, 102, and 103.

[0040] Figure 2 It is a flowchart of a method for training a deep learning model according to an embodiment of the present disclosure.

[0041] According to an embodiment of the present disclosure, by determining the baseline speech features of the sample speech, the pronunciation expression features of the sample text corresponding to the sample speech are retrieved from the speech library, the pronunciation expression features are input into the deep learning model, the output speech features are obtained, and the parameters of the large language model are adjusted based on the difference between the baseline speech features and the output speech features. The baseline speech features are used to reconstruct the speech, and the deep learning model is trained with the baseline speech features as supervision, so that the trained deep learning model can have the ability of speech synthesis.

[0042] According to the embodiments of the present disclosure, the benchmark speech features are associated with the prosodic features. Using the benchmark speech features as supervision to train a deep learning model can improve the prosodic fluctuation effect of the speech synthesis of the trained deep learning model, avoid prosodic abnormalities such as missing words, repetitions, and muteness in the synthesized speech, and improve the stability of speech synthesis.

[0043] like Figure 2 As shown, the deep learning model training method 200 includes operations S210 to S250.

[0044] In operation S210, a reference speech feature of the sample speech is determined, and the reference speech feature is associated with a prosodic feature of the sample speech.

[0045] The sample speech of this embodiment is used to train a deep learning model, and the deep learning model may be a model used for speech synthesis tasks. The reference speech features of the sample speech may be features used to reconstruct the sample speech.

[0046] The reference speech feature may be at least one of a prosodic feature and a Mel-spectrogram feature of the sample speech, the prosodic feature is determined based on the sample speech and the text of the sample speech, and the Mel-spectrogram feature of the sample speech is determined based on the prosodic feature. The prosodic feature of the sample speech can be converted into a Mel-spectrogram feature of the sample speech, and the Mel-spectrogram feature of the sample speech can be directly converted into the sample speech, so the prosodic feature and the Mel-spectrogram feature can be used to reconstruct the sample speech. The reference speech feature is associated with the prosodic feature, so the reference speech feature can also be used to reconstruct the sample speech.

[0047] The text of the sample speech is used as the sample text, and the prosodic features are determined by the timbre of the sample speech and the text features of the sample text. Therefore, the prosodic features include timbre information, text content information, and prosodic information expressed by pronunciation. In one example, the prosodic features of the sample speech can be used as benchmark speech features for the training of the deep learning model, which can reduce the prosodic gap between the synthesized speech of the deep learning model and the real person's speech, improve the simulation and authenticity of the synthesized speech, and make the synthesized speech closer to the real person.

[0048] In another example, the mel-spectrogram feature of the sample speech may be used as the reference speech feature. In another example, the combination of the prosody feature and the mel-spectrogram feature of the sample speech may be used as the reference speech feature.

[0049] In operation S220, the sample text corresponding to the sample speech is used to search the speech library to obtain the pronunciation expression features of the sample text.

[0050] For example, the speech library can be a database containing historical speech of the same speaker, or a database containing historical speech of multiple speakers. In one example, if the speech synthesis task is to synthesize speech of a specific person's timbre, the speech library can be a database containing the historical speech of the specific person.

[0051] For example, the speech database contains a large number of historical sentences. If the speech database contains the sample text itself, the pronunciation expression features of the sample text can be directly retrieved from the speech database. If the speech database does not contain the sample text itself, the pronunciation expression features of each word in the sample text can be retrieved from the speech database, and then the pronunciation expression features of each word are spliced ​​together to obtain the pronunciation expression features of the sample text. The above pronunciation expression features may include the rhythmic fluctuation of the pronunciation.

[0052] In operation S230, the pronunciation expression features are input into a deep learning model to obtain output speech features.

[0053] The deep learning model may be a large language model, and the above-mentioned pronunciation expression features may be used as prompt information of the large language model, prompting the large language model to output output speech features corresponding to the pronunciation expression mode. The output speech features are used for speech synthesis.

[0054] The embodiments of the present disclosure utilize a large language model to generate output speech features based on pronunciation expression features, and synthesize speech based on the output speech features. Compared with directly synthesizing speech based on text using a large language model, this can improve the accuracy of speech synthesis and the stability of rhythmic fluctuations.

[0055] In operation S240, a loss of a deep learning model is determined based on the reference speech feature and the output speech feature.

[0056] In operation S250, parameters of the deep learning model are adjusted according to the loss.

[0057] Since the reference speech features are features that can reconstruct the sample speech, the purpose of model training is to synthesize the sample speech based on the sample text. Therefore, the output speech features generated by the model should be consistent with the reference speech in order to reconstruct the sample speech based on the output speech features. Therefore, the training of the deep learning model can be supervised by the reference speech features, and the loss of the model training is the difference between the output speech features and the reference speech.

[0058] The difference such as cross entropy or mean square error between the output speech features and the reference speech can be calculated as the loss of the deep learning model, and the parameters of the deep learning model are adjusted according to the loss to obtain a trained deep learning model.

[0059] The input of the trained deep learning model is the speech expression feature, and the output is the output speech feature. Since the training goal of the model in the training stage is to make the output speech feature consistent with the reference speech feature, the trained deep learning model is equivalent to establishing a mapping relationship between the speech expression feature and the reference speech feature. Therefore, for a new text, after obtaining the pronunciation expression feature of the new text, the trained deep learning model can be used to generate the corresponding reference speech feature based on the pronunciation expression feature, which can be converted into mel-spectrogram feature based on the reference speech feature, and can be converted into the speech of the new text based on the mel-spectrogram feature to achieve speech synthesis.

[0060] Figure 3 It is a schematic diagram of a training method for a deep learning model according to an embodiment of the present disclosure.

[0061] like Figure 3 As shown, the reference speech features may be obtained using a pre-trained speech synthesis model 310. The pronunciation expression features may be obtained using a pre-trained speech retrieval model 320. The deep learning model 330 is trained using the pronunciation expression features and the reference speech features.

[0062] For example, the sample speech is input into the pre-trained speech synthesis model 310 to obtain the benchmark speech feature. The pre-trained speech synthesis model 310 may be an end-to-end speech synthesis model. The speech synthesis model encodes the sample speech and the text of the sample speech to obtain the timbre feature and the text feature respectively. The timbre feature and the text feature are encoded to obtain the prosodic feature. The prosodic feature is then encoded to obtain the mel-spectrogram feature. At least one of the prosodic feature and the mel-spectrogram feature in the above process may be used as the benchmark speech feature.

[0063] For example, the sample text is a text corresponding to the sample speech. The sample text is input into the pre-trained speech retrieval model 320 to obtain a pronunciation expression feature. The pre-trained speech retrieval model 320 can be trained based on a speech library. When the sample text is retrieved using the speech retrieval model 320, the cross-attention feature between the sample text and the speech features in the speech library can be calculated, and the mel-spectrogram feature of the sample text can be generated based on the cross-attention feature. At least one of the attention feature and the mel-spectrogram feature in the above process can be used as a pronunciation expression feature.

[0064] The pronunciation expression feature is input into the deep learning model 330 to obtain the output speech feature. The difference between the output speech feature and the reference speech feature is calculated to obtain the loss of the deep learning model, and the parameters of the deep learning model 330 are adjusted according to the loss. The deep learning model 330 is trained with the reference speech feature as supervision, and the trained deep learning model 330 can establish a mapping relationship between the pronunciation expression feature and the reference speech feature. That is, the input of the trained deep learning model is the pronunciation expression feature, and the output is the output speech feature consistent with the reference speech feature.

[0065] Combine the following Figure 4-5 The speech synthesis model and speech retrieval model provided by the present disclosure are explained respectively.

[0066] Figure 4 It is a schematic diagram of a method for determining a reference speech feature using a speech synthesis model according to an embodiment of the present disclosure.

[0067] like Figure 4 As shown, the speech synthesis model may include multiple encoders and a backbone decoder. The multiple encoders include a timbre encoder, a content encoder, a prosody encoder, and a variational autoencoder.

[0068] According to an embodiment of the present disclosure, determining the baseline speech feature includes determining the timbre feature of the sample speech; determining the text feature of the sample text; determining the prosodic feature of the sample text based on the timbre feature and the text feature; determining the mel-spectrogram feature of the sample speech based on the timbre feature, the text feature, and the prosodic feature; and using at least one of the prosodic feature and the mel-spectrogram feature as the baseline speech feature.

[0069] Inputting the sample speech into the timbre encoder can obtain the timbre features at the sentence level, which characterize the speaker characteristics. Inputting the sample text corresponding to the sample speech into the content encoder can obtain the text features at the phoneme granularity. Inputting the sample speech and sample text into the prosody encoder can obtain the prosody information at the phoneme or state granularity, and inputting the prosody information into the variational autoencoder to obtain the discretized prosody features. Among them, the variational autoencoder can be VQ-VAE (Vector Quantized-Variational AutoEncoder).

[0070] The timbre features, text features and rhythm features are input into the decoder to obtain a spectral feature sequence, namely a mel-spectrometric feature sequence. The decoder can be a Transformer Based Autoregressive Model.

[0071] The above-mentioned speech synthesis model can be trained based on a dataset of fine-marked samples. Fine-marked samples refer to sample speech that is annotated with corresponding sample text, and the sample speech quality is high, such as the sample speech is clear, without missing words, repetition, or mute. Correspondingly, the sample text quality is also high, without missing words, repetition, etc.

[0072] In the training phase of the speech synthesis model, the timbre features, text features, and prosody features output by multiple encoders are used as input conditions and input into the decoder of the backbone. The decoder of the backbone outputs a Mel-spectrogram feature sequence based on the input conditions. The actual Mel-spectrogram feature sequence of the sample speech is determined, and the difference between the actual Mel-spectrogram feature sequence and the Mel-spectrogram feature sequence output by the decoder is calculated as the loss of the speech synthesis model. The parameters of each encoder and decoder in the speech synthesis model are adjusted according to the loss to obtain a trained speech synthesis model.

[0073] The trained speech synthesis model can be used to synthesize speech in the application stage. For example, the text to be synthesized is input into the trained speech synthesis model to obtain a mel spectrum feature sequence. The mel spectrum feature sequence is converted into speech to obtain the synthesized speech of the text to be synthesized.

[0074] The above-mentioned trained speech synthesis model is also used to reconstruct spectral features during the application stage. For example, by inputting the speech whose spectral features are to be reconstructed into the trained speech synthesis model, intermediate prosodic features and mel-spectrogram feature sequences can be obtained. The embodiment of the present disclosure utilizes the ability of the speech synthesis model to reconstruct spectral features, and uses at least one of the prosodic features and mel-spectrogram feature sequences generated during the spectral feature reconstruction process as a reference speech feature for the training of a deep learning model.

[0075] The embodiments of the present disclosure use a timbre encoder and a prosody encoder to encode timbre features and prosody features respectively, which can achieve decoupling of speaker features and prosody features and improve the accuracy of reference speech features.

[0076] Figure 5 It is a schematic diagram of a method for retrieving pronunciation expression features using a speech retrieval model according to an embodiment of the present disclosure.

[0077] like Figure 5 As shown, the speech retrieval model can be a Transformer-based model. The speech retrieval model can be trained based on a speech library. The speech in the speech library can be coarsely labeled speech data. Coarsely labeled means that the speech quality requirements in the speech library are low, and unclear pronunciation, missing words, repeated words, etc. can be allowed. Correspondingly, the quality requirements of the text label are also low, and missing words, repetitions, etc. are allowed. For example, the coarsely labeled speech can be obtained from a public network.

[0078] The voices in the voice database can be from the same speaker or from different speakers. When the speech synthesis task is to synthesize the speech of a specific person, the speech in the voice database from the same specific person will make the speech synthesis effect better.

[0079] In the training phase of the speech retrieval model, each sentence in the speech library can be used to train the speech retrieval network. For example, for each sentence in the speech library (current sentence), during training, the current sentence is removed from the retrieval range of the speech library, the text of the current sentence is used as the query statement (Query), and the other sentences in the speech library except the current sentence are used as the key (Key) and value (Value). The text of the previous sentence (Query) is used to perform retrieval based on the key (Key) and value (Value), and the attention feature between the text feature of the current sentence text and the speech feature of the other sentences is obtained. Based on the attention feature, the mel-spectrogram feature can be obtained. The retrieval unit of the above retrieval process can be a phoneme or a syllable, and at least one of the above attention feature and mel-spectrogram feature can be used as a pronunciation expression feature.

[0080] In one example, the difference between the retrieved mel-spectrogram feature and the true mel-spectrogram feature of the current sentence can be calculated as the loss of the current sentence. The overall loss is determined based on the losses of all sentences in the speech library. Based on the overall loss, the parameters of the speech retrieval model can be adjusted to obtain a trained speech retrieval model.

[0081] The trained speech retrieval model can be used to retrieve the pronunciation expression features of the text to be synthesized during the application stage. For example, the text to be synthesized is input into the trained speech retrieval model, and the speech retrieval model retrieves the speech features in the speech library based on the text to be synthesized, and obtains the attention features between the text features of the text to be synthesized and the speech features in the speech library, and the mel-spectrogram features can be obtained based on the attention features. At least one of the attention features and the mel-spectrogram features can be used as the pronunciation expression features of the text to be synthesized for the training of the deep learning model.

[0082] During the training phase of the speech retrieval model of the embodiment of the present disclosure, for each sentence of speech, the sentence is removed from the retrieval scope during training, forcing the speech retrieval model to learn the ability to reconstruct the speech features of the current sentence from other sentences in the speech library, so that the model has the ability of speech retrieval.

[0083] During the application stage, the speech retrieval model of the embodiment of the present disclosure has speech retrieval capability, and therefore, for the text to be synthesized, the optimal pronunciation expression of the text to be synthesized can be retrieved from the speech library.

[0084] The embodiments of the present disclosure can customize the historical speech data of the corresponding speaker as a retrieval source according to the synthesis task requirements, thereby realizing speech synthesis for a specific person, realizing the user's personalized speech customization needs, and improving speech diversity.

[0085] Figure 6 It is a schematic diagram of a training method for a deep learning model according to an embodiment of the present disclosure.

[0086] like Figure 6 As shown, the deep learning model of this embodiment can be a large language model, and the sample data used for training the deep learning model can be an unlabeled speech training set. The unlabeled speech refers to speech without corresponding text labels.

[0087] For example, the unlabeled speech is input into the speech synthesis model, and the speech synthesis model outputs the reference speech features of the unlabeled speech. The speech synthesis model can be trained using the accurately labeled speech data, so the reference speech features output by the speech synthesis model have high accuracy and can accurately synthesize speech.

[0088] For example, the unlabeled speech is converted into text as the text to be synthesized, and the text to be synthesized is input into the speech retrieval model to obtain the pronunciation expression characteristics of the text. The above-mentioned speech retrieval model can be trained based on the coarsely labeled speech data, and can reconstruct the pronunciation expression of the text to be synthesized based on the speech features in the speech library.

[0089] For example, the pronunciation expression features output by the speech retrieval model are input into the deep learning model, and the deep learning model generates output speech features. The parameters of the deep learning model are adjusted based on the difference between the output speech features and the reference speech features, with the aim of making the deep learning model generate output speech features that are consistent with the reference speech features, so that the output speech features can be used to reconstruct speech, i.e., synthesize speech.

[0090] This deep learning model is trained using unlabeled speech and uses more accurate baseline speech features as supervision, which can improve model training results and reduce annotation costs.

[0091] The above-mentioned benchmark speech features are associated with prosodic features. Prosodic features are determined based on timbre features and text features. Therefore, the prosodic features contain the speaker's timbre information, speech content, and prosodic fluctuations. Therefore, using benchmark speech features for supervised training can make the synthesized speech effect of the deep learning model more stable in terms of rhythm.

[0092] The deep learning model disclosed in the present invention can be a large language model. Compared with the related art that uses a large language to directly synthesize speech based on text features, the embodiment of the present invention uses baseline speech features as supervision, determines output speech features based on pronunciation expression features, and synthesizes speech based on the output speech features, which can improve the speech synthesis effect and effectively improve problems such as missing words, skipped words, and abnormal rhythm in speech synthesis.

[0093] In addition, compared with the related art that uses a large language to directly synthesize speech based on text features and improves the synthesis effect through continuous training, the embodiments of the present disclosure use more accurate benchmark speech features as supervision, which can improve the ability of large models to reconstruct speech, reduce the number of training times, and reduce costs.

[0094] According to an embodiment of the present disclosure, the present disclosure also provides a speech synthesis method.

[0095] Figure 7 is a flowchart of a speech synthesis method according to an embodiment of the present disclosure.

[0096] like Figure 7 As shown, the speech synthesis method 700 includes operations S710 to S730.

[0097] In operation S710, the pronunciation expression features of the text to be synthesized are retrieved from a speech database using the text to be synthesized.

[0098] For example, if the speech synthesis task is to synthesize speech with the timbre of a specific person, the speech library can be a database containing the historical speech of the specific person. If the speech synthesis task does not require the synthesis of the timbre of a specific person, the speech library can be a database containing the historical speech of multiple people. In addition, in one example, if the speech synthesis task is to synthesize speech with the timbre of a specific person, but the historical speech data of the specific person is insufficient, the historical speech of other people can also be included. This is because the focus of retrieving the pronunciation expression features of the sample text in this step is to retrieve the rhythmic features of the pronunciation of the text. If the speech library contains the historical speech of the same person, the timbre and rhythmic effects of the subsequently synthesized speech of the specific person will be better.

[0099] In one example, a trained speech retrieval model may be used to retrieve the pronunciation expression features of the text to be synthesized from the speech library. The trained speech retrieval model is obtained by training based on the speech library.

[0100] By using the trained speech retrieval model to retrieve the text to be synthesized, a cross-attention feature between the text feature and the speech feature in the speech library can be obtained, and based on the cross-attention feature, a mel-spectrogram feature of the text to be synthesized can be obtained. At least one of the above attention feature and mel-spectrogram feature can be used as a speech expression feature.

[0101] In operation S720, the pronunciation expression features are input into a deep learning model to obtain output speech features.

[0102] The deep learning model can be trained according to the above-mentioned deep learning model training method. The training of the deep learning model is supervised by benchmark speech features. The benchmark speech features are associated with prosodic features and can be used to reconstruct speech.

[0103] By inputting the pronunciation expression features into the trained deep learning model, output speech features can be obtained. The output speech features are consistent with the reference speech features and can be used for speech synthesis.

[0104] The deep learning model may be a large language model, and the above-mentioned pronunciation expression features may be input into the large language model as prompt information to prompt the large language model to generate output speech features consistent with the benchmark speech features.

[0105] In operation S730, a synthesized speech of the text to be synthesized is generated according to the output speech feature.

[0106] According to an embodiment of the present disclosure, a mel-spectrogram feature of a text to be synthesized is determined based on output speech features; and a synthesized speech of the text to be synthesized is generated based on the mel-spectrogram feature of the text to be synthesized.

[0107] The output speech features are consistent with the reference speech features, and the reference speech features can be used for speech reconstruction. Therefore, the output speech features can be converted into Mel-spectrogram features, and the Mel-spectrogram features can be converted into speech, that is, the synthesized speech of the text to be synthesized.

[0108] The embodiments of the present disclosure retrieve the speech expression features of the text to be synthesized from the speech library, input the speech expression features into the deep learning model, and obtain output speech features. Since the deep learning model is trained with reference speech features as supervision, the output speech features of the trained deep learning model are consistent with the reference speech features. Using the output speech features to synthesize speech can improve the accuracy of speech synthesis.

[0109] According to an embodiment of the present disclosure, operation S710 includes determining text features of the text to be synthesized; and using the text features to retrieve speech features in the speech library to obtain pronunciation expression features. The retrieval includes: performing cross-attention processing on the text features and the speech features in the speech library to obtain attention features; determining the mel-spectrogram features of the text to be synthesized based on the attention features; and determining at least one of the attention features and the mel-spectrogram features of the text to be synthesized as the pronunciation expression feature.

[0110] For example, a trained speech retrieval model is used to perform retrieval based on text features, and a cross-attention process can be performed on the text features and the speech features in the speech library to obtain an attention feature. A mel-spectrogram feature is determined based on the attention feature. At least one of the attention feature and the mel-spectrogram feature can be used as a pronunciation expression feature.

[0111] The trained speech retrieval model is obtained by searching based on a speech library. Using the trained speech retrieval model for searching can retrieve the optimal pronunciation expression of the text to be synthesized from the speech library.

[0112] Combine the following Figure 8 The training of the speech retrieval model provided in the present disclosure is explained.

[0113] According to an embodiment of the present disclosure, the present disclosure also provides another training method for a deep learning model.

[0114] Figure 8 is a flowchart of a method for training a deep learning model according to another embodiment of the present disclosure.

[0115] like Figure 8 As shown, the speech synthesis method 800 includes operations S810 to S840.

[0116] In operation S810, a sample speech and a sample text corresponding to the sample speech are determined from a speech library.

[0117] The deep learning model of this embodiment may be a speech retrieval model. The speech library may include multiple sample speech, each sample speech may be a sentence-level speech. Each sample speech may have a corresponding sample text. The sample speech and sample text of this embodiment are used to train the speech retrieval model.

[0118] The speech library can be a database containing historical speech of the same speaker, or a database containing historical speech of multiple speakers. In one example, if the subsequent speech synthesis task is to synthesize speech of a specific person's timbre, the speech library can be a database containing the historical speech of the specific person.

[0119] In operation S820, a pronunciation expression feature of the sample text is retrieved from other voices in the voice database except the sample voice based on the sample text using a deep learning model.

[0120] According to an embodiment of the present disclosure, text features of a sample text are determined; cross-attention processing is performed on the text features and speech features of other speech in a speech library except the sample speech to obtain attention features; based on the attention features, mel-spectrogram features of the sample text are determined; and at least one of the attention features and the mel-spectrogram features is determined as a pronunciation expression feature.

[0121] Each sentence in the speech database can be used to train the speech retrieval network. For example, for each sentence in the speech database (current sentence), during training, the current sentence is removed from the retrieval range of the speech database, the text of the current sentence is used as the query statement, and the other sentences in the speech database except the current sentence are used as keys and values. The text of the previous sentence is used to perform cross-attention processing based on the keys and values ​​to obtain the attention features between the text features of the current sentence text and the speech features of other sentences, and the mel-spectrogram features can be obtained based on the attention features. The retrieval unit of the above retrieval process can be a phoneme or a syllable. At least one of the above attention features and mel-spectrogram features can be used as a pronunciation expression feature.

[0122] In operation S830, a loss of a deep learning model is determined according to the speech features of the sample speech and the pronunciation expression features of the sample text.

[0123] In operation S840, parameters of the deep learning model are adjusted according to the loss.

[0124] According to an embodiment of the present disclosure, the pronunciation expression feature is a mel-spectrogram feature. Determining the loss of the deep learning model includes: determining the mel-spectrogram feature of the sample speech; and determining the loss of the deep learning model according to the difference between the mel-spectrogram feature of the sample speech and the pronunciation expression feature.

[0125] For example, the difference between the retrieved mel-spectrogram feature and the true mel-spectrogram feature of the current sentence can be calculated as the loss of the current sentence. The overall loss can be determined based on the losses of all sentences in the speech library. Based on the overall loss, the parameters of the speech retrieval model can be adjusted to obtain a trained speech retrieval model.

[0126] During the training phase, the speech retrieval model of the embodiment of the present disclosure removes each speech sentence from the retrieval scope during training, forcing the speech retrieval model to learn the ability to reconstruct the speech features of the current sentence from other sentences in the speech library, so that the model has the ability of speech retrieval. The trained speech retrieval model can retrieve the optimal pronunciation expression of the text to be synthesized from the speech library.

[0127] Fig. 9 is a schematic diagram of a speech synthesis method according to an embodiment of the present disclosure.

[0128] like Fig. 9 As shown, this embodiment includes a trained speech retrieval model, a trained deep learning model, and a trained speech synthesis model.

[0129] The text to be synthesized is input into the trained speech retrieval model, and the speech retrieval model retrieves the speech expression features of the text to be synthesized from the speech library. The pronunciation expression features are input into the trained deep learning model to obtain the output speech features. The deep learning model is trained with the baseline speech features as supervision, so the output speech features are consistent with the baseline speech features.

[0130] The reference speech features are associated with the prosodic features, and the mel-spectrogram features can be obtained based on the prosodic features, and the mel-spectrogram features can be used for speech reconstruction. The output speech features consistent with the reference speech features are input into the trained speech synthesis model to obtain the mel-spectrogram features. The mel-spectrogram features can be converted into speech, that is, the synthesized speech of the text to be synthesized.

[0131] The embodiment of the present disclosure retrieves the pronunciation expression features of the text to be synthesized by using a trained speech retrieval model, inputs the pronunciation expression features into a trained deep learning model, and obtains output speech features. Since the output speech features are consistent with the reference speech features and can be used for speech reconstruction, the trained speech synthesis model can be used to reconstruct speech based on the output speech features, that is, synthesize the speech of the text to be synthesized. Since the reference speech features are associated with the prosodic features, the synthesized speech has a high stability in terms of prosody.

[0132] According to an embodiment of the present disclosure, the present disclosure also provides a training device for a deep learning model, a speech synthesis device, and another training device for a deep learning model.

[0133] Fig.10 is a block diagram of a training device for a deep learning model according to an embodiment of the present disclosure.

[0134] like Fig.10 As shown, the training device 1000 of the deep learning model includes a reference speech feature determination module 1001, a first retrieval module 1002, an output speech determination module 1003, a first loss determination module 1004 and a first adjustment module 1005.

[0135] The reference speech feature determination module 1001 is used to determine the reference speech feature of the sample speech, where the reference speech feature is associated with the prosodic feature of the sample speech.

[0136] The first search module 1002 is used to search the speech database using the sample text corresponding to the sample speech to obtain the pronunciation expression features of the sample text.

[0137] The output speech determination module 1003 is used to input the pronunciation expression features into the deep learning model to obtain the output speech features.

[0138] The first loss determination module 1004 is used to determine the loss of the deep learning model based on the baseline speech features and the output speech features.

[0139] The first adjustment module 1005 is used to adjust the parameters of the deep learning model according to the loss.

[0140] Fig.11 is a block diagram of a speech synthesis apparatus according to an embodiment of the present disclosure.

[0141] like Fig.11 As shown, the speech synthesis device 1100 includes a second retrieval module 1101 , a feature processing module 1102 and a speech synthesis module 1103 .

[0142] The second retrieval module 1101 is used to retrieve the pronunciation expression features of the text to be synthesized from the speech database using the text to be synthesized.

[0143] The feature processing module 1102 is used to input the pronunciation expression features into the deep learning model to obtain output speech features, wherein the deep learning model is trained according to the training device of the above-mentioned deep learning model.

[0144] The speech synthesis module 1103 is used to generate synthesized speech of the text to be synthesized according to the output speech features.

[0145] Fig.12 is a block diagram of a training device for a deep learning model according to another embodiment of the present disclosure.

[0146] like Fig.12 As shown, the speech synthesis device 1200 includes a sample determination module 1201 , a third retrieval module 1202 , a second loss determination module 1203 , and a second adjustment module 1204 .

[0147] The sample determination module 1201 is used to determine a sample speech and a sample text corresponding to the sample speech from a speech library.

[0148] The third retrieval module 1202 is used to retrieve the pronunciation expression features of the sample text from other voices in the voice library except the sample voice based on the sample text by using the deep learning model.

[0149] The second loss determination module 1203 is used to determine the loss of the deep learning model according to the speech features of the sample speech and the pronunciation expression features of the sample text.

[0150] The second adjustment module 1204 is used to adjust the parameters of the deep learning model according to the loss.

[0151] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0152] Fig.13 A schematic block diagram of an example electronic device 1300 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0153] like Fig.13 As shown, the device 1300 includes a computing unit 1301, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1302 or a computer program loaded from a storage unit 1308 into a random access memory (RAM) 1303. In the RAM 1303, various programs and data required for the operation of the device 1300 can also be stored. The computing unit 1301, the ROM 1302, and the RAM 1303 are connected to each other via a bus 1304. An input / output (I / O) interface 1305 is also connected to the bus 1304.

[0154] A number of components in the device 1300 are connected to the I / O interface 1305, including: an input unit 1306, such as a keyboard, a mouse, etc.; an output unit 1307, such as various types of displays, speakers, etc.; a storage unit 1308, such as a disk, an optical disk, etc.; and a communication unit 1309, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1309 allows the device 1300 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0155] The computing unit 1301 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1301 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1301 performs the various methods and processes described above, such as at least one of a training method for a deep learning model, a speech synthesis method, and a training method for another deep learning model. For example, in some embodiments, at least one of a training method for a deep learning model, a speech synthesis method, and a training method for another deep learning model may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 1308. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 1300 via the ROM 1302 and / or the communication unit 1309. When the computer program is loaded into RAM 1303 and executed by the computing unit 1301, one or more steps of at least one of the training method for the deep learning model, the speech synthesis method, and the training method for another deep learning model described above may be performed. Alternatively, in other embodiments, the computing unit 1301 may be configured to execute at least one of the training method for the deep learning model, the speech synthesis method, and the training method for another deep learning model in any other appropriate manner (e.g., by means of firmware).

[0156] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0157] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0158] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0159] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0160] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0161] A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.

[0162] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0163] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A training method for a deep learning model, comprising: Determining a reference speech feature of the sample speech, wherein the reference speech feature is associated with a prosodic feature of the sample speech; Using a pre-trained speech retrieval model, searching a speech library based on a sample text corresponding to the sample speech to obtain a pronunciation expression feature of the sample text; Inputting the pronunciation expression features into a deep learning model to obtain output speech features; Determining the loss of the deep learning model according to the baseline speech feature and the output speech feature; as well as According to the loss, adjusting the parameters of the deep learning model; The method of using a pre-trained speech retrieval model to search a speech library based on a sample text corresponding to the sample speech to obtain the pronunciation expression features of the sample text includes: In response to the speech database including the sample speech, retrieving the pronunciation expression features of the sample text from the speech database; In response to the speech database not including the sample speech, the pronunciation expression feature of each word in the sample text is retrieved from the speech database, and the pronunciation expression features of the individual words are concatenated to obtain the pronunciation expression feature of the sample text.

2. The method according to claim 1, wherein: The step of determining the reference speech features of the sample speech comprises: Determine at least one of the prosodic feature and the mel-spectrogram feature of the sample speech as the reference speech feature; The prosodic feature is determined based on the sample speech and the sample text, and the mel-spectrogram feature of the sample speech is determined based on the prosodic feature.

3. The method according to claim 2, further comprising: Determining the timbre characteristics of the sample speech; Determining text features of the sample text; as well as The prosodic features of the sample text are determined according to the timbre features and the text features.

4. The method according to claim 3, further comprising: The mel-spectrogram feature of the sample speech is determined according to the timbre feature, the text feature, and the prosody feature.

5. The method according to claim 1, wherein: The method of using the pre-trained speech retrieval model to search the speech library based on the sample text corresponding to the sample speech to obtain the pronunciation expression features of the sample text includes: Determining text features of the sample text; and Using a pre-trained speech retrieval model, the speech features in the speech library are retrieved based on the text features to obtain the pronunciation expression features.

6. The method according to claim 5, wherein: The method of using the pre-trained speech retrieval model to retrieve speech features in the speech library based on the text features to obtain the pronunciation expression features includes: Performing cross-attention processing on the text features and the speech features in the speech library to obtain attention features; Determining a mel spectrum feature of the sample text according to the attention feature; and At least one of the attention feature and the mel-spectrogram feature of the sample text is determined as the pronunciation expression feature.

7. The method according to claim 1, wherein: The deep learning model is a large language model; the inputting the pronunciation expression feature into the deep learning model to obtain the output speech feature comprises: The pronunciation expression feature is input into the large language model as prompt information to prompt the large language model to generate the output speech feature.

8. A speech synthesis method, comprising: Retrieving the pronunciation expression features of the text to be synthesized from a speech library using the text to be synthesized; Inputting the pronunciation expression features into a deep learning model to obtain output speech features, wherein the deep learning model is trained by the method according to any one of claims 1 to 7; and Generate synthesized speech of the text to be synthesized according to the output speech features.

9. The method according to claim 8, wherein: The method of using the text to be synthesized to retrieve the pronunciation expression features of the text to be synthesized from the speech library includes: Determining text features of the text to be synthesized; and The text features are used to retrieve the speech features in the speech database to obtain the pronunciation expression features.

10. The method according to claim 9, wherein: The using of the text features to retrieve the speech features in the speech database to obtain the pronunciation expression features comprises: Performing cross-attention processing on the text features and the speech features in the speech library to obtain attention features; Determining the mel spectrum feature of the text to be synthesized according to the attention feature; and At least one of the attention feature and the mel-spectrogram feature of the text to be synthesized is determined as the pronunciation expression feature.

11. The method according to claim 8, wherein: The deep learning model is a large language model; the inputting the pronunciation expression feature into the deep learning model to obtain the output speech feature comprises: The pronunciation expression feature is input into the large language model as prompt information to prompt the large language model to generate the output speech feature.

12. The method according to claim 8, wherein: The step of generating the synthesized speech of the text to be synthesized according to the output speech feature comprises: Determining the Mel-spectrogram features of the text to be synthesized according to the output speech features; and Generate synthesized speech of the text to be synthesized according to the mel-spectrogram features of the text to be synthesized.

13. A method for training a deep learning model, comprising: Determine a sample speech and a sample text corresponding to the sample speech from a speech library; Using a deep learning model based on the sample text, retrieving the pronunciation expression features of the sample text from other voices in the voice library except the sample voice; Determining the loss of the deep learning model according to the speech features of the sample speech and the pronunciation expression features of the sample text; and According to the loss, adjusting the parameters of the deep learning model; The trained deep learning model is used to perform voice retrieval on the synthesized text, including: In response to the speech database including the speech of the text to be synthesized, retrieving the pronunciation expression features of the text to be synthesized from the speech database; In response to the speech database not including the speech of the text to be synthesized, the pronunciation expression features of each word of the text to be synthesized are retrieved from the speech database, and the pronunciation expression features of each word are concatenated to obtain the pronunciation expression features of the text to be synthesized.

14. The method according to claim 13, wherein: The method of using the deep learning model to retrieve the pronunciation expression features of the sample text from other voices in the voice library except the sample voice based on the sample text includes: Determining text features of the sample text; Performing cross-attention processing on the text features and speech features of other speech in the speech library except the sample speech to obtain attention features; Determining a mel spectrum feature of the sample text according to the attention feature; and At least one of the attention feature and the mel-spectrogram feature is determined as the pronunciation expression feature.

15. The method according to claim 14, wherein: The pronunciation expression feature is the mel spectrum feature; The determining the loss of the deep learning model according to the speech features of the sample speech and the pronunciation expression features of the sample text includes: Determining Mel-spectrogram features of the sample speech; as well as The loss of the deep learning model is determined according to the difference between the Mel-spectrogram feature of the sample speech and the pronunciation expression feature.

16. A training device for a deep learning model, comprising: A reference speech feature determination module, used to determine a reference speech feature of a sample speech, wherein the reference speech feature is associated with a prosodic feature of the sample speech; A first retrieval module is used to use a pre-trained speech retrieval model to search a speech library based on a sample text corresponding to the sample speech to obtain a pronunciation expression feature of the sample text; An output speech determination module, used for inputting the pronunciation expression features into a deep learning model to obtain output speech features; A first loss determination module, configured to determine the loss of the deep learning model according to the reference speech feature and the output speech feature; and A first adjustment module, used to adjust the parameters of the deep learning model according to the loss; Among them, the first retrieval module is used to retrieve the pronunciation expression features of the sample text from the voice database in response to the voice database including the sample voice; in response to the voice database not including the sample voice, retrieve the pronunciation expression features of each word of the sample text from the voice database, and splice the pronunciation expression features of each word to obtain the pronunciation expression features of the sample text.

17. A speech synthesis device, comprising: A second retrieval module is used to retrieve the pronunciation expression features of the text to be synthesized from the speech library using the text to be synthesized; a feature processing module, configured to input the pronunciation expression feature into a deep learning model to obtain output speech features, wherein the deep learning model is trained by the apparatus according to claim 16; and The speech synthesis module is used to generate the synthesized speech of the text to be synthesized according to the output speech features.

18. A training device for a deep learning model, comprising: A sample determination module, used to determine a sample speech and a sample text corresponding to the sample speech from a speech library; A third retrieval module is used to retrieve the pronunciation expression features of the sample text from other voices in the voice library except the sample voice based on the sample text using a deep learning model; A second loss determination module, configured to determine the loss of the deep learning model according to the speech features of the sample speech and the pronunciation expression features of the sample text; and A second adjustment module, used to adjust the parameters of the deep learning model according to the loss; Among them, the third retrieval module is used to perform voice retrieval on the text to be synthesized, specifically for retrieving the pronunciation expression features of the text to be synthesized from the voice library in response to the voice library including the voice of the text to be synthesized; in response to the voice library not including the voice of the text to be synthesized, retrieving the pronunciation expression features of each word of the text to be synthesized from the voice library, splicing the pronunciation expression features of each word to obtain the pronunciation expression features of the text to be synthesized.

19. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 15.

20. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 15.

21. A computer program product, comprising a computer program, wherein the computer program is stored on at least one of a readable storage medium and an electronic device, and when the computer program is executed by a processor, the method according to any one of claims 1 to 15 is implemented.

Images