Text-based speech synthesis method, device, equipment and storage medium

Abstract

The present disclosure provides a text-based speech synthesis method and device, equipment and storage medium, relates to the field of artificial intelligence, in particular to the field of text-to-speech, and can be applied to the fields of intelligent customer service, voice reading assistant, voice content creation, voice education and training. The specific implementation scheme is as follows: obtaining a target text and a reference audio; wherein the target text is a text to be synthesized into speech, and the reference audio is an audio of a preset timbre; performing feature extraction processing on the target text to obtain a text feature vector of the target text; wherein the text feature vector represents semantic features and prosodic features of the text; obtaining a target audio corresponding to the target text according to the text feature vector and the reference audio; wherein the timbre of the target audio is consistent with the timbre of the reference audio. The present disclosure does not need to maintain the method of converting text into phonemes, and improves the synthesis accuracy of speech by extracting semantic features and prosodic features.

Description

Technical Field

[0001] The present disclosure relates to the field of text-to-speech in the field of artificial intelligence, and can be applied to intelligent customer service, voice reading assistants, voice content creation, voice education and training, and more particularly to a text-based speech synthesis method, apparatus, device, and storage medium. Background Art

[0002] Speech synthesis technology can convert text into natural and fluent speech. It is used in various fields and brings convenience to people's lives.

[0003] With the development of artificial intelligence, people have higher and higher requirements for speech synthesis technology. Speech synthesis technology needs to be able to quickly generate accurate and rhythmic speech to improve users' listening experience. Summary of the Invention

[0004] The present disclosure provides a text-based speech synthesis method, apparatus, device, and storage medium.

[0005] According to a first aspect of the present disclosure, a text-based speech synthesis method is provided, comprising:

[0006] Obtaining a target text and a reference audio; wherein the target text is a text to be synthesized into speech, and the reference audio is an audio with a preset timbre;

[0007] Performing feature extraction on the target text to obtain a text feature vector of the target text; wherein the text feature vector represents the semantic features and prosodic features of the text;

[0008] A target audio corresponding to the target text is obtained based on the text feature vector and the reference audio; wherein the timbre of the target audio is consistent with the timbre of the reference audio.

[0009] According to a second aspect of the present disclosure, there is provided a text-based speech synthesis apparatus, comprising:

[0010] A data acquisition unit, configured to acquire a target text and a reference audio; wherein the target text is a text to be synthesized into speech, and the reference audio is an audio with a preset timbre;

[0011] A feature extraction unit, configured to perform feature extraction processing on the target text to obtain a text feature vector of the target text; wherein the text feature vector represents the semantic features and prosodic features of the text;

[0012] An audio obtaining unit is used to obtain a target audio corresponding to the target text based on the text feature vector and the reference audio; wherein the timbre of the target audio is consistent with the timbre of the reference audio.

[0013] According to a third aspect of the present disclosure, there is provided an electronic device, including:

[0014] at least one processor; and

[0015] a memory communicatively coupled to the at least one processor;

[0016] The memory stores instructions that can be executed by the at least one processor. The instructions are executed by the at least one processor to enable the at least one processor to perform the method described in the first aspect of the present disclosure.

[0017] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute the method according to the first aspect of the present disclosure.

[0018] According to a fifth aspect of the present disclosure, a computer program product is provided, comprising a computer program, which implements the steps of the method described in the first aspect of the present disclosure when executed by a processor.

[0019] The technology disclosed herein improves the efficiency and accuracy of text-to-speech conversion and enhances user experience.

[0020] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings are provided to facilitate a better understanding of the present invention and do not constitute a limitation of the present disclosure.

[0022] Figure 1 is a flowchart of a text-based speech synthesis method provided according to an embodiment of the present disclosure;

[0023] Figure 2 is a flowchart of a text-based speech synthesis method provided according to an embodiment of the present disclosure;

[0024] Figure 3 is a network diagram for generating audio feature vectors according to an embodiment of the present disclosure;

[0025] Figure 4 is a flowchart of a text-based speech synthesis method provided according to an embodiment of the present disclosure;

[0026] Figure 5 Schematic diagram of the training process of the speech synthesis model provided according to an embodiment of the present disclosure;

[0027] Figure 6 is a flowchart of a text-based speech synthesis method provided according to an embodiment of the present disclosure;

[0028] Figure 7 1 is a schematic diagram of an architecture of a text-based speech synthesis network provided according to an embodiment of the present disclosure;

[0029] Figure 8 is a structural block diagram of a text-based speech synthesis device provided according to an embodiment of the present disclosure;

[0030] Figure 9 is a structural block diagram of a text-based speech synthesis device provided according to an embodiment of the present disclosure;

[0031] Figure 10 is a block diagram of an electronic device for implementing the text-based speech synthesis method of an embodiment of the present disclosure;

[0032] Figure 11 The block diagram is a block diagram of an electronic device for implementing the text-based speech synthesis method according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0033] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding. These details should be considered as merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0034] Speech synthesis technology is used in a variety of fields. For example, it can be used as a voice reading assistant to provide users with voice reading services and assist visually impaired people in reading; it can be used to create voice content, such as voice advertisements, voice broadcasts, and voice novels; it can be used for voice education and training, such as voice courses and voice answering systems, etc., providing personalized and interactive education and training services through speech synthesis technology.

[0035] Most current speech synthesis technologies first generate phonemes from text and then encode them. However, different languages ​​and dialects use different methods to convert text into phonemes, requiring a separate method for each language, which increases the difficulty of speech synthesis. Furthermore, current synthesized speech only utilizes semantic information from the text, lacking prosodic information such as pauses and intonation. Consequently, the speech synthesis effect fails to meet user needs.

[0036] The present disclosure provides a text-based speech synthesis method, apparatus, device, and storage medium, which are applied to the text-to-speech field in the field of artificial intelligence. The method can be applied to intelligent customer service, voice reading assistants, voice content creation, voice education and training, and other fields to improve the efficiency and accuracy of speech synthesis.

[0037] It should be noted that the model in this embodiment is not targeted at a specific user and cannot reflect the personal information of a specific user. It should be noted that the data in this embodiment comes from a public data set.

[0038] In the technical solutions disclosed herein, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0039] In order to enable readers to have a deeper understanding of the implementation principle of this disclosure, the following Figures 1-11 The embodiment is further refined.

[0040] Figure 1 FIG. 1 is a flow chart of a text-based speech synthesis method according to an embodiment of the present disclosure, which can be performed by a text-based speech synthesis device. Figure 1 As shown, the method includes the following steps:

[0041] S101. Obtain target text and reference audio; wherein the target text is the text of the speech to be synthesized, and the reference audio is the audio with a preset timbre.

[0042] For example, a reference audio file is pre-set and contains preset speech content. In this embodiment, the speech content of the reference audio file is not specifically limited. The timbre of the reference audio file is a preset timbre, which is the timbre that the user wishes the target text to have when converted into audio. The target text is the text to be synthesized into speech; that is, the target text is to be converted into speech, and the converted speech serves as the target audio file. In other words, the target text and the reference audio file are acquired to obtain audio corresponding to the target text with the timbre of the reference audio file.

[0043] The user can upload the target text and reference audio in real time, or retrieve the target text and reference audio from a preset storage location.

[0044] S102 : Perform feature extraction on the target text to obtain a text feature vector of the target text; wherein the text feature vector represents the semantic features and prosodic features of the text.

[0045] For example, a text encoder is pre-set. The text encoder can be a pre-trained neural network model that can be used to extract feature vectors from text data. The target text is input into the pre-set text encoder, which performs feature extraction on the target text and determines the extracted feature vector as the text feature vector.

[0046] In this embodiment, semantic features and prosodic features can be extracted from the target text. That is, the text feature vector can represent the semantic features and prosodic features of the target text. The semantic features can represent the semantic information of the text content in the target text, and the prosodic features can represent information such as pauses and tone of the target text.

[0047] S103. Obtain target audio corresponding to the target text according to the text feature vector and the reference audio; wherein the timbre of the target audio is consistent with the timbre of the reference audio.

[0048] Exemplarily, based on the text feature vector and the reference audio, a target audio having the timbre of the reference audio is obtained. That is, the timbre of the target audio is consistent with the timbre of the reference audio, and the speech content in the target audio is consistent with the text content of the target text.

[0049] The target audio's speech content and prosody can be determined based on the text feature vector. The target audio's timbre can also be determined based on the reference audio. For example, spectral features can be extracted from the reference audio to obtain timbre-related vector information. The target audio is then generated by combining the target audio's speech content, prosody, and timbre.

[0050] The text feature vector and the timbre-related feature vector can be input together into a pre-trained encoder, which can be a text encoder, which is different from the text encoder for extracting the text feature vector. After the text feature vector and the timbre-related feature vector are input into the pre-trained text encoder, the text encoder encodes and maps the input data. For example, the text feature vector and the timbre feature vector are first fused, and the fused feature vector is used as text_encoder_y, and text_encoder_y is the text encoding. The text encoding text_encoder_y is projected and mapped by a projector, and the data obtained after projection is then passed through Flow (generative model) to obtain the hidden layer data of the target audio. The hidden layer data is decoded by a preset audio decoder to obtain an audio waveform signal, and the target audio can be obtained through the audio waveform signal.

[0051] In the disclosed embodiment, target text and reference audio are obtained, and feature extraction is performed directly on the target text, eliminating the text-to-phoneme process. This eliminates the need to maintain a set of text-to-phoneme methods for each language, reduces the difficulty of speech synthesis, and improves the efficiency of speech synthesis. By extracting semantic and prosodic features, the problem of information loss caused by removing phonemes is compensated, the accuracy of speech synthesis is improved, and the user's listening experience is enhanced. The reference audio is audio with a preset timbre. Based on the text feature vector and the reference audio, the target audio with the preset timbre is obtained to meet the actual needs of the user and enhance the user experience.

[0052] Figure 2 A flowchart of a text-based speech synthesis method provided in an embodiment of the present disclosure is provided.

[0053] In this embodiment, the target audio corresponding to the target text is obtained based on the text feature vector and the reference audio, including: inputting the reference audio into a preset Mel encoder to obtain a Mel spectrogram of the reference audio; wherein the Mel spectrogram represents the timbre and spectral information of the audio; and obtaining the target audio corresponding to the target text based on the text feature vector and the Mel spectrogram.

[0054] This embodiment is based on the above embodiment. Figure 2 As shown, the method includes the following steps:

[0055] S201. Obtain target text and reference audio; wherein the target text is the text of the speech to be synthesized, and the reference audio is the audio with a preset timbre.

[0056] For example, this step may refer to the above-mentioned step S101 and will not be described in detail.

[0057] S202 : Perform feature extraction on the target text to obtain a text feature vector of the target text; wherein the text feature vector represents the semantic features and prosodic features of the text.

[0058] For example, this step may refer to the above-mentioned step S102 and will not be described in detail.

[0059] S203: Input the reference audio into a preset Mel encoder to obtain a Mel spectrogram of the reference audio; wherein the Mel spectrogram represents the timbre and spectrum information of the audio.

[0060] For example, a mel encoder is pre-trained, and the reference audio is input into the preset mel encoder. The mel encoder can output a mel spectrogram of the reference audio, which can represent the spectral information of the audio, thereby obtaining a target audio containing the timbre of the reference audio. In this embodiment, the encoding process of the mel encoder is not specifically limited. By adopting the mel encoder, the information of the reference audio can be fully utilized, thereby optimizing the zero-shot effect, achieving direct cloning of 10s audio, and improving the efficiency and accuracy of speech synthesis.

[0061] The Mel encoder can be trained in advance, and the loss function used for training the Mel encoder is recorded as loss_mel. Loss_mel can be a Mel Spectrogram Loss function, which is used to measure the difference between the generated Mel spectrogram and the Mel spectrogram of the real audio.

[0062] S204: Obtain target audio corresponding to the target text according to the text feature vector and the mel-spectrogram.

[0063] For example, a mel-spectrogram is also a feature vector. Combining the text feature vector and the mel-spectrogram can generate the target audio corresponding to the target text. Specifically, the text feature vector and the mel-spectrogram are input into a preset text encoder. After encoding, projection, and decoding, the audio waveform signal of the target audio is generated.

[0064] In this embodiment, the mel-spectrogram can make full use of the information of the reference audio during the prediction process. By combining the mel-spectrogram and the text feature vector, various information such as semantics, rhythm, and timbre can be taken into account when generating the target audio, thereby improving the accuracy of speech synthesis and enhancing the user's listening experience of the audio.

[0065] In this embodiment, the target audio corresponding to the target text is obtained based on the text feature vector and the mel-spectrogram, including: obtaining a reference text corresponding to the reference audio; wherein the text content in the reference text is consistent with the speech content in the reference audio; determining the audio feature vector of the target text based on the target text, the reference text, and the reference audio; wherein the audio feature vector represents the semantic features of the audio; and obtaining the target audio corresponding to the target text based on the text feature vector, the mel-spectrogram, and the audio feature vector.

[0066] Specifically, a reference text corresponding to the reference audio is pre-set, and the text content in the reference text is consistent with the voice content in the reference audio. For example, the reference audio is the voice of "hello", and the reference text is the text of "hello".

[0067] Feature extraction is performed on the target text, reference text, and reference audio to obtain an audio feature vector of the target audio corresponding to the target text. The audio feature vector can represent the semantic features of the audio. Different semantic features can correspond to different pronunciations, that is, it can represent information such as the pronunciation of the target audio. Feature extraction can be performed on the target text, reference text, and reference audio separately, or the target text, reference text, and reference audio can be combined first and then feature extraction can be performed. For example, text feature extraction can be performed on both the target text and the reference text, audio feature extraction can be performed on the reference audio, and then the two extracted feature vectors can be fused into an audio feature vector.

[0068] Text feature vectors, mel-spectrograms, and audio feature vectors are all vector data. Text feature vectors can be denoted as Text Vecs, mel-spectrograms as Mel Encoder Vecs, and audio feature vectors as SpeechTokens Vecs. Text Vecs, Mel Encoder Vecs, and Speech Tokens Vecs are all input into a preset text encoder, which outputs the target audio waveform corresponding to the target text.

[0069] The beneficial effect of this setting is that it combines the three features of text feature vector, Mel-spectrogram, and audio feature vector to synthesize speech, avoids information omission, improves the synthesis accuracy of speech, and enhances user experience.

[0070] In this embodiment, the audio feature vector of the target text is determined based on the target text, the reference text, and the reference audio, including: inputting the reference audio into a preset speech encoder to obtain the audio encoding of the reference audio; wherein the audio encoding is a discrete identifier that represents the pronunciation of the audio; and obtaining the audio feature vector of the target text based on the audio encoding of the reference audio, the target text, and the reference text.

[0071] Specifically, a speech encoder is pre-configured, and the reference audio is input into the speech encoder. The speech encoder can be referred to as a Speech Tokenizer. The speech encoder determines the audio encoding corresponding to the reference audio. The audio encoding is a discrete identifier, which can be in the form of a number, and different numbers can represent different pronunciations. For example, the audio encoding is [1, 3, 5, 4, 8], and the identifier is used as the ID, where ID 1 represents the pronunciation of the pinyin "wo" or the pronunciation of the pinyin "w."

[0072] Feature extraction is performed on the target text and reference text to obtain text-related features. Combining the audio codec with the text-related features yields an audio feature vector. For example, the audio codec corresponding to the target audio can be generated based on the text features and the audio codec of the reference audio. The audio codec of the reference audio can then be converted into a vector to obtain the audio feature vector.

[0073] The beneficial effect of this setting is that the audio encoder can generate discrete audio coding representations, and the discretized coding features can be applied to more accurate encoder models, such as GPT (Generative Pre-trained Transformer), to improve the efficiency and accuracy of audio generation.

[0074] In this embodiment, the reference audio is input into a preset voice encoder to obtain audio encoding of the reference audio, including: based on the preset voice encoder, feature extraction processing is performed on the reference audio to obtain a feature vector of the reference audio; based on a preset quantizer, the feature vector of the reference audio is converted into audio encoding.

[0075] Specifically, the speech encoder may include CN-HuBERT (CN-Hidden-Unit BERT, Chinese language processing model) and a quantizer. CN-HuBERT is a self-supervised learning model that learns speech features by using Hu (Hidden Units) for prediction without requiring a large amount of labeled data.

[0076] First, input the reference audio into CN-HuBERT, which can be the audio waveform of the reference audio. CN-HuBERT extracts feature vectors from the reference audio, which represent the pronunciation information of the reference audio. The extracted feature vectors are then passed through a quantizer, which converts them into discrete audio codes.

[0077] A linear layer can be connected after CN-HuBERT to convert the reference audio into a feature vector. The linear layer can then be used to perform dimensionality conversion on the audio feature vector. For example, a target dimension can be preset, and finally a quantizer can be used to obtain the audio encoding.

[0078] This setup has the beneficial effect of encoding first, followed by quantization, to obtain a discrete representation of the audio signal, facilitating subsequent synthesis of the target audio encoding. Furthermore, by employing CN-HuBERT, supervised semantic information can be incorporated into model training, further improving speech synthesis accuracy.

[0079] In this embodiment, an audio feature vector of the target text is obtained based on the audio encoding of the reference audio, the target text, and the reference text, including: inputting the target text and the reference text into a preset text encoder to obtain a fused text vector; wherein the fused text vector represents the target text and the reference text; and determining the audio feature vector of the target text based on the fused text vector and the audio encoding of the reference audio.

[0080] Specifically, a text encoder is pre-set, and the target text and reference text are input into the text encoder. The text encoder can be denoted as a Text Tokenizer. The target text and reference text are subjected to feature extraction processing by the text encoder, and the extracted feature vector is determined as a fused text vector. In other words, the fused text vector can represent the target text and the reference text. The fused text vector and the audio encoding of the reference audio are combined to obtain an audio feature vector of the target text. For example, an autoregressive model can be used to combine the fused text vector and the audio encoding of the reference audio.

[0081] The beneficial effect of this setting is that the target text and the reference text are first combined, and then combined with the audio encoding of the reference audio to obtain the audio feature vector of the target audio corresponding to the target text, thereby realizing the use of reference audio and reference text to assist in obtaining the audio feature vector of the target audio and improve the accuracy of speech synthesis.

[0082] In this embodiment, the audio feature vector of the target text is determined based on the fused text vector and the audio encoding of the reference audio, including: determining the audio encoding of the target text based on the fused text vector and the audio encoding of the reference audio; and converting the audio encoding of the target text into the audio feature vector of the target text based on a preset quantizer.

[0083] Specifically, the autoregressive model is first used to combine the fused text vector and the audio encoding of the reference audio. The output data is still a discrete encoding. The output encoding is the audio encoding of the target text, that is, the audio encoding of the target audio corresponding to the target text, which can represent the pronunciation of the target audio.

[0084] Then, the target text's audio code is converted into an audio feature vector using the opposite processing method of the Speech Tokenizer. That is, the target text's audio code is first converted into a feature vector using a quantizer, and then the decoder decodes the converted feature vector to obtain the audio feature vector Speech Tokens Vecs.

[0085] The beneficial effect of this setting is that the discrete audio encoding of the reference audio is converted into the discrete audio encoding of the target text, so as to determine the pronunciation of the target audio corresponding to the target text, and the discrete audio encoding can be applied in the GPT model, thereby improving the determination accuracy and efficiency of the target audio.

[0086] Figure 3 Schematic diagram of the network for generating audio feature vectors. Figure 3 In this example, the reference audio is input into the audio encoder SpeechTokenizer, while the target text and reference text are input into the text encoder Text Tokenizer. The reference audio passes through CN-HuBERT and the vector quantizer Vector Quantizer to obtain the audio encoding of the reference audio, Ref Tokens. The target text and reference text pass through the Text Tokenizer to obtain a fused text vector. The Ref Tokens and fused text vector pass through the autoregressive model Autoregressive Transformer to obtain the audio encoding Speech Tokens corresponding to the target text. The Speech Tokens are then quantized and decoded to obtain the final audio feature vector Speech Tokens Vecs.

[0087] In the disclosed embodiment, target text and reference audio are obtained, and feature extraction is performed directly on the target text, eliminating the text-to-phoneme process. This eliminates the need to maintain a set of text-to-phoneme methods for each language, reduces the difficulty of speech synthesis, and improves the efficiency of speech synthesis. By extracting semantic and prosodic features, the problem of information loss caused by removing phonemes is compensated, the accuracy of speech synthesis is improved, and the user's listening experience is enhanced. The reference audio is audio with a preset timbre. Based on the text feature vector and the reference audio, the target audio with the preset timbre is obtained to meet the actual needs of the user and enhance the user experience.

[0088] Figure 4 A flowchart of a text-based speech synthesis method provided in an embodiment of the present disclosure is provided.

[0089] This embodiment is based on the above embodiment. Figure 4 As shown, the method includes the following steps:

[0090] S401. Obtain target text and reference audio; wherein the target text is the text of the speech to be synthesized, and the reference audio is the audio with a preset timbre.

[0091] For example, this step may refer to the above-mentioned step S101 and will not be described in detail.

[0092] S402 : Perform feature extraction on the target text to obtain a text feature vector of the target text; wherein the text feature vector represents the semantic features and prosodic features of the text.

[0093] For example, this step can refer to the above step S102 and will not be repeated here.

[0094] S403: Input the target text, reference text, reference audio, and text feature vector of the target text into a preset speech synthesis model to obtain the output target audio.

[0095] Exemplarily, a speech synthesis model is pre-built and trained. In this embodiment, the speech synthesis model can be a GPT model, and the model structure of VITS (Variational Inference Text-to-Speech, a deep learning model for text to speech) can be adopted. The speech synthesis model can contain a Mel encoder for determining the Mel spectrogram of the reference audio. The Mel encoder can also be independent of the speech synthesis model, that is, the Mel spectrogram of the reference audio is first determined by the Mel encoder, and then the target text, reference text, reference audio, text feature vector of the target text, and Mel spectrogram of the reference audio are input into the preset speech synthesis model to obtain the output target audio.

[0096] In the speech synthesis model, the audio feature vector of the target text can be determined based on the target text, reference text, and reference audio. The target audio corresponding to the target text can then be obtained based on the text feature vector, mel-spectrogram, and audio feature vector.

[0097] In this embodiment, the method also includes: obtaining a data set to be trained; wherein the data set to be trained includes a text to be trained, an audio to be trained, a reference text, a reference audio, a text feature vector of the text to be trained, and a mel-spectrogram of the reference audio; inputting the text feature vector of the text to be trained, the mel-spectrogram of the reference audio, the text to be trained, the reference text, and the reference audio into an initial model to obtain a prediction vector of the text to be trained; wherein the prediction vector represents the predicted audio corresponding to the text to be trained; according to the audio to be trained and the prediction vector, based on a preset loss function, the initial model is trained to obtain a trained speech synthesis model; wherein the speech synthesis model is used to determine the audio feature vector of the target text according to the target text, the reference text, and the reference audio, and obtain the target audio corresponding to the target text according to the text feature vector of the target text, the mel-spectrogram of the reference audio, and the audio feature vector.

[0098] Specifically, the speech synthesis model is pre-trained. During model training, a training dataset is obtained. The training dataset may include the target text, the target audio, reference text, and reference audio. If the speech synthesis model cannot generate text feature vectors for the target text and mel-spectrograms for the reference audio, the training dataset also includes text feature vectors for the target text and mel-spectrograms for the reference audio. The reference text used during model training and the reference text used during model application can be the same or different. The reference text corresponds to the reference audio, and the training text corresponds to the training audio.

[0099] The text feature vector of the text to be trained, the mel-spectrogram of the reference audio, the text to be trained, the reference text, and the reference audio are input into a preset initial model. The model undergoes encoding and projection processing to obtain a prediction vector for the text to be trained. The prediction vector represents the predicted audio corresponding to the text to be trained. That is, the feature vector of the audio corresponding to the text to be trained is predicted. The audio to be trained is the correct audio corresponding to the text to be trained. Based on the audio to be trained and the prediction vector, the initial model is trained based on a preset loss function. After continuous backpropagation, a trained speech synthesis model is obtained. For example, the preset loss function may include a discriminator loss function, a generator loss function, a KL (Kullback-Leibler) divergence loss function, etc.

[0100] The beneficial effect of this setting is that the model is supervised and trained using a preset loss function, thereby improving the training accuracy and thus improving the speech synthesis accuracy.

[0101] In this embodiment, the initial model is trained based on the preset loss function according to the audio to be trained and the prediction vector to obtain a trained speech synthesis model, including: performing feature extraction processing on the audio to be trained to obtain the vector to be trained corresponding to the audio to be trained; according to the vector to be trained and the prediction vector, the initial model is trained based on the preset loss function to obtain a trained speech synthesis model; wherein the preset loss function includes at least a quantizer loss function or a sequence alignment loss function.

[0102] Specifically, the training audio corresponding to the training text is obtained in advance. The training audio is then processed through a posterior encoder for feature extraction to obtain a training vector corresponding to the training audio. The training vector is then decoded through a decoder to output an audio signal. The training audio before being input into the posterior encoder can be compared with the output audio. If the two are consistent, the training vector is correct and the model can be trained based on the training vector.

[0103] According to the preset loss function, the training vector and the prediction vector are calculated, and whether the model training is completed is determined according to the loss value. For example, if the loss value is less than the preset loss threshold, it is determined that a speech synthesis model has been trained. In this embodiment, the preset loss function includes at least a quantizer loss function or a sequence alignment loss function. For example, the loss function may include loss_disc (Discriminator Loss, Discriminator Loss Function), loss_gen (GeneratorLoss, Generator Loss Function), loss_fm (Feature Matching Loss, Feature Matching Loss Function), loss_mel (Mel Spectrogram Loss, Mel Spectrogram Loss Function), loss_kl (KL Divergence Loss, KL Divergence Loss Function), on this basis, it also includes loss_quantizer (Quantizer Loss Function) and loss_ctc (Sequence Alignment Loss Function).

[0104] Among them, loss_fm can ensure that the generated audio is closer to the real audio in the feature space; loss_mel can be used to measure the difference between the mel spectrogram of the generated audio and the mel spectrogram of the real audio; loss_kl can be used to measure the KL divergence between the posterior distribution and the prior distribution of the latent variable, promoting the consistency of the generated latent variable with the expected prior distribution, which is the loss function acting on the Flow (generative model) method; loss_quantizer calculates the difference between the input feature and its corresponding quantized vector, using the mean square error loss function; loss_ctc is the loss function for the audio-to-text task, which uses the maximum likelihood loss function. The final loss function can be expressed as:

[0105] L=loss_VITS+loss_quantizer+loss_ctc;

[0106] loss_VITS=[loss_disc,loss_gen,loss_fm,loss_mel,loss_kl];

[0107] Among them, L is the final loss function of the model.

[0108] The beneficial effect of this setting is that the model is trained through the vector to be trained, the loss of the quantizer and the loss of sequence alignment are increased, and the training accuracy of the model is improved.

[0109] Figure 5 Schematic diagram of the training process of the speech synthesis model. Figure 5The waveform in represents the audio signal of the training audio. During model training, the text feature vector, mel-spectrogram, and audio feature vector are input into the Text Encoder. After projection, a vector is generated, serving as the prediction vector z1. The training vector z2 is then passed through the generative model to calculate the generating function, which can be represented by function f, resulting in f(z2). z1 and f(z2) are aligned. The alignment process determines the deviation between the training vector and the prediction vector, thereby confirming whether the model can output the correct prediction vector. The model can also detect the presence of noise in the synthesized speech and perform denoising. For example, noise can be accurately extracted from the Text Encoder output data, preventing it from affecting the speech synthesis quality. During model training, the output data of the Text Encoder and the aligned output data can be obtained. The aligned output data is then denoised to obtain the noise value, thereby improving speech synthesis accuracy and enhancing the effectiveness of model training.

[0110] In the disclosed embodiment, target text and reference audio are obtained, and feature extraction is performed directly on the target text, eliminating the text-to-phoneme process. This eliminates the need to maintain a set of text-to-phoneme methods for each language, reduces the difficulty of speech synthesis, and improves the efficiency of speech synthesis. By extracting semantic and prosodic features, the problem of information loss caused by removing phonemes is compensated, the accuracy of speech synthesis is improved, and the user's listening experience is enhanced. The reference audio is audio with a preset timbre. Based on the text feature vector and the reference audio, the target audio with the preset timbre is obtained to meet the actual needs of the user and enhance the user experience.

[0111] Figure 6 A flowchart of a text-based speech synthesis method provided in an embodiment of the present disclosure is provided.

[0112] In this embodiment, feature extraction processing is performed on the target text to obtain a text feature vector of the target text, including: extracting semantic features and prosodic features from the target text respectively; and obtaining a text feature vector of the target text based on the semantic features and prosodic features of the target text.

[0113] This embodiment is based on the above embodiment. Figure 6 As shown, the method includes the following steps:

[0114] S601. Obtain target text and reference audio; wherein the target text is the text of the speech to be synthesized, and the reference audio is the audio with a preset timbre.

[0115] For example, this step may refer to the above-mentioned step S101 and will not be described in detail.

[0116] S602: Extract semantic features and prosodic features from the target text.

[0117] Exemplarily, feature extraction is performed on the target text. In this embodiment, semantic features and prosodic features of the target text need to be extracted. For example, the target text can be input into a feature extraction network, which can be in the form of a text encoder, and can output the semantic features and prosodic features of the target text.

[0118] In this embodiment, semantic features and prosodic features are extracted from the target text respectively, including: extracting semantic features from the target text based on a preset first encoder; wherein the preset first encoder is a first pre-trained model for capturing semantic information in the text; and extracting prosodic features from the target text based on a preset second encoder; wherein the preset second encoder is a second pre-trained model for capturing prosodic information in the text.

[0119] Specifically, the preset feature extraction network may include two text encoders, namely a first encoder and a second encoder, and the first encoder and the second encoder are both pre-trained models, wherein the model structures of the first encoder and the second encoder may be different, the first encoder is a first pre-trained model, and the second encoder is a second pre-trained model. The first encoder is used to capture semantic information in the text, and the second encoder is used to capture prosodic information in the text. In this embodiment, the model structures of the first encoder and the second encoder are not specifically limited. For example, the first encoder is a BERT (Bidirectional Encoder Representations from Transformers) model, and the second encoder is a CLAP (Contrastive Language-Audio Pretraining) model.

[0120] In other words, the target text is fed into the first and second encoders, respectively. The first encoder extracts semantic features from the target text, while the second encoder extracts prosodic features from the target text. The feature extraction network containing the first and second encoders is a text encoder that combines semantic and prosodic features, and is denoted as a Joint Text Encoder.

[0121] The beneficial effect of this setting is that, in the process of obtaining text encoding, the Bert and Clap methods are used simultaneously to capture the semantic information and prosodic information in the text, avoid missing information in the text, and improve the accuracy of speech synthesis.

[0122] S603: Obtain a text feature vector of the target text according to the semantic features and prosodic features of the target text.

[0123] Exemplarily, the semantic features and prosodic features of the target text are combined to obtain a text feature vector of the target text, that is, the two feature vectors are combined into one feature vector.

[0124] In this embodiment, a text encoder that combines semantic and prosodic features is proposed to simultaneously capture the semantic information and prosodic information in the target text, making up for the problem of information reduction caused by removing phonemes, ensuring the comprehensiveness of information, and improving the accuracy of speech synthesis.

[0125] In this embodiment, obtaining the text feature vector of the target text according to the semantic features and prosodic features of the target text includes: adding the semantic features and prosodic features of the target text to obtain the text feature vector of the target text.

[0126] Specifically, the semantic features and prosodic features can be combined by adding the semantic features and prosodic features of the target text. The result of the addition is the text feature vector of the target text. For example, the semantic features and prosodic features are vectors in matrix form, and the text feature vector is obtained by adding the two matrices.

[0127] The beneficial effect of this arrangement is that, through addition processing, feature vectors containing semantics and rhythm can be quickly obtained, thereby improving the efficiency and accuracy of speech synthesis.

[0128] S604: Obtain target audio corresponding to the target text based on the text feature vector and the reference audio; wherein the timbre of the target audio is consistent with the timbre of the reference audio.

[0129] For example, this step may refer to the above-mentioned step S103 and will not be described in detail.

[0130] In the disclosed embodiment, target text and reference audio are obtained, and feature extraction is performed directly on the target text, eliminating the text-to-phoneme process. This eliminates the need to maintain a set of text-to-phoneme methods for each language, reduces the difficulty of speech synthesis, and improves the efficiency of speech synthesis. By extracting semantic and prosodic features, the problem of information loss caused by removing phonemes is compensated, the accuracy of speech synthesis is improved, and the user's listening experience is enhanced. The reference audio is audio with a preset timbre. Based on the text feature vector and the reference audio, the target audio with the preset timbre is obtained to meet the actual needs of the user and enhance the user experience.

[0131] Figure 7 A schematic diagram of the architecture of a text-based speech synthesis network provided in an embodiment of the present disclosure.

[0132] This embodiment is based on the above embodiment. Figure 7 As shown, in this network architecture:

[0133] The target text is input into the joint text encoder, and the semantic features are obtained through the first encoder BERT in the joint text encoder, and the prosodic features are obtained through the second encoder CLAP in the joint text encoder. The semantic features and prosodic features are combined to obtain the text feature vector.

[0134] Input the reference audio into the Mel encoder to obtain the Mel spectrogram of the reference audio.

[0135] An audio feature vector is obtained according to the reference audio, the target text, and the reference text.

[0136] In this embodiment, the order of steps for determining the text feature vector, the mel-spectrogram, and the audio feature vector is not specifically limited.

[0137] To determine the audio feature vector, the reference audio can be input into the audio encoder Speech Tokenizer. This is then passed through CN-HuBERT and Vector Quantizer to obtain the audio encoding Ref Tokens for the reference audio. The target text and reference text are then input into the Text Tokenizer to obtain a fused feature vector. The Ref Tokens and fused feature vector are passed through an autoregressive model to obtain the audio encoding Speech Tokens corresponding to the target text. The Speech Tokens are then quantized and decoded to obtain the audio feature vector.

[0138] The text feature vector, mel-spectrogram, and audio feature vector are collectively input into a text encoder, TextEncoder, which can adopt the VITS structure. After projection and generative modeling, the latent representation of the audio is obtained, and the target audio is output through a decoder. After passing through the text encoder, the output data of the text encoder can be denoised to obtain noise-free data. When this noise-free data and the projected output data are jointly input into the generative model, the generative model can combine the noise-free data to generate clean, noise-free target audio. For example, the location of noise features can be determined from the projected output data based on the noise-free data, thereby performing denoising and improving the accuracy of target audio generation.

[0139] Figure 8 This is a structural block diagram of a text-based speech synthesis device provided by an embodiment of the present disclosure. For ease of explanation, only the parts related to the embodiment of the present disclosure are shown. Figure 8The text-based speech synthesis device 800 includes: a data acquisition unit 801, a feature extraction unit 802 and an audio acquisition unit 803.

[0140] The data acquisition unit 801 is used to acquire a target text and a reference audio; wherein the target text is a text to be synthesized into speech, and the reference audio is an audio with a preset timbre;

[0141] A feature extraction unit 802 is configured to perform feature extraction processing on the target text to obtain a text feature vector of the target text; wherein the text feature vector represents the semantic features and prosodic features of the text;

[0142] The audio obtaining unit 803 is configured to obtain a target audio corresponding to the target text based on the text feature vector and the reference audio; wherein the timbre of the target audio is consistent with the timbre of the reference audio.

[0143] Figure 9 A structural block diagram of a text-based speech synthesis device provided in an embodiment of the present disclosure is shown in FIG. Figure 9 As shown, the text-based speech synthesis device 900 includes a data acquisition unit 901, a feature extraction unit 902 and an audio acquisition unit 903, wherein the audio acquisition unit 903 includes a Mel encoding module 9031 and an audio synthesis module 9032.

[0144] Mel encoding module 9031, configured to input the reference audio into a preset Mel encoder to obtain a Mel spectrogram of the reference audio; wherein the Mel spectrogram represents the timbre and spectrum information of the audio;

[0145] The audio synthesis module 9032 is used to obtain the target audio corresponding to the target text according to the text feature vector and the mel-spectrogram.

[0146] In one example, the audio synthesis module 9032 includes:

[0147] A text acquisition submodule, configured to acquire a reference text corresponding to the reference audio; wherein the text content in the reference text is consistent with the speech content in the reference audio;

[0148] A vector determination submodule, configured to determine an audio feature vector of the target text based on the target text, the reference text, and the reference audio; wherein the audio feature vector represents a semantic feature of the audio;

[0149] The audio synthesis submodule is used to obtain the target audio corresponding to the target text based on the text feature vector, the Mel-spectrogram, and the audio feature vector.

[0150] In one example, the vector determination submodule is specifically used to:

[0151] Inputting the reference audio into a preset speech encoder to obtain an audio code of the reference audio; wherein the audio code is a discrete identifier representing the pronunciation of the audio;

[0152] An audio feature vector of the target text is obtained according to the audio encoding of the reference audio, the target text, and the reference text.

[0153] In one example, the vector determination submodule is specifically used to:

[0154] Performing feature extraction processing on the reference audio based on a preset speech encoder to obtain a feature vector of the reference audio;

[0155] Based on a preset quantizer, the feature vector of the reference audio is converted into the audio code.

[0156] In one example, the vector determination submodule is specifically used to:

[0157] Inputting the target text and the reference text into a preset text encoder to obtain a fused text vector; wherein the fused text vector represents the target text and the reference text;

[0158] An audio feature vector of the target text is determined according to the fused text vector and the audio encoding of the reference audio.

[0159] In one example, the vector determination submodule is specifically used to:

[0160] Determining the audio encoding of the target text according to the fused text vector and the audio encoding of the reference audio;

[0161] Based on a preset quantizer, the audio code of the target text is converted into an audio feature vector of the target text.

[0162] In one example, it also includes:

[0163] A data set acquisition unit is used to acquire a data set to be trained; wherein the data set to be trained includes a text to be trained, an audio to be trained, a reference text, a reference audio, a text feature vector of the text to be trained, and a mel spectrogram of the reference audio;

[0164] A data input unit, configured to input the text feature vector of the text to be trained, the mel-spectrogram of the reference audio, the text to be trained, the reference text, and the reference audio into an initial model to obtain a prediction vector of the text to be trained; wherein the prediction vector represents the predicted audio corresponding to the text to be trained;

[0165] A model training unit is used to train the initial model based on the audio to be trained and the prediction vector based on a preset loss function to obtain a trained speech synthesis model; wherein the speech synthesis model is used to determine the audio feature vector of the target text based on the target text, the reference text, and the reference audio, and obtain the target audio corresponding to the target text based on the text feature vector of the target text, the mel-spectrogram of the reference audio, and the audio feature vector.

[0166] In one example, the model training unit includes:

[0167] An audio training module is used to perform feature extraction processing on the audio to be trained to obtain a training vector corresponding to the audio to be trained;

[0168] A loss determination module is used to train the initial model based on the vector to be trained and the predicted vector based on a preset loss function to obtain a trained speech synthesis model; wherein the preset loss function includes at least a quantizer loss function or a sequence alignment loss function.

[0169] In one example, the feature extraction unit 902 includes:

[0170] A feature extraction module, configured to extract semantic features and prosodic features from the target text;

[0171] The feature acquisition module is used to obtain a text feature vector of the target text according to the semantic features and prosodic features of the target text.

[0172] In one example, the feature extraction module includes:

[0173] A semantic extraction submodule, configured to extract the semantic features from the target text based on a preset first encoder; wherein the preset first encoder is a first pre-trained model for capturing semantic information in the text;

[0174] The prosody extraction submodule is used to extract the prosody features from the target text based on a preset second encoder; wherein the preset second encoder is a second pre-trained model used to capture the prosody information in the text.

[0175] In one example, the feature acquisition module includes:

[0176] The feature fusion submodule is used to perform addition processing on the semantic features and prosodic features of the target text to obtain a text feature vector of the target text.

[0177] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device.

[0178] Figure 10 A structural block diagram of an electronic device provided in an embodiment of the present disclosure, such as Figure 10 As shown, the electronic device 1000 includes: at least one processor 1002; and a memory 1001 communicatively connected to the at least one processor 1002; wherein the memory stores instructions that can be executed by the at least one processor 1002, and the instructions are executed by the at least one processor 1002 to enable the at least one processor 1002 to execute the text-based speech synthesis method disclosed herein.

[0179] The electronic device 1000 further includes a receiver 1003 and a transmitter 1004. The receiver 1003 is used to receive instructions and data sent by other devices, and the transmitter 1004 is used to send instructions and data to external devices.

[0180] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0181] According to an embodiment of the present disclosure, the present disclosure also provides a computer program product, which includes: a computer program, the computer program is stored in a readable storage medium, at least one processor of an electronic device can read the computer program from the readable storage medium, and at least one processor executes the computer program so that the electronic device executes the solution provided by any of the above embodiments.

[0182] Figure 11 A schematic block diagram of an example electronic device 1100 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are provided as examples only and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0183] like Figure 11As shown, the device 1100 includes a computing unit 1101, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1102 or a computer program loaded from a storage unit 1108 into a random access memory (RAM) 1103. Various programs and data required for the operation of the device 1100 can also be stored in the RAM 1103. The computing unit 1101, the ROM 1102, and the RAM 1103 are connected to each other via a bus 1104. An input / output (I / O) interface 1105 is also connected to the bus 1104.

[0184] Various components in device 1100 are connected to I / O interface 1105, including an input unit 1106, such as a keyboard and mouse; an output unit 1107, such as various types of displays and speakers; a storage unit 1108, such as a magnetic disk and optical disk; and a communication unit 1109, such as a network card, a modem, a wireless communication transceiver, etc. Communication unit 1109 allows device 1100 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0185] The computing unit 1101 can be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1101 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1101 performs the various methods and processes described above, such as the text-based speech synthesis method. For example, in some embodiments, the text-based speech synthesis method can be implemented as a computer software program that is tangibly contained in a machine-readable medium, such as a storage unit 1108. In some embodiments, part or all of the computer program can be loaded and / or installed on the device 1100 via the ROM 1102 and / or the communication unit 1109. When the computer program is loaded into the RAM 1103 and executed by the computing unit 1101, one or more steps of the text-based speech synthesis method described above can be performed. Alternatively, in other embodiments, the computing unit 1101 may be configured to execute the text-based speech synthesis method in any other appropriate manner (eg, by means of firmware).

[0186] Various embodiments of the systems and techniques described above can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0187] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0188] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0189] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0190] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0191] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact via a communication network. This client-server relationship is established by computer programs running on the respective computers, establishing a client-server relationship. The server may be a cloud server, also known as a cloud computing server or cloud host, a host product within the cloud computing service ecosystem that addresses the management difficulties and limited scalability of traditional physical hosts and VPS services ("Virtual Private Servers" or simply "VPS"). The server may also be a server in a distributed system or a server integrated with blockchain.

[0192] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not a limitation herein.

[0193] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. A text-based speech synthesis method, comprising: Obtaining a target text and a reference audio; wherein the target text is a text to be synthesized into speech, and the reference audio is an audio with a preset timbre; Performing feature extraction on the target text to obtain a text feature vector of the target text; wherein the text feature vector represents the semantic features and prosodic features of the text; Obtaining a reference text corresponding to the reference audio; wherein the text content in the reference text is consistent with the voice content in the reference audio; Determining an audio feature vector of the target text based on the target text, the reference text, and the reference audio; wherein the audio feature vector represents a semantic feature of the audio; A target audio corresponding to the target text is obtained based on the text feature vector, the Mel-spectrogram of the reference audio, and the audio feature vector; wherein the Mel-spectrogram represents the timbre and spectral information of the audio, the Mel-spectrogram is obtained by inputting the reference audio into a preset Mel encoder, and the timbre of the target audio is consistent with the timbre of the reference audio.

2. The method according to claim 1, wherein The step of determining the audio feature vector of the target text according to the target text, the reference text, and the reference audio includes: Inputting the reference audio into a preset speech encoder to obtain an audio code of the reference audio; wherein the audio code is a discrete identifier representing the pronunciation of the audio; An audio feature vector of the target text is obtained according to the audio encoding of the reference audio, the target text, and the reference text.

3. The method according to claim 2, wherein: Inputting the reference audio into a preset speech encoder to obtain audio encoding of the reference audio includes: Performing feature extraction processing on the reference audio based on a preset speech encoder to obtain a feature vector of the reference audio; Based on a preset quantizer, the feature vector of the reference audio is converted into the audio code.

4. The method according to claim 2, wherein: The step of obtaining an audio feature vector of the target text according to the audio encoding of the reference audio, the target text, and the reference text includes: Inputting the target text and the reference text into a preset text encoder to obtain a fused text vector; wherein the fused text vector represents the target text and the reference text; An audio feature vector of the target text is determined according to the fused text vector and the audio encoding of the reference audio.

5. The method according to claim 4, wherein The step of determining the audio feature vector of the target text according to the fused text vector and the audio encoding of the reference audio includes: Determining the audio encoding of the target text according to the fused text vector and the audio encoding of the reference audio; Based on a preset quantizer, the audio code of the target text is converted into an audio feature vector of the target text.

6. The method according to claim 1, further comprising: Obtaining a data set to be trained; wherein the data set to be trained includes a text to be trained, an audio to be trained, a reference text, a reference audio, a text feature vector of the text to be trained, and a mel spectrogram of the reference audio; Inputting the text feature vector of the text to be trained, the mel spectrogram of the reference audio, the text to be trained, the reference text, and the reference audio into the initial model to obtain a prediction vector of the text to be trained; wherein the prediction vector represents the predicted audio corresponding to the text to be trained; According to the audio to be trained and the prediction vector, the initial model is trained based on a preset loss function to obtain a trained speech synthesis model; wherein the speech synthesis model is used to determine the audio feature vector of the target text based on the target text, the reference text, and the reference audio, and obtain the target audio corresponding to the target text based on the text feature vector of the target text, the mel-spectrogram of the reference audio, and the audio feature vector.

7. The method according to claim 6, wherein: The step of training the initial model based on the audio to be trained and the prediction vector and a preset loss function to obtain a trained speech synthesis model includes: Performing feature extraction processing on the audio to be trained to obtain a vector to be trained corresponding to the audio to be trained; According to the vector to be trained and the predicted vector, based on a preset loss function, the initial model is trained to obtain a trained speech synthesis model; wherein the preset loss function includes at least a quantizer loss function or a sequence alignment loss function.

8. The method according to claim 1, wherein The step of performing feature extraction on the target text to obtain a text feature vector of the target text includes: extracting semantic features and prosodic features from the target text respectively; A text feature vector of the target text is obtained according to the semantic features and prosodic features of the target text.

9. The method according to claim 8, wherein The extracting of semantic features and prosodic features from the target text includes: Extracting the semantic features from the target text based on a preset first encoder; wherein the preset first encoder is a first pre-trained model for capturing semantic information in the text; Based on a preset second encoder, the prosodic feature is extracted from the target text; wherein the preset second encoder is a second pre-trained model for capturing prosodic information in the text.

10. The method according to claim 8, wherein Obtaining a text feature vector of the target text based on the semantic features and prosodic features of the target text includes: The semantic features and prosodic features of the target text are added together to obtain a text feature vector of the target text.

11. A text-based speech synthesis device, comprising: A data acquisition unit, configured to acquire a target text and a reference audio; wherein the target text is a text to be synthesized into speech, and the reference audio is an audio with a preset timbre; A feature extraction unit, configured to perform feature extraction processing on the target text to obtain a text feature vector of the target text; wherein the text feature vector represents the semantic features and prosodic features of the text; An audio obtaining unit is used to obtain a reference text corresponding to the reference audio; wherein the text content in the reference text is consistent with the speech content in the reference audio; an audio feature vector of the target text is determined based on the target text, the reference text, and the reference audio; wherein the audio feature vector represents the semantic features of the audio; a target audio corresponding to the target text is obtained based on the text feature vector, the Mel spectrogram of the reference audio, and the audio feature vector; wherein the Mel spectrogram represents the timbre and spectrum information of the audio, the Mel spectrogram is obtained by inputting the reference audio into a preset Mel encoder, and the timbre of the target audio is consistent with the timbre of the reference audio.

12. The device according to claim 11, wherein The vector determination submodule is specifically used to: Inputting the reference audio into a preset speech encoder to obtain an audio code of the reference audio; wherein the audio code is a discretized ID representing the pronunciation of the audio; An audio feature vector of the target text is obtained according to the audio encoding of the reference audio, the target text, and the reference text.

13. The device according to claim 12, wherein The vector determination submodule is specifically used to: Performing feature extraction processing on the reference audio based on a preset speech encoder to obtain a feature vector of the reference audio; Based on a preset quantizer, the feature vector of the reference audio is converted into the audio code.

14. The device according to claim 12, wherein The vector determination submodule is specifically used to: Inputting the target text and the reference text into a preset text encoder to obtain a fused text vector; wherein the fused text vector represents the target text and the reference text; An audio feature vector of the target text is determined according to the fused text vector and the audio encoding of the reference audio.

15. The device according to claim 14, wherein The vector determination submodule is specifically used to: Determining the audio encoding of the target text according to the fused text vector and the audio encoding of the reference audio; Based on a preset quantizer, the audio code of the target text is converted into an audio feature vector of the target text.

16. The apparatus according to claim 11, further comprising: A data set acquisition unit is used to acquire a data set to be trained; wherein the data set to be trained includes a text to be trained, an audio to be trained, a reference text, a reference audio, a text feature vector of the text to be trained, and a mel spectrogram of the reference audio; A data input unit, configured to input the text feature vector of the text to be trained, the mel-spectrogram of the reference audio, the text to be trained, the reference text, and the reference audio into an initial model to obtain a prediction vector of the text to be trained; wherein the prediction vector represents the predicted audio corresponding to the text to be trained; A model training unit is used to train the initial model based on the audio to be trained and the prediction vector based on a preset loss function to obtain a trained speech synthesis model; wherein the speech synthesis model is used to determine the audio feature vector of the target text based on the target text, the reference text, and the reference audio, and obtain the target audio corresponding to the target text based on the text feature vector of the target text, the mel-spectrogram of the reference audio, and the audio feature vector.

17. The device according to claim 16, wherein The model training unit includes: An audio training module is used to perform feature extraction processing on the audio to be trained to obtain a training vector corresponding to the audio to be trained; A loss determination module is used to train the initial model based on the vector to be trained and the predicted vector based on a preset loss function to obtain a trained speech synthesis model; wherein the preset loss function includes at least a quantizer loss function or a sequence alignment loss function.

18. The device according to claim 11, wherein The feature extraction unit includes: A feature extraction module, configured to extract semantic features and prosodic features from the target text; The feature acquisition module is used to obtain a text feature vector of the target text according to the semantic features and prosodic features of the target text.

19. The device according to claim 18, wherein The feature extraction module includes: A semantic extraction submodule, configured to extract the semantic features from the target text based on a preset first encoder; wherein the preset first encoder is a pre-trained model for capturing semantic information in the text; The prosody extraction submodule is used to extract the prosody features from the target text based on a preset second encoder; wherein the preset second encoder is a pre-trained model for capturing prosody information in the text.

20. The apparatus according to claim 18, wherein The feature acquisition module includes: The feature fusion submodule is used to perform addition processing on the semantic features and prosodic features of the target text to obtain a text feature vector of the target text.

21. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 10.

22. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-10.

23. A computer program product, wherein The invention comprises a computer program, which implements the steps of the method according to any one of claims 1 to 10 when executed by a processor.

Images