Speech synthesis method and device based on retrieval enhanced generation

Abstract

The present application provides a speech synthesis method and device based on retrieval-enhanced generation, the method comprising: based on the retrieval-enhanced generation mode, using a context-aware contrastive language and audio model to determine each prompt voice and prompt text as a voice prompt of a target text; splicing the prompt text with the target text to obtain a spliced ​​text, and splicing each prompt voice to obtain a spliced ​​voice; inputting the spliced ​​text and the spliced ​​voice into a prompt-based speech synthesis model, so that the prompt-based speech synthesis model outputs a synthesized voice corresponding to the target text. The present application can improve the effectiveness and reliability of speech prompt selection using the context-aware contrastive language and audio model by designing a context-aware contrastive language and audio model that can map text and audio to a unified shared feature space based on context perception, thereby improving the accuracy and reliability of the rhythmic expression of the synthesized voice generated according to the voice prompt.

Description

Technical Field

[0001] The present application relates to the technical field of speech synthesis processing, and in particular to a speech synthesis method and device based on retrieval-enhanced generation. Background Art

[0002] TTS (Text To Speech) can be called a speech synthesis system or model, that is, "from text to speech", which can convert computer-generated or externally input text information into understandable and fluent Chinese spoken language (or other language speech) output. Among them, prompt-based models such as GPT-SoVITS, which can support a small amount of speech conversion and text-to-speech timbre synthesis, have become speech synthesis models worthy of attention at present because of their ability to generate audio that is very similar to the target person's voice and the advantage of only needing very short speech samples for effective training.

[0003] At present, the speech synthesis model based on prompts (also known as voice prompts, prompts or prompt words, etc.) usually randomly selects any speech of the target person as a prompt. In the case of random selection, a speech with completely opposite emotions and expressions may be selected. As a result, in the process of generating speech based on prompts, the speech synthesis model based on prompts may refer to inappropriate prosodic expressions from the randomly selected voice prompts, thereby generating speech with inappropriate prosodic expressions, which is not sufficient to effectively guide the speech synthesis system to imitate the required speaking style and timbre of the target person, and cannot guarantee that the prompts can change according to different texts. Summary of the invention

[0004] In view of this, the embodiments of the present application provide a speech synthesis method and device based on retrieval enhanced generation to eliminate or improve one or more defects existing in the prior art.

[0005] One aspect of the present application provides a speech synthesis method based on retrieval enhancement generation, comprising:

[0006] Based on the retrieval-enhanced generation method, a preset context-aware contrastive language and audio model is used to determine each prompt voice as a voice prompt of a target text, and a prompt text corresponding to each of the prompt voices, wherein the context-aware contrastive language and audio model is used to extract voice data and text data based on context-aware contrastive learning and convert them into a shared feature space to extract feature vectors corresponding to each of the voice data and the text data;

[0007] Each of the prompt texts is concatenated with the target text to obtain a corresponding concatenated text, and each of the prompt voices is concatenated to obtain a corresponding concatenated voice; the concatenated text and the concatenated voice are input into a preset prompt-based speech synthesis model, so that the prompt-based speech synthesis model outputs the synthesized voice corresponding to the target text.

[0008] In some embodiments of the present application, the retrieval-enhanced generation method uses a preset context-aware contrastive language and audio model to determine each prompt voice as a voice prompt of the target text, and the prompt text corresponding to each of the prompt voices, including:

[0009] Inputting the speech data in each pre-acquired text and speech data pair into an audio encoder in a preset context-aware contrastive language and audio model, respectively, so that the audio encoder outputs an audio feature vector associated with the target style corresponding to each of the speech data;

[0010] Inputting a target text and the context of the target text into a text encoder in the context-aware contrastive language and audio model, and then inputting an initial text feature vector corresponding to the target text and an initial text feature vector corresponding to the context of the target text output by the text encoder into a cross-attention module in the context-aware contrastive language and audio model, so that the cross-attention module outputs a text feature vector based on context information, and selects a plurality of the audio feature vectors according to the text feature vectors to serve as voice prompts of the target text respectively;

[0011] In each of the text and voice data pairs, the voice data corresponding to the voice prompt of each of the target texts are selected as the prompt voices, and the text data corresponding to the prompt voices are selected as the prompt texts.

[0012] In some embodiments of the present application, the audio feature vectors associated with the target style corresponding to each of the speech data are stored in an audio vector database;

[0013] Correspondingly, selecting a plurality of the audio feature vectors according to the text feature vector to serve as voice prompts of the target text respectively includes:

[0014] Determine the cosine similarity between the text feature vector and each of the audio feature vectors in the audio vector database respectively;

[0015] Among the results of sorting the cosine similarities from large to small, the audio feature vectors corresponding to the first P cosine similarities are selected as the voice prompts of the target text, respectively, where P is a positive integer greater than 1.

[0016] In some embodiments of the present application, the context-aware contrastive language and audio model includes:

[0017] An audio encoder is used to extract audio features from the speech data in each pair of text and speech data to obtain audio feature vectors corresponding to each of the speech data;

[0018] A text encoder, used for performing text feature extraction on the text data in each pair of text and speech data, and the context data corresponding to each of the text data, so as to obtain a text initial feature vector corresponding to each of the text data and a text initial feature vector corresponding to each of the context data;

[0019] A cross attention module is used to perform feature fusion on the text initial feature vectors corresponding to each of the text data output by the text encoder and the text initial feature vectors corresponding to each of the context data, so as to obtain the text feature vectors corresponding to each of the text data;

[0020] And, a first linear layer and a second linear layer, the first linear layer is used to output each of the audio feature vectors obtained by the audio encoder, and the second linear layer is used to output each of the text feature vectors obtained by the cross-attention module, so as to generate a matrix composed of cosine similarities between each of the audio feature vectors output by the first linear layer and each of the text feature vectors output by the second linear layer in the shared feature space, and use the matrix to iteratively train the context-aware contrastive language and audio model based on a preset loss function.

[0021] In some embodiments of the present application, before the retrieval-enhanced generation method is used to determine each prompt voice as a voice prompt of the target text using a preset context-aware contrastive language and audio model, and the prompt text corresponding to each of the prompt voices, the method further includes:

[0022] Divide each text and speech data pair into training data, test data, and verification data;

[0023] Using the training data, pre-training the context-aware contrastive language and audio model based on a preset loss function;

[0024] The pre-trained context-aware contrastive language and audio model is tested based on the test data, and the tested context-aware contrastive language and audio model is verified using the verification data.

[0025] In some embodiments of the present application, before dividing each text and voice data pair into training data, test data and verification data, the method further includes:

[0026] Get multiple audiobook data;

[0027] Separating the background application of each audio book data to obtain the complete voice data corresponding to each audio book data;

[0028] Segment each of the complete speech data into audio data corresponding to each independent sentence;

[0029] Transcribing each of the audio data to obtain the corresponding each of the voice data;

[0030] Each of the voice data and its corresponding text data are respectively combined into different text and voice data pairs.

[0031] In some embodiments of the present application, the audio encoder includes a HTSAT model, and / or the text encoder includes a RoBERTa model.

[0032] Another aspect of the present application provides a speech synthesis method based on retrieval enhanced generation, comprising:

[0033] A retrieval enhancement generation module, used to determine, based on a retrieval enhancement generation method, each prompt voice as a voice prompt of a target text and a prompt text corresponding to each of the prompt voices using a preset context-aware contrastive language and audio model, wherein the context-aware contrastive language and audio model is used to extract voice data and text data based on a context-aware contrastive learning method and convert them into a shared feature space to extract feature vectors corresponding to each of the voice data and the text data;

[0034] A speech synthesis module is used to splice each of the prompt texts with the target text to obtain a corresponding spliced ​​text, and to splice each of the prompt voices to obtain a corresponding spliced ​​voice; the spliced ​​text and the spliced ​​voice are input into a preset prompt-based speech synthesis model so that the prompt-based speech synthesis model outputs a synthesized voice corresponding to the target text.

[0035] The third aspect of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the speech synthesis method based on retrieval-enhanced generation when executing the computer program.

[0036] The fourth aspect of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the speech synthesis method based on retrieval-enhanced generation.

[0037] The fifth aspect of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the speech synthesis method based on retrieval-enhanced generation.

[0038] The speech synthesis method based on retrieval enhancement generation provided in the present application is based on the retrieval enhancement generation mode, and uses a preset context-aware contrastive language and audio model to determine each prompt voice as a voice prompt of a target text, and a prompt text corresponding to each of the prompt voices, wherein the context-aware contrastive language and audio model is used to extract voice data and text data based on context-aware contrastive learning and convert them into a shared feature space to extract feature vectors corresponding to each of the voice data and the text data; each of the prompt texts is spliced ​​with the target text to obtain a corresponding spliced ​​text, and each of the prompt voices is spliced ​​to obtain a corresponding spliced ​​voice; the spliced ​​text and the spliced ​​voice are input into a preset prompt-based speech synthesis model, so that the prompt-based speech synthesis model outputs the synthesized voice corresponding to the target text. The present application improves the effectiveness and reliability of speech prompt selection using the context-aware contrastive language and audio model by designing a context-aware contrastive language and audio model that can map text and audio to a unified shared feature space based on context perception, thereby improving the accuracy and reliability of the rhythmic expression of the synthesized voice generated according to the voice prompt.

[0039] Additional advantages, purposes, and features of the present application will be partially described in the following description, and will become partially apparent to those skilled in the art after studying the following, or may be learned from the practice of the present application. The purposes and other advantages of the present application can be achieved and obtained by the structures specifically pointed out in the specification and the drawings.

[0040] Those skilled in the art will understand that the purposes and advantages that can be achieved by the present application are not limited to the above specific description, and the above and other purposes that can be achieved by the present application will be more clearly understood based on the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The drawings described herein are used to provide a further understanding of the present application, constitute a part of the present application, and do not constitute a limitation of the present application. The components in the drawings are not drawn to scale, but are only for the purpose of illustrating the principles of the present application. In order to facilitate the illustration and description of some parts of the present application, the corresponding parts in the drawings may be enlarged, that is, they may become larger relative to other components in the exemplary device actually manufactured according to the present application. In the drawings:

[0042] Figure 1 This is a first flow chart of a speech synthesis method based on retrieval-enhanced generation in one embodiment of the present application.

[0043] Figure 2 This is a second flow chart of the speech synthesis method based on retrieval-enhanced generation in one embodiment of the present application.

[0044] Figure 3 The flowchart is a schematic diagram of the three main processes integrated in the search enhancement generation method in one embodiment of the present application.

[0045] Figure 4 1 is a flowchart of step 120 in the speech synthesis method based on retrieval enhanced generation in one embodiment of the present application.

[0046] Figure 5 Schematic diagram of the architecture of a context-aware comparative language and audio model in one embodiment of the present application.

[0047] Figure 6 It is a structural diagram of a speech synthesis device based on retrieval enhancement generation in one embodiment of the present application. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the implementation modes and the accompanying drawings. Here, the illustrative implementation modes and descriptions of the present application are used to explain the present application, but are not intended to limit the present application.

[0049] It should also be noted here that in order to avoid obscuring the present application due to unnecessary details, only the structures and / or processing steps closely related to the scheme according to the present application are shown in the accompanying drawings, while other details that are not very relevant to the present application are omitted.

[0050] It should be emphasized that the term “include / comprises” when used herein refers to the presence of features, elements, steps or components, but does not exclude the presence or addition of one or more other features, elements, steps or components.

[0051] It should also be noted that, unless otherwise specified, the term “connection” herein may refer not only to a direct connection but also to an indirect connection involving an intermediate.

[0052] Hereinafter, embodiments of the present application will be described with reference to the accompanying drawings. In the accompanying drawings, the same reference numerals represent the same or similar components, or the same or similar steps.

[0053] The embodiments of the present application respectively provide a speech synthesis method based on retrieval-enhanced generation, a speech synthesis device based on retrieval-enhanced generation for executing the speech synthesis method based on retrieval-enhanced generation, a physical device, a computer-readable storage medium and a computer program product, which can improve the generation quality and adaptability of the speech synthesis system.

[0054] The details are described in detail through the following examples.

[0055] Based on this, the embodiment of the present application provides a speech synthesis method based on retrieval enhancement generation that can be implemented by a speech synthesis device based on retrieval enhancement generation, see Figure 1 The speech synthesis method based on retrieval enhancement generation specifically includes the following contents:

[0056] Step 100: Based on the retrieval enhanced generation method, a preset context-aware contrastive language and audio model is used to determine each prompt voice as a voice prompt of the target text, and the prompt text corresponding to each of the prompt voices, wherein the context-aware contrastive language and audio model is used to extract voice data and text data based on context-aware contrastive learning and convert them into a shared feature space to extract feature vectors corresponding to each of the voice data and the text data.

[0057] It should be noted that the target text refers to the input text that is currently to be speech synthesized and specifies the target person. The target person refers to the person whose speaking style and timbre are to be simulated. The voice prompt of the target text determined or selected based on the context-aware contrast language and audio model refers to the voice data related to the speaking style and timbre of the target person. In one or more of the present applications, "style" is defined as a higher level of expression possessed by the target speaker, such as emotion, expressiveness, etc.

[0058] In order to solve the problems that existing prompt-based speech synthesis methods are prone to generate speech with inappropriate prosodic expressions and cannot ensure that the prompts can change according to different texts, the existing technical difficulties are: when inferring speech generation, there are only texts to be synthesized and contextual texts. Therefore, if the most suitable speech is found as a prompt based only on these existing texts, the matching process is difficult to achieve. This means that the text vector needs to be queried with the speech vector for similarity to find the audio expression that is most similar to the text. If only clustering operations are used, all speech is clustered, and speech in the class that matches the current synthesized text prosody is selected. However, this approach is highly dependent on clustering results, and the number of categories of the clustering results depends on manually set hyperparameters, which are not directly linked to the actual prosodic labels.

[0059] Based on this, the designers of the present application designed a context-aware contrastive language and audio model. However, a simple context-aware contrastive language and audio model still cannot provide intelligent and automated prompt selection services for the prompt-based speech synthesis process. Therefore, the designers of the present application also proposed to use a retrieval-enhanced generation method to apply the context-aware contrastive language and audio model to generate the voice prompts of the target speaking person required to be simulated by the prompt-based speech synthesis model.

[0060] The context-aware contrastive language and audio model used in the embodiments of the present application can be referred to as the CA-CLAP (Context-Aware Contrastive Language-Audio Pre-training) model to solve the problem of feature unified space when calculating similarity and the challenge of similarity considering context.

[0061] The context-aware contrast language and audio model used in the embodiments of the present application can not only effectively improve the appropriateness of the expression of the target style of the selected voice prompt, but also can combine the context-aware means to select different voice prompts according to different texts. In the speech synthesis scenario containing context information, such as audio book speech synthesis and conversational speech synthesis, the selection of voice prompts should also consider the context information and select different voice prompts corresponding to audio book speech synthesis and conversational speech synthesis. Among them, the voice prompts mentioned in this application can be referred to as prompts, and the two have the same meaning.

[0062] That is to say, compared with the prior art, this application proposes a CA-CLAP model to map text and audio into a unified feature space while considering the context. The CA-CLAP model is pre-trained and used as a feature extractor for subsequent voice prompt selection, which can solve the current difficulties in voice prompt selection.

[0063] In one or more embodiments of the present application, the retrieval-augmented generation method refers to the RAG (Retrieval-Augmented Generation) method. In order to solve the problems of existing prompt-based speech synthesis methods that are prone to generate speech with inappropriate prosodic expression and cannot ensure that the prompts can change according to different texts, the embodiments of the present application not only design the context-aware contrastive language and audio model, but also adapt the RAG concept to the speech field in order to realize the context-aware contrastive language and audio model in the speech synthesis process, so as to solve the challenge of selecting appropriate speech prompts for existing prompt-based speech synthesis models.

[0064] Step 200: splicing each of the prompt texts with the target text to obtain a corresponding spliced ​​text, and splicing each of the prompt voices to obtain a corresponding spliced ​​voice; inputting the spliced ​​text and the spliced ​​voice into a preset prompt-based speech synthesis model, so that the prompt-based speech synthesis model outputs the synthesized voice corresponding to the target text.

[0065] In step 200, the prompt-based speech synthesis model may adopt a speech synthesis model such as a GPT-SoVITS model.

[0066] From the above description, it can be seen that the speech synthesis method based on retrieval enhanced generation provided in the embodiment of the present application not only takes into account text data, but also introduces sound input in the retrieval process. The retrieval enhanced generation (RAG) method is based on the purpose of solving the challenges of speech prompt selection in the current speech synthesis system. The framework introduces context-aware contrastive language and audio models, aiming to retrieve context-aware features from audio, which exist in the same space shared by acoustics and text, and selects appropriate voice prompts through retrieval to improve the quality and relevance of the retrieved content, thereby improving the generation quality and adaptability of the TTS system.

[0067] In order to further improve the effectiveness and accuracy of determining the voice prompt of the target text using the preset context-aware contrastive language and audio model, in a speech synthesis method based on retrieval enhanced generation provided in an embodiment of the present application, see Figure 2 , step 100 in the speech synthesis method based on retrieval enhancement generation specifically includes the following contents:

[0068] Step 110: Input the speech data in each pre-acquired text and speech data pair into an audio encoder in a preset context-aware contrastive language and audio model, so that the audio encoder outputs an audio feature vector associated with the target style corresponding to each of the speech data.

[0069] Step 120: Input the target text and the context of the target text into the text encoder in the context-aware contrastive language and audio model, and then input the text initial feature vector corresponding to the target text and the text initial feature vector corresponding to the context of the target text output by the text encoder into the cross-attention module in the context-aware contrastive language and audio model, so that the cross-attention module outputs a text feature vector that takes context information into account, and selects multiple features from each of the audio feature vectors according to the text feature vectors to serve as voice prompts for the target text.

[0070] Step 130: In each of the text and voice data pairs, select the voice data corresponding to each voice prompt of the target text as the prompt voice, and select the text data corresponding to each prompt voice as the prompt text.

[0071] That is to say, the embodiment of the present application can enhance the prompt-based speech synthesis system by introducing the retrieval-augmented generation method RAG (Retrieval-Augmented Generation).

[0072] See also Figure 3 The retrieval enhanced generation method integrates three main processes: indexing (corresponding to the above step 110), retrieval (corresponding to the above step 120) and generation (corresponding to the above steps 130 and 200). Figure 3 The red words in the text “The teacher was very angry and said I was naughty” are the target text, and the content containing the red words “Later I found out that I was imitating his actions. The teacher was very angry and said I was naughty. After only one month at school, the teacher came to visit my home (realized I was mimicking him. He got angry, and called me naughty. He visited home, criticizing)” is the context of the target text.

[0073] In order to further improve the effectiveness and accuracy of the voice prompt for selecting the target text, in a speech synthesis method based on retrieval enhancement generation provided in an embodiment of the present application, the audio feature vectors associated with the target style corresponding to each of the speech data are stored in an audio vector database, see Figure 4 Step 120 in the speech synthesis method based on retrieval enhancement generation specifically includes the following contents:

[0074] Step 121: Input the target text and the context of the target text into the text encoder in the context-aware contrastive language and audio model, and then input the text initial feature vector corresponding to the target text output by the text encoder and the text initial feature vector corresponding to the context of the target text into the cross-attention module in the context-aware contrastive language and audio model, so that the cross-attention module outputs a text feature vector associated with the target style.

[0075] Step 122: respectively determine the cosine similarity between the text feature vector and each of the audio feature vectors in the audio vector database.

[0076] Step 123: From the sorting results of the cosine similarities from large to small, select the audio feature vectors corresponding to the first P cosine similarities as the voice prompts of the target text, respectively, where P is a positive integer greater than 1.

[0077] Specifically, first, in the indexing stage, all potential speech cues in the text-to-speech database are converted into representations related to the target style, namely audio feature vectors, through the audio encoder in the pre-trained context-aware contrastive language and audio model (CA-CLAP), and stored in the audio vector database. At the same time, the text input is converted into text feature vectors (STFs) related to the target style through the text encoder in the pre-trained CA-CLAP model.

[0078] Next, in the retrieval phase, the context of the current target text is used as a query to compute similarity (e.g., cosine similarity) scores with the speech prompts indexed in the audio vector database to retrieve the top K (i.e., TopK) speech prompts that exhibit the highest similarity. K is greater than P.

[0079] Specifically, during the retrieval process, the current text and context are encoded into style-dependent text features (STFs) via a pre-trained CA-CLAP text encoder. These generated context-aware text representations are used as queries to compute similarity scores with speech cues vectorized in the indexed audio vector database. The model prioritizes retrieving the top K speech cues that exhibit the greatest similarity.

[0080] The context-dependent text-to-speech similarity calculation formula is as follows:

[0081] U con =Concat(U i-l , U i-l+1 , ..., U i , ..., U i+l-1 , U i+l )

[0082] Q=W Q E cur , K=W K con , V=W V E con

[0083]

[0084] The above formula provides a context length of 2l and a current text U i Context U con The definition of and through the attention weight matrix W K , W Q and W V The method of calculating cross attention. Among them, U i Indicates the i-th article in the paragraph, E cur represents the hidden vector of the current text after passing through the CA-CLAP text encoder, E con Represents the hidden vector of the current text after passing through the CA-CLAP text encoder.

[0085] Finally, in the generation phase, the retrieved speech prompts are concatenated and used to guide a prompt-based speech synthesis model, which can adopt GPT-SoVITS pre-trained model parameters. GPT-SoVITS can use discrete semantic tags as intermediate features and use the GPT method to convert text into discrete semantic tags.

[0086] Specifically, during the generation process, the first P prompts can be used and concatenated as the final speech prompt to guide the pre-trained prompt-based TTS system to generate suitable speech.

[0087] To train GPT-SoVITS, a two-stage process is adopted. First, the VITS decoder and vector quantization (VQ) model are jointly trained via the VITS loss and the VQ commitment loss. Then, the text-to-semantics model is trained in a GPT-like manner to predict the next semantic token, leveraging the pre-trained HuBERT model and the well-trained VQ model.

[0088] In order to further improve the application effectiveness and accuracy of context-aware contrastive language and audio models, in a speech synthesis method based on retrieval-enhanced generation provided in an embodiment of the present application, see Figure 5 The context-aware contrastive language and audio model in the speech synthesis method based on retrieval-enhanced generation specifically includes the following contents:

[0089] An audio encoder is used to extract audio features from the speech data in each pair of text and speech data to obtain audio feature vectors corresponding to each of the speech data;

[0090] A text encoder, used for performing text feature extraction on the text data in each pair of text and speech data, and the context data corresponding to each of the text data, so as to obtain a text initial feature vector corresponding to each of the text data and a text initial feature vector corresponding to each of the context data;

[0091] A cross attention module is used to perform feature fusion on the text initial feature vectors corresponding to each of the text data output by the text encoder and the text initial feature vectors corresponding to each of the context data, so as to obtain the text feature vectors corresponding to each of the text data;

[0092] And, a first linear layer and a second linear layer, the first linear layer is used to output each of the audio feature vectors obtained by the audio encoder, and the second linear layer is used to output each of the text feature vectors obtained by the cross-attention module, so as to generate a matrix composed of cosine similarities between each of the audio feature vectors output by the first linear layer and each of the text feature vectors output by the second linear layer in the shared feature space, and use the matrix to iteratively train the context-aware contrastive language and audio model based on a preset loss function.

[0093] exist Figure 5 In order to facilitate the representation that the text encoder processes each of the text data and the context data corresponding to each of the text data, the text encoder is represented as two shared ones, and the representations of the two text encoders refer to the same text encoder. That is, in order to calculate cosine similarity, the text modality and acoustic modality input (text and audio) need to be embedded in a shared feature space. The present application implements two encoders to process audio data and text data respectively, and utilizes additional context information through a cross-attention mechanism.

[0094] Specifically, the context-aware contrastive language and audio model includes a text encoder and an audio encoder, which are used to embed text modality and acoustic modality input (text and audio) into a shared feature space, respectively. During training, given N pairs of audio and text as input, CA-CLAP calculates an N×N matrix M. The value of the i-th row and j-th column in the matrix M represents the text embedding T of the i-th text i (obtained by CA-CLAP text encoder) and the audio embedding A of the jth speech i The cosine similarity between (obtained by CA-CLAP audio encoder), A Nis the Nth audio embedding (i.e., audio feature vector), T N is the Nth text embedding (i.e., text feature vector).

[0095] The text encoder and audio encoder strive to maximize the cosine similarity of the N correct pairs in the batch and minimize the similarity of the incorrect pairs. The model will be trained using a contrastive learning paradigm between paired audio and text embeddings, following the same loss function L as the CLAP model. The two logarithmic terms take into account the audio-to-text logic or the text-to-audio logic.

[0096]

[0097] Where τ is a learnable temperature parameter used to scale the values.

[0098] The trained CA-CLAP text encoder and audio encoder are then used to extract features from text and audio pairs respectively, and the correlation of the extracted embeddings is scored by calculating the cosine distance.

[0099] That is, the embodiment of the present application mainly includes prompt-based speech synthesis combined with RAG, and the main key module is the context-aware contrastive language and audio model, which acts as an encoding model in the indexing and retrieval stage of RAG, converting multimodal input into a shared feature space based on context-aware contrastive learning. Therefore, the context-aware style-related text features can retrieve style-related audio as speech prompts.

[0100] In order to further improve the application effectiveness and accuracy of context-aware contrastive language and audio models, in a speech synthesis method based on retrieval-enhanced generation provided in an embodiment of the present application, see Figure 2 The speech synthesis method based on retrieval enhancement generation further specifically includes the following contents before step 100:

[0101] Step 010: Divide each text and speech data pair into training data, test data and verification data.

[0102] Step 020: Using the training data, pre-train the context-aware contrastive language and audio model based on a preset loss function.

[0103] Step 030: Testing the pre-trained context-aware contrastive language and audio model based on the test data, and verifying the tested context-aware contrastive language and audio model using the verification data.

[0104] In order to further improve the application effectiveness and accuracy of context-aware contrastive language and audio models, in a speech synthesis method based on retrieval-enhanced generation provided in an embodiment of the present application, see Figure 2 The speech synthesis method based on retrieval enhancement generation further specifically includes the following contents before step 010:

[0105] Step 001: Acquire multiple audiobook data.

[0106] Step 002: Separate the background application of each audio book data to obtain the complete voice data corresponding to each audio book data.

[0107] Step 003: Segment each of the complete speech data into audio data corresponding to each independent sentence.

[0108] Step 004: transcribe each of the audio data to obtain the corresponding each of the voice data.

[0109] Step 005: Each of the voice data and its corresponding text data are respectively combined into different text and voice data pairs.

[0110] In one example, the present application can collect 3254 Chinese audiobooks with a total duration of 616 hours. First, the background music is separated and the entire speech is divided into separate sentences (utterances). Secondly, the audio is transcribed using the Paraformer tool. The training data includes speech and text pairs. The CA-CLAP model calculates an N×N matrix M, in which each element represents the cosine similarity between the text embedding and the speech embedding. The training data and test data are divided into 100 audiobooks (containing 9000 text-audio pairs) as the test set. Another 10 audiobooks are used as the validation set. The remaining 3144 audiobooks (containing 285,000 text-audio pairs) are used as the training set.

[0111] In order to further improve the application effectiveness and accuracy of context-aware contrastive language and audio models, in a speech synthesis method based on retrieval enhanced generation provided in an embodiment of the present application, the audio encoder includes an HTSAT model, and / or the text encoder RoBERTa model.

[0112] Among them, through the pre-trained context-aware contrastive language and audio model HTSAT model, all possible voice cues are converted into style-related representations, and an audio vector database is constructed, which is used for key similarity comparisons in the retrieval stage.

[0113] To summarize, the speech synthesis method based on retrieval-enhanced generation provided in the embodiment of the present application and the proposed RAG enhancement method combine the indexing, retrieval and generation stages, use CA-CLAP for context-aware contrastive learning, and then use GPT-SoVITS for prompt-based speech synthesis, thereby improving the rhythmic performance of TTS.

[0114] Specifically, the speech synthesis method based on retrieval enhancement generation provided by the present application has the following beneficial effects:

[0115] 1. Introducing the RAG (Retrieval-Augmented Generation) method from the field of large language models (LLM) to the field of speech synthesis. This innovative application introduces the concept of retrieval-augmented generation into the prompt-based speech synthesis system, aiming to solve the challenge of selecting appropriate speech prompts.

[0116] 2. Unlike traditional RAG methods, this application incorporates acoustic input in the retrieval process instead of relying solely on text data. The benefit of this is that it can utilize richer information provided by the acoustic modality, including sentiment and speaking style, thereby improving the overall quality and relevance of the retrieved content.

[0117] 3. In order to make full use of additional contextual information, the CA-CLAP model used in the indexing and retrieval stages of this application combines current language features with contextual information through a cross-attention mechanism. This can make the selection of voice prompts more intelligent and personalized, and help generate more expected voice output.

[0118] From the software level, the present application also provides a speech synthesis device based on retrieval enhancement generation for executing all or part of the speech synthesis method based on retrieval enhancement generation, see Figure 6 The speech synthesis device based on retrieval enhancement generation specifically includes the following contents:

[0119] The retrieval enhancement generation module 10 is used to determine, based on the retrieval enhancement generation method, each prompt voice as a voice prompt of the target text, and the prompt text corresponding to each of the prompt voices using a preset context-aware contrastive language and audio model, wherein the context-aware contrastive language and audio model is used to extract voice data and text data based on context-aware contrastive learning and convert them into a shared feature space to extract feature vectors corresponding to each of the voice data and the text data.

[0120] The speech synthesis module 20 is used to splice each of the prompt texts with the target text to obtain a corresponding spliced ​​text, and to splice each of the prompt voices to obtain a corresponding spliced ​​voice; the spliced ​​text and the spliced ​​voice are input into a preset prompt-based speech synthesis model so that the prompt-based speech synthesis model outputs a synthesized voice corresponding to the target text.

[0121] The embodiment of the speech synthesis device based on retrieval enhancement generation provided in the present application can be specifically used to execute the processing flow of the embodiment of the speech synthesis method based on retrieval enhancement generation in the above-mentioned embodiment. Its functions will not be repeated here, and reference can be made to the detailed description of the above-mentioned embodiment of the speech synthesis method based on retrieval enhancement generation.

[0122] The part of the speech synthesis device based on retrieval enhancement generation that performs speech synthesis based on retrieval enhancement generation can be executed in the server or completed in the client device. The specific selection can be based on the processing capability of the client device and the limitations of the user's usage scenario. This application does not limit this. If all operations are completed in the client device, the client device may also include a processor for specific processing of speech synthesis based on retrieval enhancement generation.

[0123] The client device may have a communication module (i.e., a communication unit) that can communicate with a remote server to achieve data transmission with the server. The server may include a server on the task scheduling center side, and other implementation scenarios may also include a server on an intermediate platform, such as a server on a third-party server platform that has a communication link with the task scheduling center server. The server may include a single computer device, or a server cluster consisting of multiple servers, or a server structure of a distributed device.

[0124] The server and the client device may communicate with each other using any suitable network protocol, including network protocols that have not yet been developed on the date of filing this application. The network protocols may include, for example, TCP / IP, UDP / IP, HTTP, HTTPS, etc. Of course, the network protocols may also include, for example, RPC (Remote Procedure Call Protocol) and REST (Representational State Transfer) protocols used on top of the above protocols.

[0125] From the above description, it can be seen that the speech synthesis device based on retrieval enhanced generation provided in the embodiment of the present application, by designing a context-aware contrastive language and audio model that can map text and audio to a unified shared feature space based on context awareness, can improve the effectiveness and reliability of voice prompt selection using the context-aware contrastive language and audio model, so that the prompt can change according to the different texts, thereby improving the accuracy and reliability of the rhythmic expression of the synthesized speech generated according to the voice prompt, and can effectively guide the prompt-based speech synthesis model to imitate the speaking style and timbre of the required target person.

[0126] The embodiment of the present application also provides an electronic device, which may include a processor, a memory, a receiver and a transmitter, wherein the processor is used to execute the speech synthesis method based on retrieval enhancement generation mentioned in the above embodiment, wherein the processor and the memory may be connected via a bus or other means, such as by bus connection. The receiver may be connected to the processor and the memory via wired or wireless means.

[0127] The processor may be a central processing unit (CPU). The processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or a combination of the above chips.

[0128] As a non-transitory computer-readable storage medium, the memory can be used to store non-transitory software programs, non-transitory computer executable programs and modules, such as the program instructions / modules corresponding to the speech synthesis method based on retrieval enhancement generation in the embodiment of the present application. The processor executes various functional applications and data processing of the processor by running the non-transitory software programs, instructions and modules stored in the memory, that is, the speech synthesis method based on retrieval enhancement generation in the above method embodiment is implemented.

[0129] The memory may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created by the processor, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0130] The one or more modules are stored in the memory, and when executed by the processor, perform the speech synthesis method based on retrieval enhancement generation in the embodiment.

[0131] In some embodiments of the present application, the user equipment may include a processor, a memory, and a transceiver unit, which may include a receiver and a transmitter. The processor, memory, receiver, and transmitter may be connected through a bus system. The memory is used to store computer instructions, and the processor is used to execute the computer instructions stored in the memory to control the transceiver unit to send and receive signals.

[0132] As an implementation method, the functions of the receiver and the transmitter in the present application can be considered to be implemented through a transceiver circuit or a dedicated chip for transceiver, and the processor can be considered to be implemented through a dedicated processing chip, a processing circuit or a general chip.

[0133] As another implementation method, it is possible to use a general-purpose computer to implement the server provided in the embodiment of the present application, that is, to store the program code for implementing the functions of the processor, receiver, and transmitter in a memory, and the general-purpose processor implements the functions of the processor, receiver, and transmitter by executing the code in the memory.

[0134] The embodiment of the present application also provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the steps of the speech synthesis method based on retrieval enhancement generation are implemented. The computer-readable storage medium can be a tangible storage medium, such as a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a floppy disk, a hard disk, a removable storage disk, a CD-ROM, or any other form of storage medium known in the technical field.

[0135] An embodiment of the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the aforementioned speech synthesis method based on retrieval enhancement generation.

[0136] It should be understood by those skilled in the art that the exemplary components, systems and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software or a combination of the two. Whether it is specifically performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link via a data signal carried in a carrier.

[0137] It should be clear that the present application is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present application is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present application.

[0138] In the present application, features described and / or illustrated for one embodiment may be used in the same manner or in a similar manner in one or more other embodiments, and / or combined with features of other embodiments or replace features of other embodiments.

[0139] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the embodiments of the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A speech synthesis method based on retrieval-enhanced generation, characterized in that: include: Based on the retrieval-enhanced generation method, a preset context-aware contrastive language and audio model is used to determine each prompt voice as a voice prompt of a target text, and a prompt text corresponding to each of the prompt voices, wherein the context-aware contrastive language and audio model is used to extract voice data and text data based on context-aware contrastive learning and convert them into a shared feature space to extract feature vectors corresponding to each of the voice data and the text data; Each of the prompt texts is concatenated with the target text to obtain a corresponding concatenated text, and each of the prompt voices is concatenated to obtain a corresponding concatenated voice; the concatenated text and the concatenated voice are input into a preset prompt-based voice synthesis model, so that the prompt-based voice synthesis model outputs a synthesized voice corresponding to the target text; The context-aware contrastive language and audio model includes: An audio encoder is used to extract audio features from the speech data in each pair of text and speech data to obtain audio feature vectors corresponding to each of the speech data; A text encoder, used for performing text feature extraction on the text data in each pair of text and speech data, and the context data corresponding to each of the text data, so as to obtain a text initial feature vector corresponding to each of the text data and a text initial feature vector corresponding to each of the context data; A cross attention module is used to perform feature fusion on the text initial feature vectors corresponding to each of the text data output by the text encoder and the text initial feature vectors corresponding to each of the context data, so as to obtain the text feature vectors corresponding to each of the text data; and, a first linear layer and a second linear layer, wherein the first linear layer is used to output each of the audio feature vectors obtained by the audio encoder, and the second linear layer is used to output each of the text feature vectors obtained by the cross attention module, so as to generate a matrix consisting of cosine similarities between each of the audio feature vectors output by the first linear layer and each of the text feature vectors output by the second linear layer in the shared feature space, and use the matrix to iteratively train the context-aware contrastive language and audio model based on a preset loss function; The retrieval-enhanced generation method uses a preset context-aware contrastive language and audio model to determine each prompt voice as a voice prompt of the target text, and the prompt text corresponding to each of the prompt voices, including: Inputting the speech data in each pre-acquired text and speech data pair into an audio encoder in a preset context-aware contrastive language and audio model, respectively, so that the audio encoder outputs an audio feature vector associated with the target style corresponding to each of the speech data; Inputting a target text and the context of the target text into a text encoder in the context-aware contrastive language and audio model, and then inputting an initial text feature vector corresponding to the target text and an initial text feature vector corresponding to the context of the target text output by the text encoder into a cross-attention module in the context-aware contrastive language and audio model, so that the cross-attention module outputs a text feature vector based on context information, and selects a plurality of the audio feature vectors according to the text feature vectors to serve as voice prompts of the target text respectively; In each of the text and voice data pairs, the voice data corresponding to the voice prompt of each of the target texts are selected as the prompt voices, and the text data corresponding to the prompt voices are selected as the prompt texts.

2. The speech synthesis method based on retrieval enhancement generation according to claim 1, characterized in that: The audio feature vectors associated with the target style corresponding to each of the speech data are stored in an audio vector database; Correspondingly, selecting a plurality of the audio feature vectors according to the text feature vector to serve as voice prompts of the target text respectively includes: Determine the cosine similarity between the text feature vector and each of the audio feature vectors in the audio vector database respectively; Among the results of sorting the cosine similarities from large to small, the audio feature vectors corresponding to the first P cosine similarities are selected as the voice prompts of the target text, respectively, where P is a positive integer greater than 1.

3. The speech synthesis method based on retrieval enhancement generation according to claim 1 or 2, characterized in that: Before the retrieval-enhanced generation method is used to determine each prompt voice as a voice prompt of the target text and the prompt text corresponding to each prompt voice, the method further includes: Divide each text and speech data pair into training data, test data, and verification data; Using the training data, pre-training the context-aware contrastive language and audio model based on a preset loss function; The pre-trained context-aware contrastive language and audio model is tested based on the test data, and the tested context-aware contrastive language and audio model is verified using the verification data.

4. The speech synthesis method based on retrieval enhancement generation according to claim 3 is characterized in that: Before dividing each text and speech data pair into training data, test data and verification data, the method further includes: Get multiple audiobook data; Separating the background application of each audio book data to obtain the complete voice data corresponding to each audio book data; Segment each of the complete speech data into audio data corresponding to each independent sentence; Transcribing each of the audio data to obtain the corresponding each of the voice data; Each of the voice data and its corresponding text data are respectively combined into different text and voice data pairs.

5. The speech synthesis method based on retrieval enhancement generation according to claim 1, characterized in that: The audio encoder includes a HTSAT model, and / or the text encoder includes a RoBERTa model.

6. A speech synthesis method based on retrieval-enhanced generation, characterized in that: include: A retrieval enhancement generation module, used to determine, based on a retrieval enhancement generation method, each prompt voice as a voice prompt of a target text and a prompt text corresponding to each of the prompt voices using a preset context-aware contrastive language and audio model, wherein the context-aware contrastive language and audio model is used to extract voice data and text data based on a context-aware contrastive learning method and convert them into a shared feature space to extract feature vectors corresponding to each of the voice data and the text data; A speech synthesis module, used for splicing each of the prompt texts with the target text to obtain a corresponding spliced ​​text, and splicing each of the prompt voices to obtain a corresponding spliced ​​voice; inputting the spliced ​​text and the spliced ​​voice into a preset prompt-based speech synthesis model, so that the prompt-based speech synthesis model outputs a synthesized voice corresponding to the target text; The context-aware contrastive language and audio model includes: An audio encoder, used for extracting audio features from the speech data in each pair of text and speech data to obtain audio feature vectors corresponding to each of the speech data; A text encoder, used for performing text feature extraction on the text data in each pair of text and speech data, and the context data corresponding to each of the text data, so as to obtain a text initial feature vector corresponding to each of the text data and a text initial feature vector corresponding to each of the context data; A cross attention module is used to perform feature fusion on the text initial feature vectors corresponding to each of the text data output by the text encoder and the text initial feature vectors corresponding to each of the context data, so as to obtain the text feature vectors corresponding to each of the text data; and, a first linear layer and a second linear layer, wherein the first linear layer is used to output each of the audio feature vectors obtained by the audio encoder, and the second linear layer is used to output each of the text feature vectors obtained by the cross attention module, so as to generate a matrix consisting of cosine similarities between each of the audio feature vectors output by the first linear layer and each of the text feature vectors output by the second linear layer in the shared feature space, and use the matrix to iteratively train the context-aware contrastive language and audio model based on a preset loss function; The retrieval-enhanced generation method uses a preset context-aware contrastive language and audio model to determine each prompt voice as a voice prompt of the target text, and the prompt text corresponding to each of the prompt voices, including: Inputting the speech data in each pre-acquired text and speech data pair into an audio encoder in a preset context-aware contrastive language and audio model, respectively, so that the audio encoder outputs an audio feature vector associated with the target style corresponding to each of the speech data; Inputting a target text and the context of the target text into a text encoder in the context-aware contrastive language and audio model, and then inputting an initial text feature vector corresponding to the target text and an initial text feature vector corresponding to the context of the target text output by the text encoder into a cross-attention module in the context-aware contrastive language and audio model, so that the cross-attention module outputs a text feature vector based on context information, and selects a plurality of the audio feature vectors according to the text feature vectors to serve as voice prompts of the target text respectively; In each of the text and voice data pairs, the voice data corresponding to the voice prompt of each of the target texts are selected as the prompt voices, and the text data corresponding to the prompt voices are selected as the prompt texts.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the speech synthesis method based on retrieval enhancement generation as described in any one of claims 1 to 5 is implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the speech synthesis method based on retrieval enhancement generation as described in any one of claims 1 to 5 is implemented.

Images