Speech generation framework training method and device, equipment and storage medium
By using three different speech data training speech generation frameworks, including text-to-acoustic tokens, large language model, stream model, and vocoding model, the problem that the existing technology cannot achieve personalized high-natural speech generation is solved, and the problem of the existing technology is realized. Reproduce the effect of the speaker's tone and rhythm.
Patent Information
- Application Number
- CN202411659432.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-19
- Publication Date
- 2025-02-18
AI Technical Summary
The prior art cannot perfectly reproduce the tone and rhythm of the speaker that has never been seen in training, and cannot achieve personalized high-natural speech generation.
Three different speech data training and optimized speech generation frameworks are adopted, including text-to-acoustic token large language model, stream model, and vocoding model. The first speech data training model is used to adapt to different language styles, the second speech data training model learns sublingual information, and the third speech data training text embedding module and stream model to reproduce the tone and rhythm of the target speaker.
It realizes personalized high-natural speech generation, which can reproduce the tone and rhythm of the speaker that has never been seen in training.
Smart Images
Figure CN119479610A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of natural language processing, and in particular to a training method, apparatus, device and storage medium for a speech generation framework. Background Art
[0002] In recent years, text-to-speech based on large language models has become the mainstream model for speech generation due to its high naturalness and zero-shot capabilities.
[0003] Traditional speech generation methods all use mel spectrogram as an intermediate feature, the acoustic model generates mel spectrogram, and the vocoder synthesizes audio through the mel spectrogram; while speech generation based on a large language model discretizes the speech signal into speech acoustic tokens through the Codec's Encoder, then models the contextual dependencies in the speech through the pre-trained model in NLP (Natural Language Processing), and finally decodes the speech waveform through the Codec's Encoder. However, these two methods of generating speech cannot perfectly reproduce the timbre and rhythm of speakers that have not been seen in training, and cannot achieve personalized and highly natural speech generation. Summary of the invention
[0004] The embodiments of the present application provide a training method, apparatus, device and storage medium for a speech generation framework, so as to achieve the effect of achieving personalized and highly natural speech generation.
[0005] On the one hand, an embodiment of the present application provides a training method for a speech generation framework, wherein the framework includes a large language model, a stream model, and a vocoding model for text-to-acoustic tokens; the large language model for text-to-acoustic tokens includes a text embedding module, a text encoding module, a large language model-encoder module, and a large language model-decoder module, and the training method includes: using first speech data as training data, performing model training on the large language model, the stream model, and the vocoding model for text-to-acoustic tokens in the speech generation framework, and obtaining a speech generation framework in a first training stage; wherein the first speech data includes a first audio, a text corresponding to the first audio, and a speech representation corresponding to the text corresponding to the first audio; Using the second voice data as training data, model training is performed on the large language model, flow model, and voice coding model of text to acoustic tokens in the speech generation framework to obtain the speech generation framework of the second training stage; wherein the second voice data includes the second audio under different paralinguistic information annotations, the text corresponding to the second audio, and the speech representation corresponding to the text corresponding to the second audio; using the third voice data as training data, model training is performed on the text embedding module in the flow model and the large language model of text to acoustic tokens to obtain the trained speech generation framework; the third voice data includes the third audio of the target speaker, the text corresponding to the third audio, and the speech representation corresponding to the text corresponding to the third audio.
[0006] In some examples, the audio in the training data is input into a voiceprint model to obtain a speaker feature vector corresponding to the audio; the text corresponding to the audio in the training data is input into a text encoder to obtain a text encoding sequence corresponding to the audio; a first acoustic token is obtained based on the speech representation corresponding to the text corresponding to the audio in the training data; a large language model of text to acoustic token is trained based on the first acoustic token, the speaker feature vector and the text encoding sequence; the first acoustic token, the speaker feature vector and the text corresponding to the audio are input into the large language model of text to acoustic token to predict a second acoustic token; a stream model is trained based on the first acoustic token; the second acoustic token is input into the stream model to obtain a second speech representation; and a voice coding model is trained based on the speech representation corresponding to the text corresponding to the audio.
[0007] In some examples, the third audio is input into a voiceprint model to obtain a speaker feature vector corresponding to the third audio; the text corresponding to the third audio is input into a text encoder to obtain a text encoding sequence corresponding to the third audio; a third acoustic token is obtained based on the speech representation corresponding to the text corresponding to the third audio; a text embedding model is trained based on the third acoustic token, the speaker feature vector corresponding to the third audio, and the text encoding sequence corresponding to the third audio; the third acoustic token, the speaker feature vector corresponding to the third audio, and the text corresponding to the third audio are input into a large language model from text to acoustic tokens to predict a fourth acoustic token; and a streaming model is trained based on the third acoustic token.
[0008] In some examples, the speech representation corresponding to the text corresponding to the audio is discretized to obtain a first acoustic token.
[0009] In some examples, input text is input into a large language model, and prompt words are designed based on the input text so that the large language model inserts paralinguistic expression symbols into the input text based on the prompt words to obtain input text with paralinguistic annotations; the input text with paralinguistic annotations is input into a trained speech generation framework to generate speech results.
[0010] On the other hand, an embodiment of the present application provides a training device for a speech generation framework, including: a first training module, which is used to perform model training on a large language model, a stream model, and a vocoding model of text to acoustic tokens in the speech generation framework using first speech data as training data, so as to obtain a speech generation framework in a first training stage; wherein the first speech data includes a first audio, a text corresponding to the first audio, and a speech representation corresponding to the text corresponding to the first audio; a second training module, which is used to perform model training on a large language model, a stream model, and a vocoding model of text to acoustic tokens in the speech generation framework using second speech data as training data, so as to obtain a speech generation framework in a second training stage; wherein the second speech data includes a second audio under different paralinguistic information annotations, a text corresponding to the second audio, and a speech representation corresponding to the text corresponding to the second audio; a third training module, which is used to perform model training on a text embedding module in a large language model of the stream model and text to acoustic tokens using third speech data as training data, so as to obtain a trained speech generation framework; and the third speech data includes a third audio of a target speaker, a text corresponding to the third audio, and a speech representation corresponding to the text corresponding to the third audio.
[0011] In some examples, the training module is used to: input the audio in the training data into the voiceprint model to obtain the speaker feature vector corresponding to the audio; input the text corresponding to the audio in the training data into the text encoder to obtain the text encoding sequence corresponding to the audio; obtain a first acoustic token based on the speech representation corresponding to the text corresponding to the audio in the training data; train a large language model of text to acoustic tokens based on the first acoustic token, the speaker feature vector and the text encoding sequence; input the first acoustic token, the speaker feature vector and the text corresponding to the audio into the large language model of text to acoustic tokens to predict a second acoustic token; train a stream model based on the first acoustic token; input the second acoustic token into the stream model to obtain a second speech representation; and train a voice coding model based on the speech representation corresponding to the text corresponding to the audio.
[0012] In some examples, the training module is used to: input the third audio into the voiceprint model to obtain the speaker feature vector corresponding to the third audio; input the text corresponding to the third audio into the text encoder to obtain the text encoding sequence corresponding to the third audio; obtain the third acoustic token based on the speech representation corresponding to the text corresponding to the third audio; train the text embedding module based on the third acoustic token, the speaker feature vector corresponding to the third audio, and the text encoding sequence corresponding to the third audio; input the third acoustic token, the speaker feature vector corresponding to the third audio, and the text corresponding to the third audio into the large language model of text to acoustic token to predict a fourth acoustic token; and train the streaming model based on the third acoustic token.
[0013] In some examples, the training module is used to discretize the speech representation corresponding to the text corresponding to the audio to obtain a first acoustic token.
[0014] On the other hand, the present application provides an electronic device, including: a processor, and a memory communicatively connected to the processor; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory to implement the above method.
[0015] On the other hand, the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the above method.
[0016] The embodiment of the present application provides a training method, device, equipment and storage medium for a speech generation module. The speech framework includes a large language model, a stream model and a vocoder model of text to acoustic tokens; the large language model of text to acoustic tokens includes a text embedding module, a text encoding module, a large language model-encoder module and a large language model-decoder module. The training method includes: using first speech data and second speech data as training data, respectively training the large language model, the stream model and the vocoder model of text to acoustic tokens in the speech generation framework to obtain a speech generation framework in a first training stage; wherein the first speech data includes a first audio, a text corresponding to the first audio, and a speech representation corresponding to the text corresponding to the first audio; the second speech data includes a second audio under different paralinguistic information annotations, a text corresponding to the second audio, and a speech representation corresponding to the text corresponding to the second audio; using third speech data as training data, training the text embedding module of the stream model and the large language model of text to acoustic tokens to obtain a trained speech generation framework; the third speech data includes a third audio of a target speaker, a text corresponding to the third audio, and a speech representation corresponding to the text corresponding to the third audio. This solution uses three different voice data to train and optimize the speech generation framework to reproduce the timbre and rhythm of speakers that have not been seen in training, thereby achieving personalized and highly natural speech generation. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.
[0018] Figure 1 A flowchart of a training method for a speech generation framework provided in this application;
[0019] Figure 2 A speech generation framework;
[0020] Figure 3 A flow chart of a training method for a speech generation framework is shown in FIG. Figure 1 ;
[0021] Figure 4 A flow chart of a training method for a speech generation framework is shown in FIG. Figure 2 ;
[0022] Figure 5 A schematic diagram of the structure of a training device for a speech generation framework provided in this application;
[0023] Figure 6 A schematic diagram of the structure of a device for a speech generation framework provided in this application.
[0024] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION
[0025] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.
[0026] The terms "first", "second", etc. in the specification and claims of this application and the above drawings are used to distinguish similar or similar objects or entities, and do not necessarily mean to limit a specific order or sequence, unless otherwise indicated. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, for example, they can be implemented in an order other than those given in the diagrams or descriptions of the embodiments of this application.
[0027] In addition, the terms "including" and "having" and any variations thereof are intended to cover but not exclude inclusion, for example, a product or device comprising a series of components is not necessarily limited to those components explicitly listed, but may include other components not explicitly listed or inherent to these products or devices. The term "module" as used in this application refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic or combination of hardware and / or software code that can perform the functions associated with the element.
[0028] Speech generation is a technology that converts text or other forms of information into natural speech. Traditional speech generation methods use mel spectrogram as an intermediate feature, the acoustic model generates mel spectrogram, and the vocoder synthesizes audio through the mel spectrogram; while speech generation based on a large language model discretizes the speech signal into speech acoustic tokens through the Codec's Encoder, then uses the pre-trained model in NLP (Natural Language Processing) to model the contextual dependencies in the speech, and finally decodes the speech waveform through the Codec's Encoder.
[0029] Currently, the high-naturalness speech generation model based on the large language model is trained in two stages: in the first stage, a large base language model is trained with a large amount of general data, and in the second stage, a small amount of data with paralinguistic text annotations is used for fine-tuning. However, the speech generated by the trained speech generation model cannot achieve personalized high-naturalness speech generation.
[0030] The technical solution provided by the present application is intended to solve the above technical problems of the prior art. In this embodiment, the speech framework includes a large language model, a stream model, and a vocoder model of text to acoustic tokens; the large language model of the text to acoustic tokens includes a text embedding module, a text encoding module, a large language model-encoder module, and a large language model-decoder module. The training method includes: using the first speech data and the second speech data as training data, respectively training the large language model, the stream model, and the vocoder model of the text to acoustic tokens in the speech generation framework to obtain the speech generation framework of the first training stage; wherein the first speech data includes the first audio, the text corresponding to the first audio, and the speech representation corresponding to the text corresponding to the first audio; the second speech data includes the second audio under different paralinguistic information annotations, the text corresponding to the second audio, and the speech representation corresponding to the text corresponding to the second audio; using the third speech data as training data, training the text embedding module in the stream model and the large language model of the text to acoustic tokens to obtain a trained speech generation framework; the third speech data includes the third audio of the target speaker, the text corresponding to the third audio, and the speech representation corresponding to the text corresponding to the third audio. This solution uses three different voice data to train and optimize the speech generation framework to reproduce the timbre and rhythm of speakers that have not been seen in training, thereby achieving personalized and highly natural speech generation.
[0031] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.
[0032] Embodiment 1
[0033] Figure 1 A flowchart of a training method for a speech generation framework provided in this application is shown in FIG. Figure 1 As shown, the method includes:
[0034] S101, using the first speech data as training data, performing model training on a large language model, a stream model, and a voice coding model of text to acoustic tokens in a speech generation framework to obtain a speech generation framework of a first training stage; wherein the first speech data includes a first audio, a text corresponding to the first audio, and a speech representation corresponding to the text corresponding to the first audio.
[0035] S102, using the second speech data as training data, performing model training on a large language model, a stream model, and a voice coding model of text to acoustic tokens in a speech generation framework to obtain a speech generation framework of a second training stage; wherein the second speech data includes a second audio labeled with different paralinguistic information, a text corresponding to the second audio, and a speech representation corresponding to the text corresponding to the second audio.
[0036] S103, using the third voice data as training data, performing model training on the text embedding module in the streaming model and the large language model of text to acoustic tokens to obtain a trained speech generation framework; the third voice data includes the third audio of the target speaker, the text corresponding to the third audio, and the speech representation corresponding to the text corresponding to the third audio.
[0037] The speech generation framework is a system architecture for converting input information (usually text) into speech; specifically, the speech generation framework is a framework based on the Cosyvoice algorithm. Figure 2 is a speech generation framework, such as Figure 2 As shown in the figure, x-vec represents the speaker feature vector corresponding to the audio, Text-Y represents the input text sequence, Text-encoder represents the text encoder, Speech X represents the speech representation, Speech Tokennizer represents the speech tagger, Text-to-token LM represents the language model for conversion from text to tokens; flow-matching represents the flow model, and Embeddinglayer represents the embedding layer.
[0038] In practical applications, the input text is input to the text encoder (Text-encoder) to obtain the input text encoding sequence (Text-Y); the speech representation (Speech X) is input to the speech tokenizer (Speech Tokennizer) to obtain the acoustic token; the language model is trained based on the text encoding sequence, the speaker feature vector (x-vec) corresponding to the audio, and the acoustic token; the flow model (flow-matching) is trained based on the speech representation. Among them, the embedding layer (Embedding layer) is a layer of the Text-to-token LM.
[0039] In this example, the first voice data is more than 100,000 hours of voice data. The second voice data is more than 50 hours of voice data with paralinguistic annotations, which can be the sum of voice data with paralinguistic text annotations of many speakers. The third voice data is more than 10 minutes of text-audio normal voice data of the target speaker, that is, voice data with normal emotional intonation.
[0040] Figure 3A flow chart of a training method for a speech generation framework is shown in FIG. Figure 1 ,like Figure 3 As shown; in some examples, the large language model, stream model, and voice coding model of the text to acoustic token in the speech generation framework are trained, specifically including:
[0041] S301, inputting the audio in the training data into a voiceprint model to obtain a speaker feature vector corresponding to the audio; inputting the text corresponding to the audio in the training data into a text encoder to obtain a text encoding sequence corresponding to the audio;
[0042] S302, obtaining a first acoustic token according to the speech representation corresponding to the text corresponding to the audio in the training data; training a large language model from text to acoustic token according to the first acoustic token, the speaker feature vector and the text encoding sequence;
[0043] S303, input the first acoustic token, the speaker feature vector, and the text corresponding to the audio into the large language model of the text to the acoustic token to predict the second acoustic token; train the stream model according to the first acoustic token; input the second acoustic token into the stream model to obtain the second speech representation; train the voice coding model according to the speech representation corresponding to the text corresponding to the audio.
[0044] Among them, the large language model from text to acoustic tokens includes a text embedding module (text-embedding), a text encoder module (text-encode), a large language model-encoder module (llm-encoder), and a large language model-decoder module (llm-decoder); the stream model is a transformation-based function to adjust the parameters of the generation model; in speech generation, the stream model can process speech features at different levels. It can act on speech representations converted from text (such as speech tokens, etc.), or directly operate on acoustic features (such as Mel spectrum, etc.).
[0045] In this example, using the first speech data to train the speech generation framework can enable the speech framework to adapt to different language styles, improve the diversity and accuracy of language generation, and at the same time improve the model's generalization ability and ability to capture speech context information.
[0046] In this example, the speech generation framework is trained using the second speech data, which enables the model to learn the ability to express paralinguistic information. For example, the text corresponding to the audio with paralinguistic annotations is as follows:
[0047] That's kind of scary. [laughter]
[0048] I don’t think I eat too much, [breath] I exercise regularly.
[0049] <laughter> That's really interesting.< / laughter> .
[0050] Team <strong> Unity< / strong> and <strong> toughness< / strong> Helped them win the championship.
[0051] Among them, the design of paralanguage information annotation includes but is not limited to the following four types: laughter, breathing, saying with a smile, and saying with emphasis. Specifically, the types of paralanguage and their meanings are as follows: <laughter> text< / laughter> Said with a smile, <strong> text< / strong> To express emphasis.
[0052] Figure 4 A flow chart of a training method for a speech generation framework is shown in FIG. Figure 2 ,like Figure 4 As shown; in some examples, the text embedding module in the large language model of the convection model and the text to acoustic token is trained, specifically including:
[0053] S401, inputting the third audio into a voiceprint model to obtain a speaker feature vector corresponding to the third audio; inputting the text corresponding to the third audio into a text encoder to obtain a text encoding sequence corresponding to the third audio;
[0054] S402, obtaining a third acoustic token according to the speech representation corresponding to the text corresponding to the third audio; training a text embedding model according to the third acoustic token, a speaker feature vector corresponding to the third audio, and a text encoding sequence corresponding to the third audio;
[0055] S403, predicting the third acoustic token, the speaker feature vector corresponding to the third audio, and the text input corresponding to the third audio into a large language model of the acoustic token to obtain a fourth acoustic token; and training the stream model according to the third acoustic token.
[0056] Among them, using the third voice data to train the text embedding model can transfer the speech rhythm expression characteristics of the target speaker to the text embedding model; using the third voice data to train the streaming model can make the generated target timbre spectrum more like the target speaker.
[0057] In addition, the acoustic tokens and speech representations in this example and the previous example are mel spectrograms. Mel spectrograms are an important intermediate form in the process of speech generation; the model first generates mel spectrograms, and then restores the mel spectrograms to speech waveforms through a vocoder to generate speech.
[0058] In some examples, obtaining a first acoustic token according to a speech representation corresponding to text corresponding to audio in training data includes:
[0059] The speech representation corresponding to the text corresponding to the audio is discretized to obtain a first acoustic token.
[0060] Discretization is the process of converting continuous speech representation into discrete values or symbols. After discretization of the rhythm, timbre, etc. in speech representation in speech generation, the acoustic token of discrete speech can be obtained. The acoustic token is extracted from the discretized speech representation according to a preset rule or algorithm, and this token will serve as important basic information for subsequent speech processing (such as speech synthesis or speech recognition). In addition, discretization of speech representation can more accurately capture and represent the key features of speech.
[0061] In some examples, an input text is input into a large language model, and prompt words are designed according to the input text, so that the large language model inserts paralinguistic expression symbols into the input text according to the prompt words, thereby obtaining an input text with paralinguistic annotations;
[0062] Input the paralinguistically annotated input text into the trained speech generation framework to generate speech results.
[0063] The large language model includes but is not limited to the Yuanjing 72b large language model. The designed prompt words are as follows:
[0064] Text="Please insert paralinguistic information that matches the semantics into the input text according to the following description, as well as interjections that simulate human conversations, such as: well, uh, and then, oh, without increasing the number of words in the original text."
[0065] It should be emphasized that each type of paralinguistic information must be inserted at least once. In practical applications, for example, inserting paralinguistic information:
[0066] Enter text:
[0067] Wow, that sounds great! And there are so many delicious foods there, all kinds of seafood are mouth-watering. \
[0068] Output text:
[0069] <laughter> Wow, that sounds great.< / laughter> ! Well, there are a lot of delicious foods there, and <strong> A variety of seafood makes people salivate< / strong> .
[0070] After obtaining the paralinguistically annotated input text, the trained speech generation framework is input to generate speech. Specifically, when using the speech generation framework to generate speech, a vocoder is applied, and the vocoder restores the speech waveform from the acoustic tokens generated by the speech generation framework to generate speech; wherein the vocoder includes but is not limited to the hifi-gan vocoder.
[0071] The present embodiment provides a training method for a speech generation framework, the speech framework includes a large language model, a stream model, and a vocoder model of text-to-acoustic tokens; the large language model of text-to-acoustic tokens includes a text embedding module, a text encoding module, a large language model-encoder module, and a large language model-decoder module. The training method includes: using first speech data and second speech data as training data, respectively performing model training on the large language model, the stream model, and the vocoder model of text-to-acoustic tokens in the speech generation framework to obtain a speech generation framework in a first training stage; wherein the first speech data includes a first audio, a text corresponding to the first audio, and a speech representation corresponding to the text corresponding to the first audio; the second speech data includes a second audio under different paralinguistic information annotations, a text corresponding to the second audio, and a speech representation corresponding to the text corresponding to the second audio; using third speech data as training data, performing model training on the text embedding module in the stream model and the large language model of text-to-acoustic tokens to obtain a trained speech generation framework; the third speech data includes a third audio of a target speaker, a text corresponding to the third audio, and a speech representation corresponding to the text corresponding to the third audio. This solution uses three different voice data to train and optimize the speech generation framework to reproduce the timbre and rhythm of speakers that have not been seen in training, thereby achieving personalized and highly natural speech generation.
[0072] Embodiment 2
[0073] Figure 5 A structural diagram of a training device for a speech generation framework provided in this application, such as Figure 5 As shown, the framework includes a large language model from text to acoustic tokens, a stream model, and a voice coding model; the large language model from text to acoustic tokens includes a text embedding module, a text encoding module, a large language model-encoder module, a large language model-decoder module, and a training device, including:
[0074] A first training module is used to perform model training on a large language model, a stream model, and a vocoding model of text to acoustic tokens in a speech generation framework using the first speech data as training data to obtain a speech generation framework in a first training phase; wherein the first speech data includes a first audio, a text corresponding to the first audio, and a speech representation corresponding to the text corresponding to the first audio;
[0075] A second training module is used to perform model training on a large language model, a stream model, and a voice coding model of text to acoustic tokens in a speech generation framework using the second speech data as training data to obtain a speech generation framework in a second training phase; wherein the second speech data includes a second audio labeled with different paralinguistic information, a text corresponding to the second audio, and a speech representation corresponding to the text corresponding to the second audio;
[0076] The third training module is used to perform model training on the text embedding module in the stream model and the large language model of text to acoustic tokens using the third voice data as training data to obtain a trained speech generation framework; the third voice data includes the third audio of the target speaker, the text corresponding to the third audio, and the speech representation corresponding to the text corresponding to the third audio.
[0077] In this example, the first voice data is more than 100,000 hours of voice data. The second voice data is more than 50 hours of voice data with paralinguistic annotations, which can be the sum of voice data with paralinguistic text annotations of many speakers. The third voice data is more than 10 minutes of text-audio normal voice data of the target speaker, that is, voice data with normal emotional intonation.
[0078] In some examples, the training module performs model training on a large language model, a streaming model, and a vocoding model of text to acoustic tokens in a speech generation framework for:
[0079] Input the audio in the training data into the voiceprint model to obtain the speaker feature vector corresponding to the audio; input the text corresponding to the audio in the training data into the text encoder to obtain the text encoding sequence corresponding to the audio;
[0080] Obtain a first acoustic token according to the speech representation corresponding to the text corresponding to the audio in the training data; train a large language model from text to acoustic token according to the first acoustic token, the speaker feature vector and the text encoding sequence;
[0081] Input the first acoustic token, speaker feature vector, and text corresponding to the audio into a large language model of text to acoustic token to predict a second acoustic token; train a stream model based on the first acoustic token; input the second acoustic token into the stream model to obtain a second speech representation; and train a voice coding model based on the speech representation corresponding to the text corresponding to the audio.
[0082] Among them, the large language model from text to acoustic tokens includes a text embedding module (text-embedding), a text encoder module (text-encoder), a large language model-encoder module (llm-encoder), and a large language model-decoder module (llm-decoder); the stream model is a transformation-based function to adjust the parameters of the generation model; in speech generation, the stream model can process speech features at different levels. It can act on speech representations converted from text (such as speech tokens, etc.), or directly operate on acoustic features (such as Mel spectrum, etc.).
[0083] In this example, using the first speech data to train the speech generation framework can enable the speech framework to adapt to different language styles, improve the diversity and accuracy of language generation, and at the same time improve the model's generalization ability and ability to capture speech context information.
[0084] In some examples, the training module performs model training on a text embedding module in a streaming model, a text-to-acoustic-tokens large language model, for:
[0085] Inputting the third audio into the voiceprint model to obtain a speaker feature vector corresponding to the third audio; inputting the text corresponding to the third audio into the text encoder to obtain a text encoding sequence corresponding to the third audio;
[0086] Obtaining a third acoustic token according to the speech representation corresponding to the text corresponding to the third audio; training a text embedding module according to the third acoustic token, a speaker feature vector corresponding to the third audio, and a text encoding sequence corresponding to the third audio;
[0087] The third acoustic token, the speaker feature vector corresponding to the third audio, and the text corresponding to the third audio are input into the large language model of the acoustic token to predict the fourth acoustic token; and the stream model is trained according to the third acoustic token.
[0088] Among them, using the third voice data to train the text embedding model can transfer the speech rhythm expression characteristics of the target speaker to the text embedding model; using the third voice data to train the streaming model can make the generated target timbre spectrum more like the target speaker.
[0089] In addition, the acoustic tokens and speech representations in this example and the previous example are mel spectrograms. Mel spectrograms are an important intermediate form in the process of speech generation; the model first generates mel spectrograms, and then restores the mel spectrograms to speech waveforms through a vocoder to generate speech.
[0090] In some examples, the speech representation corresponding to the text corresponding to the audio is discretized to obtain a first acoustic token.
[0091] Discretization is the process of converting continuous speech representation into discrete values or symbols. After discretization of the rhythm, timbre, etc. in speech representation in speech generation, the acoustic token of discrete speech can be obtained. The acoustic token is extracted from the discretized speech representation according to a preset rule or algorithm, and this token will serve as important basic information for subsequent speech processing (such as speech synthesis or speech recognition). In addition, discretization of speech representation can more accurately capture and represent the key features of speech.
[0092] The training device of a speech generation framework provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and this embodiment will not be described in detail here.
[0093] Embodiment 3
[0094] Figure 6 This is a schematic diagram of the structure of a device for a speech generation framework provided in this application. Figure 6 As shown, the electronic device 60 provided in this embodiment includes: at least one processor 601 and a memory 602. Optionally, the device 60 also includes a communication component 603. The processor 601, the memory 602 and the communication component 603 are connected via a bus 604.
[0095] In a specific implementation process, at least one processor 601 executes the computer execution instructions stored in the memory 602, so that at least one processor 601 executes the above method.
[0096] The specific implementation process of the processor 601 can be found in the above method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here.
[0097] The present application also provides a computer program product, including a computer program, which implements the above method when executed by a processor.
[0098] The present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the above method is implemented.
[0099] The above-mentioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special-purpose computer.
[0100] An exemplary readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (Application Specific Integrated Circuits, referred to as: ASIC). Of course, the processor and the readable storage medium can also exist in the device as discrete components.
[0101] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.
[0102] Finally, it should be noted that those skilled in the art will readily conceive of other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. The present invention is intended to cover any variations, uses or adaptations of the present invention, which follow the general principles of the present invention and include common knowledge or customary technical means in the art not disclosed by the present invention, are not limited to the precise structure described above and shown in the drawings, and may be modified and changed in various ways without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.
Claims
1. A method for training a speech generation framework, characterized in that: The framework includes a large language model from text to acoustic tokens, a flow model, and a vocoding model; the large language model from text to acoustic tokens includes a text embedding module, a text encoding module, a large language model-encoder module, and a large language model-decoder module, and the training method includes: Using the first speech data as training data, model training is performed on the large language model of the text to acoustic tokens, the stream model, and the vocoding model in the speech generation framework to obtain a speech generation framework in the first training phase; wherein the first speech data includes a first audio, a text corresponding to the first audio, and a speech representation corresponding to the text corresponding to the first audio; The second speech data is used as training data to perform model training on the large language model of the text to acoustic tokens, the stream model, and the vocoding model in the speech generation framework to obtain a speech generation framework in the second training phase; wherein the second speech data includes the second audio under different paralinguistic information annotations, the text corresponding to the second audio, and the speech representation corresponding to the text corresponding to the second audio; Using the third voice data as training data, model training is performed on the stream model and the text embedding module in the large language model of text to acoustic tokens to obtain a trained speech generation framework; the third voice data includes the third audio of the target speaker, the text corresponding to the third audio, and the speech representation corresponding to the text corresponding to the third audio.
2. The method according to claim 1, characterized in that: The performing model training on the large language model of the text to acoustic tokens, the stream model, and the vocoding model in the speech generation framework specifically includes: Input the audio in the training data into the voiceprint model to obtain the speaker feature vector corresponding to the audio; input the text corresponding to the audio in the training data into the text encoder to obtain the text encoding sequence corresponding to the audio; Obtaining a first acoustic token according to the speech representation corresponding to the text corresponding to the audio in the training data; training the large language model from the text to the acoustic token according to the first acoustic token, the speaker feature vector and the text encoding sequence; The first acoustic token, the speaker feature vector, and the text corresponding to the audio are input into the large language model of the text to acoustic token to predict a second acoustic token; the stream model is trained according to the first acoustic token; the second acoustic token is input into the stream model to obtain a second speech representation; and the voice coding model is trained according to the speech representation corresponding to the text corresponding to the audio.
3. The method according to claim 1, characterized in that The performing model training on the text embedding module in the stream model and the large language model from text to acoustic tokens specifically includes: Inputting the third audio into a voiceprint model to obtain a speaker feature vector corresponding to the third audio; inputting the text corresponding to the third audio into a text encoder to obtain a text encoding sequence corresponding to the third audio; Obtaining a third acoustic token according to the speech representation corresponding to the text corresponding to the third audio; training the text embedding model according to the third acoustic token, the speaker feature vector corresponding to the third audio, and the text encoding sequence corresponding to the third audio; The third acoustic token, the speaker feature vector corresponding to the third audio, and the text corresponding to the third audio are input into the large language model of the text to acoustic token to predict a fourth acoustic token; and the stream model is trained according to the third acoustic token.
4. The method according to claim 2, characterized in that: The obtaining of a first acoustic token according to the speech representation corresponding to the text corresponding to the audio in the training data comprises: The speech representation corresponding to the text corresponding to the audio is discretized to obtain the first acoustic token.
5. The method according to any one of claims 1 to 4, characterized in that: Also includes: Inputting an input text into a large language model, designing prompt words according to the input text, so that the large language model inserts paralanguage expression symbols into the input text according to the prompt words, and obtaining an input text with paralanguage annotations; The input text with the paralanguage annotation is input into the trained speech generation framework to generate a speech result.
6. A training device for a speech generation framework, characterized in that: The framework includes a large language model from text to acoustic tokens, a flow model, and a vocoding model; the large language model from text to acoustic tokens includes a text embedding module, a text encoding module, a large language model-encoder module, and a large language model-decoder module, and the training device includes: A first training module is used to perform model training on the large language model of the text to acoustic tokens, the stream model, and the vocoding model in the speech generation framework using the first speech data as training data to obtain a speech generation framework in a first training phase; wherein the first speech data includes a first audio, a text corresponding to the first audio, and a speech representation corresponding to the text corresponding to the first audio; A second training module is used to perform model training on the large language model of the text-to-acoustic token, the stream model, and the vocoding model in the speech generation framework using the second speech data as training data to obtain a speech generation framework in a second training phase; wherein the second speech data includes the second audio under different paralinguistic information annotations, the text corresponding to the second audio, and the speech representation corresponding to the text corresponding to the second audio; The third training module is used to perform model training on the stream model and the text embedding module in the large language model of text to acoustic tokens using the third voice data as training data to obtain a trained speech generation framework; the third voice data includes the third audio of the target speaker, the text corresponding to the third audio, and the speech representation corresponding to the text corresponding to the third audio.
7. The device according to claim 6, characterized in that The training module performs model training on the large language model of the text to acoustic tokens, the stream model, and the vocoding model in the speech generation framework, for: Input the audio in the training data into the voiceprint model to obtain the speaker feature vector corresponding to the audio; input the text corresponding to the audio in the training data into the text encoder to obtain the text encoding sequence corresponding to the audio; Obtaining a first acoustic token according to the speech representation corresponding to the text corresponding to the audio in the training data; training the large language model of the text to acoustic tokens based on the first acoustic token, the speaker feature vector and the text encoding sequence; Input the first acoustic token, the speaker feature vector, and the text corresponding to the audio into the large language model of the text to acoustic token to predict a second acoustic token; training the flow model based on the first acoustic token; Inputting the second acoustic token into the stream model to obtain a second speech representation; The vocoding model is trained according to the speech representation corresponding to the text corresponding to the audio.
8. The device according to claim 6, characterized in that The training module performs model training on the text embedding module in the stream model and the large language model from text to acoustic tokens, for: Inputting the third audio into a voiceprint model to obtain a speaker feature vector corresponding to the third audio; inputting the text corresponding to the third audio into a text encoder to obtain a text encoding sequence corresponding to the third audio; Obtaining a third acoustic token according to the speech representation corresponding to the text corresponding to the third audio; Training the text embedding module according to the third acoustic token, the speaker feature vector corresponding to the third audio, and the text encoding sequence corresponding to the third audio; Input the third acoustic token, the speaker feature vector corresponding to the third audio, and the text corresponding to the third audio into the large language model of the text to acoustic token to predict a fourth acoustic token; The flow model is trained based on the third acoustic token.
9. The device according to claim 7, characterized in that The training module obtains a first acoustic token according to the speech representation corresponding to the text corresponding to the audio in the training data, and is used to: The speech representation corresponding to the text corresponding to the audio is discretized to obtain the first acoustic token.
10. An electronic device, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 5.
11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 5 when executed by a processor.