Method for multi-stage training of end-to-end voice model based on large model

Abstract

The invention relates to a method for multi-stage training of an end-to-end voice model based on a large model. The method comprises the following steps: S1, collecting a question and answer voice data set; s2, constructing a multi-timbre question and answer voice data set by adopting a self-owned data training TTS model; s3, setting a single-tone question-answer voice data set, and setting a multi-emotion, multi-style, multi-speech-speed and multi-volume voice question-answer pair data set for single-tone questions and answers; s4, splicing the voice token and the text token, and training the large model from voice input to voice output based on the Chinese single-tone question and answer voice data set and the English single-tone question and answer voice data set; s5, making a request based on the multi-tone question and answer voice data set and the selected style, outputting the audio of the tone of the corresponding style, and generating the content according to the answer text set; and S6, adding multi-emotion voice input to form a reasonable voice reply. According to the method, end-to-end operation can be directly completed, time and labor are saved, and the problem of information loss of the cascade model in multiple stages can also be reduced.

Description

Technical Field

[0001] The present invention relates to the field of speech model technology, and in particular to a method for multi-stage training of an end-to-end speech model based on a large model. Background Art

[0002] Currently, most of the speech-in, speech-out models actually running online are cascade models. Information is lost at each stage. For example, the emotion and deeper information in the speech are lost in the general ASR process. It is also difficult to generate the emotion and style of the output speech during output. As a result, it is difficult to generate a model that can understand the content and emotion of your speech and give a correct response.

[0003] Therefore, it is necessary to provide a method for multi-stage training of end-to-end speech models based on large models, which can directly complete the end-to-end training, save time and effort, and reduce the problem of information loss in the cascade model at multiple stages. Summary of the invention

[0004] The purpose of the present invention is to provide a method for multi-stage training of an end-to-end speech model based on a large model, which can directly complete the end-to-end, save time and effort, and can also reduce the problem of information loss in the cascade model at multiple stages.

[0005] In order to solve the problems existing in the prior art, the present invention provides a method for multi-stage training of an end-to-end speech model based on a large model, comprising the following steps:

[0006] S1: Collect question-answering speech dataset;

[0007] S2: Use own data to train the TTS model to construct a multi-timbre question-answering speech dataset;

[0008] S3: Set up a single-timbre question-answering speech dataset, and set up a multi-emotion, multi-style, multi-speed and multi-volume voice question-answering dataset for single-timbre question-answering;

[0009] S4: Concatenate the speech tokens and text tokens, and train the large model from speech input to speech output based on the Chinese monophonic question-answering speech dataset and the English monophonic question-answering speech dataset;

[0010] S5: Make a request based on the multi-timbre question-answering speech dataset and the selected style, output the audio of the corresponding style, and the content is generated according to the answer text set;

[0011] S6: Add multi-emotional voice input to form a reasonable voice response.

[0012] Optionally, in the method for multi-stage training of an end-to-end speech model based on a large model, the method for collecting question-answering speech data is as follows:

[0013] Get the open source voice-to-voice question-answering dataset;

[0014] Based on the open source text question-answering dataset, the corresponding question-answering speech dataset is synthesized using the TTS model;

[0015] There are also Chinese and English datasets.

[0016] Optionally, in the method of multi-stage training end-to-end speech model based on a large model, multi-emotional speech input is added to form a reasonable speech response, in the following manner:

[0017] The speech input is sent to the emtion2vec model;

[0018] Encode the emotion in speech into emotion embedding;

[0019] Input speech token and emtion embedding into the big model,

[0020] The large model understands the semantic and emotional information in speech and responds to speech with emotion and style.

[0021] Optionally, in the method of multi-stage training of an end-to-end speech model based on a large model, the large model actively learns professional knowledge and skills.

[0022] Optionally, in the method of multi-stage training of an end-to-end speech model based on a large model, the collected question-answering speech dataset is used to fine-tune the large model.

[0023] Compared with the prior art, the present invention has the following advantages:

[0024] The present invention not only has professional knowledge in a specific field, but can also conduct conversations in Chinese and English. In addition, it can also switch the tone type. For a single tone, it can answer with different emotions, styles, speaking speeds, and volumes. It is equivalent to a professional voice assistant, does not need to use a cascade model, can reduce information loss and reasoning time, and can also be applied to mobile phones. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 The overall structure diagram of the large model provided for the embodiment of the present invention. DETAILED DESCRIPTION

[0026] The specific implementation of the present invention will be described in more detail below in conjunction with the schematic diagram. The advantages and features of the present invention will become clearer based on the following description. It should be noted that the drawings are all in a very simplified form and are not in exact proportions, and are only used to facilitate and clearly assist in explaining the purpose of the embodiments of the present invention.

[0027] Hereinafter, if the method described herein includes a series of steps, the order in which the steps are presented herein is not necessarily the only order in which the steps may be performed, and some of the steps described may be omitted and / or some other steps not described herein may be added to the method.

[0028] Currently, most of the speech-in, speech-out models actually running online are cascade models. Information is lost at each stage. For example, the emotion and deeper information in the speech are lost in the general ASR process. It is also difficult to generate the emotion and style of the output speech during output. As a result, it is difficult to generate a model that can understand the content and emotion of your speech and give a correct response.

[0029] In order to solve the problems existing in the prior art, the present invention provides a method for multi-stage training of an end-to-end speech model based on a large model, such as Figure 1 As shown, the method comprises the following steps:

[0030] S1: Collect question-answering speech datasets in the following ways: obtain an open source speech-to-speech question-answering dataset; synthesize the corresponding question-answering speech dataset using a TTS model based on an open source text question-answering dataset; and also provide Chinese and English datasets.

[0031] S2: Use own data to train TTS (text to speech) model to construct multi-timbre question-answering speech dataset;

[0032] S3: Set up a single-timbre question-answering speech dataset, and set up a multi-emotion, multi-style, multi-speed and multi-volume voice question-answering dataset for single-timbre question-answering;

[0033] S4: Concatenate the speech tokens and text tokens, and train the large model from speech input to speech output based on the Chinese monophonic question-answering speech dataset and the English monophonic question-answering speech dataset;

[0034] S5: Based on the multi-timbre question-and-answer speech data set and the selected style, a request is made, and the audio of the corresponding style timbre is output, and the content is generated according to the answer text set; in one embodiment: a request such as "Please use a loli voice for dubbing" is added to the text prompt, and then the timbre of the output audio is a loli voice, and the content is synthesized using the pre-designed answer text content.

[0035] S6: Add multi-emotional voice input to form a reasonable voice response.

[0036] Specifically, the method is as follows:

[0037] The speech input is sent to the emtion2vec model;

[0038] Encode the emotion in speech into emotion embedding;

[0039] Input speech token and emtion embedding into the big model,

[0040] The large model understands the semantic and emotional information in speech and responds to speech with emotion and style.

[0041] After such training, we can get an end-to-end speech input and speech output model that has basic Chinese and English conversation capabilities and supports multi-timbre type control and multi-attribute control of high-quality single timbre. In addition, in order to enable the big model to have more professional knowledge and skills, enable the big model to actively learn professional knowledge and skills, and have the capabilities of practitioners in a specific field, the collected and organized text question-answer data or voice question-answer data can be used to fine-tune the big model, so that the big model has relevant capabilities and can be better applied in daily life.

[0042] Fine-tuning refers to the process of further training a pre-trained model in deep learning to adapt it to a specific task or dataset. Fine-tuning enables the model to perform a specific task better by training it with a smaller new dataset based on the pre-trained model, without having to train the entire model from scratch. This method not only saves time and computing resources, but also effectively improves the performance of the model on specific tasks.

[0043] In summary, compared with the prior art, the present invention has the following advantages:

[0044] The present invention not only has professional knowledge in a specific field, but can also conduct conversations in Chinese and English. In addition, it can also switch the tone type. For a single tone, it can answer with different emotions, styles, speaking speeds, and volumes. It is equivalent to a professional voice assistant, does not need to use a cascade model, can reduce information loss and reasoning time, and can also be applied to mobile phones.

[0045] The above is only a preferred embodiment of the present invention and does not limit the present invention in any way. Any technician in the relevant technical field, without departing from the scope of the technical solution of the present invention, makes any form of equivalent replacement or modification to the technical solution and technical content disclosed in the present invention, which does not depart from the content of the technical solution of the present invention and still falls within the protection scope of the present invention.

Claims

1. A method for multi-stage training of an end-to-end speech model based on a large model, characterized in that: The following steps are involved: S1: Collect question-answering speech dataset; S2: Use own data to train the TTS model to construct a multi-timbre question-answering speech dataset; S3: Set up a single-timbre question-answering speech dataset, and set up a multi-emotion, multi-style, multi-speed and multi-volume voice question-answering dataset for single-timbre question-answering; S4: Concatenate the speech tokens and text tokens, and train the large model from speech input to speech output based on the Chinese monophonic question-answering speech dataset and the English monophonic question-answering speech dataset; S5: Make a request based on the multi-timbre question-answering speech dataset and the selected style, output the audio of the corresponding style, and the content is generated according to the answer text set; S6: Add multi-emotional voice input to form a reasonable voice response.

2. The method for multi-stage training of end-to-end speech model based on a large model as claimed in claim 1, characterized in that: The method of collecting question-answering voice data is as follows: Get the open source voice-to-voice question-answering dataset; Based on the open source text question-answering dataset, the corresponding question-answering speech dataset is synthesized using the TTS model; There are also Chinese and English datasets.

3. The method for training an end-to-end speech model in multiple stages based on a large model as claimed in claim 1, characterized in that: Combined with multi-emotional voice input, a reasonable voice response is formed as follows: The speech input is sent to the emtion2vec model; Encode the emotion in speech into emotion embedding; Input speech token and emtion embedding into the big model, The large model understands the semantic and emotional information in speech and responds to speech with emotion and style.

4. The method for training an end-to-end speech model in multiple stages based on a large model as claimed in claim 1, characterized in that: The big model actively learns professional knowledge and skills.

5. The method for training an end-to-end speech model in multiple stages based on a large model as claimed in claim 1, characterized in that: The large model is fine-tuned using the collected question-answering speech dataset.

Images