Speech synthesis, speech recognition method, training method, device, electronic device, storage medium

Abstract

The present application discloses a method for training a speech generation model, comprising: obtaining a training data set; selecting multiple data entries from the training data set to generate multiple batch data entry sets, wherein each batch data entry set contains pure text data entries, pure audio data entries, and text-audio pair data entries; performing discrete coding processing on the data entries in the batch data entry set to generate multiple discrete codes of the data entries; training an autoregressive model according to the multiple discrete codes of the data entries in the multiple batch data entry sets to generate a target speech synthesis model. It also includes a speech recognition model training method, a speech synthesis method, and a speech recognition method. The present application can use a large amount of unsupervised data and a mixed method of supervised data for training, thereby improving the utilization rate of the data and avoiding multi-stage training of the model.

Description

Technical Field

[0001] The present application relates to the technical field of multimedia content processing, and specifically to a training method for a speech synthesis model, a training method for a speech recognition model, a speech synthesis method, a speech recognition method, and related devices, models, electronic devices, and storage media. Background Art

[0002] In the prior art, the TTS (speech synthesis) model is trained based on the data of text and audio segments. In the current training scheme, unlabeled plain text and pure audio cannot be effectively utilized because they have no labels or annotations. In addition, if an unsupervised method is used to train the text encoder and the audio encoder, fine-tuning (adaptation to specific tasks or fields) is required in the downstream tasks. The training method is relatively complicated, and the training cost and difficulty are high.

[0003] The description of the background technology is intended to help understand the relevant technology in the relevant field, and does not constitute an admission that the content of the background technology belongs to the prior art. Summary of the invention

[0004] Therefore, the embodiments of the present application intend to provide a speech synthesis model training method, a speech recognition model training method, a speech synthesis method, a speech recognition method, and related models, electronic devices, and storage media. Through the scheme of the embodiments of the present application, text, audio, or audio-text pairs can be discretely encoded and mixed into the autoregressive model, realizing the mixed input of training data of multiple modes in unsupervised training, thereby reducing the cost and difficulty of model training.

[0005] In a first aspect, an embodiment of the present application provides a speech synthesis model training method, comprising the following steps:

[0006] Acquire a training data set, the training data set comprising a plurality of data entries, the types of the data entries comprising plain text data entries, plain audio data entries, and text-audio pair data entries;

[0007] Selecting a plurality of data entries from the training data set to generate a plurality of batch data entry sets, wherein each batch data entry set includes plain text data entries, plain audio data entries, and text-audio pair data entries, and a ratio among the plain text data entries, plain audio data entries, and text-audio pair data entries in the batch data entry set satisfies a set ratio condition;

[0008] Performing discrete coding processing on the data entries in the batch data entry set to generate a plurality of data entry discrete codes, wherein each data entry discrete code includes a text discrete code and a voice discrete code, and the text discrete code in the data entry discrete code is located before the voice discrete code;

[0009] The autoregressive model is trained according to the discrete encoding of multiple data items in the multiple batch data item sets to generate a target speech synthesis model.

[0010] In some embodiments of the present application, the text discrete code includes text discrete code content, the speech discrete code includes speech discrete code content; the discrete coding of the data entries in the batch data entry set to generate multiple data entry discrete codes includes:

[0011] In response to the data entry being a plain text data entry, discretely encoding the text content in the plain text data entry, obtaining the text discrete encoding content, and setting the voice discrete encoding content to empty;

[0012] In response to the data entry being a pure audio data entry, discretely encoding the audio content in the pure audio data entry, obtaining the speech discrete encoding content, and setting the text discrete encoding content to empty;

[0013] In response to the data entry being a text-audio pair data entry, the text content in the text-audio pair data entry is discretely encoded to obtain the text discretely encoded content, and the audio content in the text-audio pair data entry is discretely encoded to obtain the voice discretely encoded content.

[0014] In some embodiments of the present application, the text discrete code also includes a text discrete code start mark and a text discrete code end mark, and the text discrete code content is located between the text discrete code start mark and the text discrete code end mark; the speech discrete code also includes a speech discrete code start mark and a speech discrete code end mark, and the speech discrete code content is located between the speech discrete code start mark and the speech discrete code end mark.

[0015] In some embodiments of the present application, discrete encoding processing is performed on text content to obtain the discrete encoding content of the text, including:

[0016] The text content is converted into phonemes to generate discrete coding content of the text.

[0017] In some embodiments of the present application, discrete coding processing is performed on the audio content to obtain the discrete coding content of the speech, including:

[0018] The audio content is input into an audio encoder to obtain the discrete speech coding content.

[0019] In a second aspect, an embodiment of the present application provides a speech synthesis method, comprising the following steps:

[0020] Acquire input data, wherein the input data includes a target text;

[0021] Performing discrete coding processing on the input data to obtain a discrete code of the input data, wherein the discrete code of the input data includes a text discrete code and a voice discrete code, and the text discrete code is located before the voice discrete code;

[0022] Inputting the discrete code of the input data into the target speech synthesis model to obtain the discrete code of the output speech;

[0023] The output voice discrete code is input into the voice decoder for decoding to obtain the target output audio.

[0024] In some embodiments of the present application, the input data also includes a target audio segment with a set timbre and ambient sound.

[0025] In some embodiments of the present application, the text discrete code includes text discrete code content, and the speech discrete code includes speech discrete code content;

[0026] When the input data is a target text, the step of performing discrete coding processing on the input data to obtain a discrete code of the input data includes:

[0027] Performing discrete coding processing on the target text, obtaining the discrete coding content of the text, and leaving the discrete coding content of the speech blank;

[0028] When the input data is a target text and a target audio segment, the step of performing discrete coding on the input data to obtain discrete coding of the input data includes:

[0029] The target text is subjected to discrete coding processing to obtain the discrete coding content of the text, and the target audio segment is subjected to discrete coding processing to obtain the discrete coding content of the speech.

[0030] In some embodiments of the present application, when the input data is a target text, the step of inputting the discrete code of the input data into a target speech synthesis model to obtain the output speech discrete code includes:

[0031] Input the discrete code of the input data into the target speech synthesis model, and obtain the output speech discrete code after decoding calculation;

[0032] When the input data is a target text and a target audio segment, the step of inputting the discrete code of the input data into a target speech synthesis model to obtain the output speech discrete code includes:

[0033] The text discrete coding content input data is discretely coded and input into the target speech synthesis model, and the speech discrete coding content is input into the target speech synthesis model in the form of prompt words, and after decoding calculation, the output speech discrete coding is obtained.

[0034] In some embodiments of the present application, the target speech synthesis model is trained by the speech synthesis model training method in any embodiment of the present application.

[0035] In a third aspect, an embodiment of the present application provides a speech recognition model training method, comprising the following steps:

[0036] Acquire a training data set, the training data set comprising a plurality of data entries, the types of the data entries comprising plain text data entries, plain audio data entries, and text-audio pair data entries;

[0037] Selecting a plurality of data entries from the training data set to generate a plurality of batch data entry sets, wherein each batch data entry set includes plain text data entries, plain audio data entries, and text-audio pair data entries, and a ratio among the plain text data entries, plain audio data entries, and text-audio pair data entries in the batch data entry set satisfies a set ratio condition;

[0038] Performing discrete coding processing on the data entries in the batch data entry set to generate a plurality of data entry discrete codes, wherein each data entry discrete code includes a text discrete code and a voice discrete code, and the voice discrete code in the data entry discrete code is located before the text discrete code;

[0039] The autoregressive model is trained according to the discrete encoding of the multiple data items in the multiple batch data item sets to generate a speech recognition model.

[0040] In some embodiments of the present application, the text discrete code includes text discrete code content, the speech discrete code includes speech discrete code content; the discrete coding of the data entries in the batch data entry set to generate multiple data entry discrete codes includes:

[0041] In response to the data entry being a plain text data entry, discretely encoding the text content in the plain text data entry, obtaining the text discrete encoding content, and setting the voice discrete encoding content to empty;

[0042] In response to the data entry being a pure audio data entry, discretely encoding the audio content in the pure audio data entry, obtaining the speech discrete encoding content, and setting the text discrete encoding content to empty;

[0043] In response to the data entry being a text-audio pair data entry, the text content in the text-audio pair data entry is discretely encoded to obtain the text discretely encoded content, and the audio content in the text-audio pair data entry is discretely encoded to obtain the voice discretely encoded content.

[0044] In some embodiments of the present application, the text discrete code also includes a text discrete code start mark and a text discrete code end mark, and the text discrete code content is located between the text discrete code start mark and the text discrete code end mark; the speech discrete code also includes a speech discrete code start mark and a speech discrete code end mark, and the speech discrete code content is located between the speech discrete code start mark and the speech discrete code end mark.

[0045] In some embodiments of the present application, discrete encoding processing is performed on text content to obtain the discrete encoding content of the text, including:

[0046] The text content is converted into phonemes to generate discrete coding content of the text.

[0047] In some embodiments of the present application, discrete coding processing is performed on the audio content to obtain the discrete coding content of the speech, including:

[0048] The audio content is input into an audio encoder to obtain the discrete speech coding content.

[0049] In a fourth aspect, an embodiment of the present application provides a speech recognition method, comprising the following steps:

[0050] Acquire input data, wherein the input data includes target audio;

[0051] Performing discrete coding processing on the input data to obtain discrete coding of the input data, wherein the discrete coding of the input data includes a text discrete coding and a voice discrete coding, and the voice discrete coding is located before the text discrete coding;

[0052] Inputting the discrete code of the input data into the target speech recognition model to obtain the discrete code of the output text;

[0053] The output text discrete code is subjected to inverse discrete coding processing to obtain a target output text.

[0054] In some embodiments of the present application, the text discrete code includes text discrete code content, and the speech discrete code includes speech discrete code content;

[0055] The step of performing discrete coding processing on the input data to obtain discrete coding of the input data includes:

[0056] The target audio is discretely coded to obtain the discrete coded content of the speech, and the discrete coded content of the text is left blank.

[0057] In some embodiments of the present application, the step of inputting the discrete encoding of the input data into a target speech recognition model to obtain the discrete encoding of the output text includes:

[0058] The input data discrete code is input into the target speech recognition model, and after decoding calculation, the output text discrete code is obtained.

[0059] In some embodiments of the present application, the target speech recognition model is trained by the speech recognition model training method in any embodiment of the present application.

[0060] In a fifth aspect, an embodiment of the present application provides a speech synthesis model training device, including a training data set acquisition module, a batch data integration module, a discrete coding processing module and a training module, wherein:

[0061] The training data set acquisition module is configured to acquire a training data set, the training data set comprising a plurality of data entries, the types of the data entries comprising plain text data entries, plain audio data entries and text-audio pair data entries;

[0062] The batch data integration module is configured to select multiple data entries from the training data set to generate multiple batch data entry sets, wherein each batch data entry set contains plain text data entries, plain audio data entries, and text-audio pair data entries, and the ratio between the plain text data entries, plain audio data entries, and text-audio pair data entries in the batch data entry set meets a set ratio condition;

[0063] The discrete coding processing module is configured to perform discrete coding processing on the data entries in the batch data entry set to generate a plurality of data entry discrete codes, wherein each data entry discrete code includes a text discrete code and a voice discrete code, and the text discrete code in the data entry discrete code is located before the voice discrete code;

[0064] The training module trains the autoregressive model according to the discrete encoding of multiple data items in the multiple batch data item sets to generate a target speech synthesis model.

[0065] In a sixth aspect, an embodiment of the present application provides a speech synthesis model, which is trained by the speech synthesis model training method in any embodiment of the present application.

[0066] In a seventh aspect, an embodiment of the present application provides a speech synthesis device, including an input data acquisition module, a discrete coding processing module, a model calculation module and a speech decoder module, wherein:

[0067] The input data acquisition module is configured to acquire input data, wherein the input data includes a target text;

[0068] The discrete coding processing module is configured to perform discrete coding processing on the input data to obtain a discrete coding of the input data, wherein the discrete coding of the input data includes a text discrete coding and a voice discrete coding, and the text discrete coding is located before the voice discrete coding;

[0069] The model calculation module is configured to input the discrete code of the input data into the target speech synthesis model to obtain the output speech discrete code;

[0070] The speech decoder module is configured to perform speech decoding on the output speech discrete code to obtain target output audio.

[0071] In an eighth aspect, an embodiment of the present application provides a speech recognition model training device, including a training data set acquisition module, a batch data integration module, a discrete coding processing module and a training module, wherein:

[0072] The training data set acquisition module is configured to acquire a training data set, the training data set comprising a plurality of data entries, the types of the data entries comprising plain text data entries, plain audio data entries and text-audio pair data entries;

[0073] The batch data integration module is configured to select multiple data entries from the training data set to generate multiple batch data entry sets, wherein each batch data entry set contains plain text data entries, plain audio data entries, and text-audio pair data entries, and the ratio between the plain text data entries, plain audio data entries, and text-audio pair data entries in the batch data entry set meets a set ratio condition;

[0074] The discrete coding processing module is configured to perform discrete coding processing on the data entries in the batch data entry set to generate a plurality of data entry discrete codes, wherein each data entry discrete code includes a text discrete code and a voice discrete code, and the voice discrete code in the data entry discrete code is located before the text discrete code;

[0075] The training module is configured to train the autoregressive model according to the discrete encodings of multiple data entries in the multiple batch data entry sets to generate a speech recognition model.

[0076] In a ninth aspect, an embodiment of the present application provides a speech recognition model, which is trained by the speech recognition model training method in any embodiment of the present application.

[0077] In a tenth aspect, an embodiment of the present application provides a speech recognition device, including an input data acquisition module, a discrete coding processing module, a model calculation module and an inverse discrete coding processing module, wherein:

[0078] The input data acquisition module is configured to acquire input data, wherein the input data includes target audio;

[0079] The discrete coding processing module is configured to perform discrete coding processing on the input data to obtain a discrete coding of the input data, wherein the discrete coding of the input data includes a text discrete coding and a voice discrete coding, and the voice discrete coding is located before the text discrete coding;

[0080] The model calculation module is configured to input the discrete code of the input data into the target speech recognition model to obtain the discrete code of the output text;

[0081] The inverse discrete encoding module is configured to perform inverse discrete encoding processing on the discrete encoding of the output text to obtain a target output text.

[0082] In an eleventh aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, wherein when the program is executed by a processor, the method of any embodiment of the present application is implemented.

[0083] In a twelfth aspect, an embodiment of the present application provides an electronic device, comprising: a processor and a memory storing a computer program, wherein the processor is configured to execute any method of the embodiment of the present application when running the computer program.

[0084] This technical solution provides a speech synthesis model and a speech recognition model training method, wherein pure text, pure audio, and text-audio pair data are mixed in a set ratio to form batch training data, and data items in the batch training data are discretely coded to form data item discrete codes, each of which contains text discrete codes and speech discrete codes, and the data item discrete codes are input into an autoregressive model for training, and finally a target speech synthesis model or speech recognition model is obtained. By mixing and inputting the discrete codes corresponding to pure text, pure audio, and text-audio pairs into an autoregressive model, the model training speed is improved and the training difficulty is reduced. The language understanding and language generation capabilities of the autoregressive model are trained using the discrete codes corresponding to pure text, and the acoustic understanding capabilities of the autoregressive model are improved using the discrete codes corresponding to pure audio, and the model's ability to understand and generate original speech is refined, which can improve the ability to capture emotions or specific timbres. The discrete codes corresponding to text-audio pairs are used to train the autoregressive model's ability to predict the next audio or text using the previous text or audio.

[0085] The embodiment of the present application can adjust the order of text discrete coding and speech discrete coding in the data entry according to the characteristics of the required model. If the required model is a speech synthesis model, the text discrete coding is set before the speech discrete coding. If the required model is a speech recognition model, the speech discrete coding is set before the text discrete coding. For different generation tasks, there is no need to adjust the model architecture. Adjusting the front and back position relationship of the input training data can meet the requirements, improve the reuse characteristics of the model architecture, and have the value of wide promotion. The embodiment of the present application uses a mixed method of unsupervised and supervised data to train the autoregressive model, improve the utilization rate of data, and do not require multi-stage training, which reduces the difficulty of training. The technical solution also provides a speech synthesis method, which uses a discrete coding method to represent audio. Compared with the traditional Mel spectrum, it has more context learning ability. When the audio is generated by the audio prompt word, the generated audio can keep the ambient sound and speaker emotion of the audio prompt word. The speech recognition method in the technical solution can recognize the corresponding text after the input audio is discretely encoded and input into the model, avoiding the use of Mel spectrum to represent audio, and has more context learning ability, and the output text is more accurate.

[0086] Other optional features and technical effects of the embodiments of the present application are partially described below, and partially can be understood by reading this document. BRIEF DESCRIPTION OF THE DRAWINGS

[0087] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use are briefly introduced below. It should be clear that the drawings in the following description only cover some embodiments of the present application, and ordinary technicians in this field can obtain other drawings based on these drawings without performing creative work. The purpose of these drawings is to better illustrate the technical details to help understand the implementation methods of the present application, among which:

[0088] Figure 1 A schematic flow chart showing a method for training a speech synthesis model according to an embodiment of the present application is shown;

[0089] Figure 2 A schematic flow chart showing discrete coding processing in the speech synthesis model training method of an embodiment of the present application;

[0090] Figure 3 A schematic diagram showing the input data of the autoregressive model in the speech synthesis model training method of an embodiment of the present application;

[0091] Figure 4 A schematic flow chart showing a method for training a speech recognition model according to an embodiment of the present application;

[0092] Figure 5 A schematic flow chart showing discrete coding processing in the speech recognition model training method of an embodiment of the present application;

[0093] Figure 6 A schematic diagram showing the input data of the autoregressive model in the speech recognition model training method of an embodiment of the present application;

[0094] Figure 7 A schematic flow chart showing a speech synthesis method according to an embodiment of the present application;

[0095] Figure 8 A schematic diagram showing input data of a speech synthesis model in a speech synthesis method according to an embodiment of the present application;

[0096] Fig. 9 A schematic flow chart showing a speech recognition method according to an embodiment of the present application;

[0097] Fig.10 A schematic diagram showing input data of a speech recognition model in a speech recognition method according to an embodiment of the present application;

[0098] Fig.11 An exemplary structural diagram showing a speech synthesis model training device according to an embodiment of the present application;

[0099] Fig.12 An exemplary structural diagram showing a speech recognition model training device according to an embodiment of the present application;

[0100] Fig.13An exemplary structural diagram showing a speech synthesis device according to an embodiment of the present application;

[0101] Fig.14 An exemplary structural diagram showing a speech recognition device according to an embodiment of the present application;

[0102] Fig.15 An exemplary structural diagram of a speech synthesis model according to an embodiment of the present application is shown;

[0103] Fig.16 An exemplary structural diagram showing a speech recognition model according to an embodiment of the present application;

[0104] Fig.17 An exemplary structural diagram of an electronic device capable of implementing the method according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0105] The exemplary embodiments of the present application will be described in detail below, and the illustrations of the examples have been shown in the accompanying drawings. When referring to the drawings, unless otherwise specified, the same numbers or marks in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with one or more embodiments of this specification. Instead, they are only examples of some aspects of the devices and methods covered by one or more embodiments of this specification as detailed in the claims of this application.

[0106] The term "including" and its variations used in this specification are all intended to be broadly inclusive, i.e., "including but not limited to" the items listed. Unless otherwise stated, the term "or" means "and / or", the term "based on" means relying on, or at least partially relying on, the terms "an example embodiment" and "an embodiment" refer to at least one example embodiment, and the term "another embodiment" means at least one different embodiment. The terms "first", "second", etc. may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0107] In the embodiments of the present application, speech synthesis can generate target audio by inputting text or text plus an audio segment into a speech synthesis model; speech recognition can generate target text by inputting audio into a speech recognition model. In the embodiments of the present application, the speech synthesis model and the speech recognition model can be an autoregressive model or a large language model, and in some embodiments can be a large language model of only an encoder. The architecture of the speech synthesis model or speech recognition model in the embodiments of the present application can be designed with reference to the Vall-E model (https: / / arxiv.org / pdf / 2301.02111.pdf).

[0108] In the embodiments of the present application, TTS (Text To Speech) refers to a speech synthesis technology, ASR (AutoSpeech Recognition) refers to a speech recognition technology, LLM (large language model) refers to a large language model, and prompt refers to a prompt word, which, as part of the language model input, provides speaker-related information to guide the language model to perform tasks.

[0109] In the embodiments of the present application, "timbre" is a basic feature that distinguishes one voice from another, and different people have different timbres. "Rhythm" is one of the attributes of speech, including the pitch and rhythm of musical speech. "Semantics" is one of the attributes of speech, identifying the content of speech.

[0110] In the embodiments of the present application, "model" has the conventional meaning in the field of machine learning. For example, the model can be a machine learning or deep learning model, such as a machine learning or deep learning model including the above-mentioned network or composed of the above-mentioned network.

[0111] In the embodiments of the present application, “loss function” and “loss value” have conventional meanings in the field of machine learning.

[0112] In the embodiments of the present application, "BERT (Bidirectional Encoder Representation from Transformers)" is a pre-trained language representation model. "Prompt" refers to the prompt word, which is part of the language model input and provides relevant information to guide the language model to perform tasks. "Embedding" refers to the data processing step, which converts discrete text and other data into continuous data vectors so that the model can calculate them.

[0113] In the embodiment of the present application, a "phoneme" is the smallest unit of speech divided according to the natural properties of speech, and one pronunciation action forms a phoneme. A "phoneme sequence" is a sequence composed of a number of phonemes.

[0114] In the embodiments of the present application, "batch" refers to batch processing. In the model training process, "batch" refers to a batch of data to be processed before each model improvement, and the model parameters are adjusted according to the output results of this batch of training data; "token" refers to discrete coding, which can refer to word units or phonemes for text and audio discrete coding for audio.

[0115] Some embodiments of the present application provide a speech synthesis model training method, which first discretizes text and speech into tokens; then, during training, the three data forms are mixed in a certain proportion in each batch to train the LLM model, and the text and audio are separated by the start token and stop token of the text and audio. The representation forms of the three data are as follows.

[0116] Plain text data:<text start> <text><text end> <audio start>null <audio end>.

[0117] Pure audio data:<text start> null<text end> <audio start> <audio> <audio end>.

[0118] Audio text pairs:<text start> <text><text end> <audio start> <audio> <audio end>.

[0119] The three types of data are input into the LLM model or the autoregressive model to train and generate a speech synthesis model.

[0120] At the same time, if the audio token is placed before the text token, similar training can be done, and this model can be used as a speech recognition model. That is, the representation of audio and text data is as follows.

[0121] Plain text data: <audio start>null <audio end><text start> <text><text end>.

[0122] Pure audio data: <audio start> <audio> <audio end><text start>null<text end> .

[0123] Audio text pairs: <audio start> <audio> <audio end><text start> <text><text end>.

[0124] Input the three types of data into the LLM model or autoregressive model to train and generate a speech recognition model.

[0125] The embodiments of the present application provide a speech synthesis / recognition model training method and device, a speech synthesis / recognition method, a speech synthesis / recognition model, a synthesis / recognition device, a storage medium and an electronic device. The method, device / model can be implemented with the aid of one or more computers. In some embodiments, the device / model can be implemented by software, hardware, or a combination of software and hardware. In some embodiments, the electronic device or computer can be implemented by a computer described herein or other electronic device that can implement the corresponding function.

[0126] The inventor of this application found that the current TTS training scheme requires paired text and audio (labeled data), and text data without audio or audio data without text cannot be effectively utilized; in addition, the text encoder and audio encoder are trained in an unsupervised manner, and then the downstream tasks are fine-tuned, which is a relatively complex training method and has high costs. In response to this, this application proposes a method of generating a TTS model or an ASR model by training a discrete coding mixed input autoregressive model corresponding to pure text, pure audio, and text and audio data, which simplifies the training process and improves efficiency.

[0127] like Figure 1 As shown, the embodiment of the present application provides a speech synthesis model training method, comprising the following steps:

[0128] S110: Acquire a training data set, the training data set including a plurality of data entries, the types of which include plain text data entries, plain audio data entries, and text and audio pair data entries. The embodiment of the present application reduces the requirements for the training data set, so that the training method can be widely used.

[0129] S120: Select multiple data entries from the training data set to generate multiple batch data entry sets, wherein each batch data entry set contains plain text data entries, plain audio data entries and text-audio pair data entries, and the ratio between the plain text data entries, plain audio data entries and text-audio pair data entries in the batch data entry set meets the set ratio condition.

[0130] Each batch data entry set of the present application includes plain text data entries, pure audio data entries, and text-audio pair data entries. For example, the batch data entry set batch is {{text 1, text 2...text P}, {audio 1, audio 2...audio N}, {text-audio pair 1, text-audio pair 2...text-audio pair M}}, where P is the number of plain texts in the batch, N is the number of pure audios in the batch, and M is the number of text-audio pairs in the batch. In an embodiment of the present application, the set ratio conditions between the plain text data entries, the pure audio data entries, and the text-audio pair data entries can be: 20:1>P:M>10:1, 20:1>N:M>10:1.

[0131] S130: Perform discrete coding processing on the data entries in the batch data entry set to generate multiple data entry discrete codes, wherein each data entry discrete code includes a text discrete code and a voice discrete code, and the text discrete code in the data entry discrete code is located before the voice discrete code.

[0132] In some embodiments of the present application, the text discrete code includes text discrete code content, and the speech discrete code includes speech discrete code content; Figure 2 As shown, the discrete encoding of the data entries in the batch data entry set to generate multiple discrete codes of the data entries includes:

[0133] S131: In response to the data entry being a plain text data entry, discretely encode the text content in the plain text data entry, obtain the text discrete encoding content, and set the voice discrete encoding content to empty.

[0134] For example, a data entry corresponding to a plain text data entry is discretely encoded as {token(text content), token(NULL)}.

[0135] S132: In response to the data entry being a pure audio data entry, discretely encode the audio content in the pure audio data entry, obtain the discretely encoded speech content, and set the discretely encoded text content to be empty.

[0136] For example, a data entry corresponding to a pure audio data entry is discretely encoded as {token(NULL), token(audio content)}.

[0137] S133: In response to the data entry being a text-audio pair data entry, discretely encode the text content in the text-audio pair data entry to obtain the text discretely encoded content, and discretely encode the audio content in the text-audio pair data entry to obtain the speech discretely encoded content.

[0138] For example, the data entry corresponding to the text-audio pair data entry is discretely encoded as {token (text content), token (audio content)}.

[0139] In some embodiments of the present application, in order to facilitate the identification of the starting positions of text and audio, the text discrete code also includes a text discrete code start mark and a text discrete code end mark, and the text discrete code content is located between the text discrete code start mark and the text discrete code end mark; the voice discrete code also includes a voice discrete code start mark and a voice discrete code end mark, and the voice discrete code content is located between the voice discrete code start mark and the voice discrete code end mark.

[0140] For example, the text discrete encoding start flag can be set to token(text start), the text discrete encoding end flag can be set to token(text end), the speech discrete encoding start flag can be set to token(audio start), and the speech discrete encoding end flag can be set to token(audio end). Then the data entry corresponding to the plain text is discretely encoded as {token(text start), token(text content), token(text end), token(audio start), token(NULL), token(audio end)}; the data entry corresponding to the pure audio is discretely encoded as {token(text start), token(NULL), token(text end), token(audio start), token(audio content), token(audio end)}; the data entry corresponding to the text and audio pair is discretely encoded as {token(text start), token(text content), token(text end), token(audio start), token(audio content), token(audio end)}.

[0141] In some embodiments of the present application, discrete encoding processing is performed on text content to obtain the discrete encoding content of the text, including:

[0142] The text content is converted into phonemes to generate discrete coding content of the text.

[0143] In some embodiments of the present application, discrete coding processing is performed on the audio content to obtain the discrete coding content of the speech, including:

[0144] The audio content is input into an audio encoder to obtain the discrete speech coding content.

[0145] S140: Training an autoregressive model according to discrete encodings of a plurality of data items in the plurality of batch data item sets to generate a target speech synthesis model.

[0146] like Figure 3 As shown, the autoregressive model has a first input entry input1 and a second input entry input2. During the training process, the data entry discrete code needs to be input into the autoregressive model, wherein the text discrete code content is input into the first input entry input1, and the speech discrete code content is input into the second input entry input2. After the autoregressive model is calculated, the autoregressive model outputs the audio discrete code, and then the audio discrete code is de-coded to obtain the output audio. Finally, a trained speech synthesis model will be obtained.

[0147] The training and parameter adjustment steps of the autoregressive model in this application are similar to those of the LLM model.

[0148] The speech synthesis model training method of the embodiment of the present application mixes pure text, pure audio, and text-audio pair data in a set ratio to form batch training data, performs discrete coding processing on the data items in the batch training data to form data item discrete coding, each data item discrete coding has a text discrete coding and a speech discrete coding, and inputs the data item discrete coding into the autoregressive model for training, and finally obtains the target speech synthesis model. By mixing and inputting the discrete coding corresponding to pure text, pure audio, and text-audio pair into the autoregressive model, the model training speed is improved and the training difficulty is reduced. The language understanding and language generation capabilities of the autoregressive model are trained using the discrete coding corresponding to pure text, and the acoustic understanding capabilities of the autoregressive model are improved using the discrete coding corresponding to pure audio, and the model's ability to understand and generate original speech is refined, which can improve the ability to capture emotions or specific timbres. The discrete coding corresponding to the text-audio pair is used to train the autoregressive model's ability to predict the next audio segment using the previous segment of text and audio.

[0149] like Figure 4 As shown, the embodiment of the present application provides a speech recognition model training method, comprising the following steps:

[0150] S210: Acquire a training data set, the training data set including a plurality of data entries, the types of which include plain text data entries, plain audio data entries, and text and audio pair data entries. The embodiment of the present application reduces the requirements for the training data set, so that the training method can be widely used.

[0151] S220: Select multiple data entries from the training data set to generate multiple batch data entry sets, wherein each batch data entry set contains plain text data entries, plain audio data entries and text-audio pair data entries, and the ratio between the plain text data entries, plain audio data entries and text-audio pair data entries in the batch data entry set meets the set ratio condition.

[0152] Each batch data entry set of the present application includes plain text data entries, pure audio data entries, and text-audio pair data entries. For example, the batch data entry set batch is {{text 1, text 2...text P}, {audio 1, audio 2...audio N}, {text-audio pair 1, text-audio pair 2...text-audio pair M}}, where P is the number of plain texts in the batch, N is the number of pure audios in the batch, and M is the number of text-audio pairs in the batch. In an embodiment of the present application, the set ratio conditions between the plain text data entries, the pure audio data entries, and the text-audio pair data entries can be: 20:1>P:M>10:1, 20:1>N:M>10:1.

[0153] S230: Perform discrete coding processing on the data entries in the batch data entry set to generate multiple data entry discrete codes, wherein each data entry discrete code includes a text discrete code and a voice discrete code, and the voice discrete code in the data entry discrete code is located before the text discrete code.

[0154] In some embodiments of the present application, the text discrete code includes text discrete code content, and the speech discrete code includes speech discrete code content; Figure 5 As shown, the discrete encoding of the data entries in the batch data entry set to generate multiple discrete codes of the data entries includes:

[0155] S231: In response to the data entry being a plain text data entry, discretely encode the text content in the plain text data entry, obtain the text discrete encoding content, and set the voice discrete encoding content to empty.

[0156] For example, a data entry corresponding to a plain text data entry is discretely encoded as {token(NULL), token(text content)}.

[0157] S232: In response to the data entry being a pure audio data entry, discretely encode the audio content in the pure audio data entry, obtain the discretely encoded speech content, and set the discretely encoded text content to be empty.

[0158] For example, a data entry corresponding to a pure audio data entry is discretely encoded as {token(audio content), token(NULL)}.

[0159] S233: In response to the data entry being a text-audio pair data entry, discretely encode the text content in the text-audio pair data entry to obtain the text discretely encoded content, and discretely encode the audio content in the text-audio pair data entry to obtain the speech discretely encoded content.

[0160] For example, the data entry corresponding to the text-audio pair data entry is discretely encoded as {token (audio content), token (text content)}.

[0161] In some embodiments of the present application, in order to facilitate the identification of the starting positions of text and audio, the text discrete code also includes a text discrete code start mark and a text discrete code end mark, and the text discrete code content is located between the text discrete code start mark and the text discrete code end mark; the voice discrete code also includes a voice discrete code start mark and a voice discrete code end mark, and the voice discrete code content is located between the voice discrete code start mark and the voice discrete code end mark.

[0162] For example, the text discrete encoding start flag can be set to token(text start), the text discrete encoding end flag can be set to token(text end), the speech discrete encoding start flag can be set to token(audio start), and the speech discrete encoding end flag can be set to token(audio end). Then the data entry corresponding to the plain text is discretely encoded as {token(audio start), token(NULL), token(audio end), token(text start), token(text content), token(text end)}; the data entry corresponding to the pure audio is discretely encoded as {token(audio start), token(audio content), token(audio end), token(text start), token(NULL), token(text end)}; the data entry corresponding to the text and audio pair is discretely encoded as {token(audio start), token(audio content), token(audioend), token(text start), token(text content), token(text end)}.

[0163] In some embodiments of the present application, discrete encoding processing is performed on text content to obtain the discrete encoding content of the text, including:

[0164] The text content is converted into phonemes to generate discrete coding content of the text.

[0165] In some embodiments of the present application, discrete coding processing is performed on the audio content to obtain the discrete coding content of the speech, including:

[0166] The audio content is input into an audio encoder to obtain the discrete speech coding content.

[0167] S240: Training an autoregressive model according to discrete encodings of a plurality of data items in the plurality of batch data item sets to generate a speech recognition model.

[0168] like Figure 6 As shown, the autoregressive model has a first input entry input1 and a second input entry input2. During the training process, the data entry discrete code needs to be input into the autoregressive model, wherein the text discrete code content is input into the second input entry input2, and the speech discrete code content is input into the first input entry input1. After the autoregressive model is calculated, the autoregressive model outputs the text discrete code, and then the text discrete code is de-coded to obtain the output text. Finally, a trained speech recognition model can be obtained.

[0169] The training and parameter adjustment steps of the autoregressive model in this application are similar to those of the LLM model.

[0170] The speech recognition model training method of the embodiment of the present application mixes pure text, pure audio, and text-audio pair data in a set ratio to form batch training data, performs discrete coding processing on the data items in the batch training data to form data item discrete coding, each data item discrete coding has a text discrete coding and a speech discrete coding, and inputs the data item discrete coding into the autoregressive model for training, and finally obtains the target speech recognition model. By mixing and inputting the discrete coding corresponding to pure text, pure audio, and text-audio pair into the autoregressive model, the model training speed is improved and the training difficulty is reduced. The language understanding and language generation capabilities of the autoregressive model are trained using the discrete coding corresponding to pure text, and the acoustic understanding capabilities of the autoregressive model are improved using the discrete coding corresponding to pure audio, and the model's ability to understand and generate original speech is refined, which can improve the ability to capture emotions or specific timbres, and the ability to predict the next paragraph of text using the previous paragraph of text and audio is trained using the discrete coding corresponding to the text-audio pair.

[0171] like Figure 7 As shown, the embodiment of the present application provides a speech synthesis method, comprising the following steps:

[0172] S310: Acquire input data, where the input data includes a target text.

[0173] The embodiments of the present application can directly generate target output audio based on the input target text. In some embodiments, in order to obtain output audio with a specific timbre or a specific ambient sound, the input data may further include a target audio segment with a set timbre and ambient sound.

[0174] S320: Perform discrete coding processing on the input data to obtain a discrete code of the input data, where the discrete code of the input data includes a text discrete code and a voice discrete code, and the text discrete code is located before the voice discrete code.

[0175] In some embodiments of the present application, the text discrete code includes text discrete code content, and the speech discrete code includes speech discrete code content.

[0176] When the input data is a target text, the step of performing discrete coding processing on the input data to obtain a discrete code of the input data includes:

[0177] The target text is discretely coded to obtain the text discrete coding content, and the speech discrete coding content is left blank.

[0178] For example, the input data is discretely encoded as {token(target text), token(NULL)}.

[0179] In some embodiments of the present application, the target text is converted into phonemes to generate discrete encoding content of the text.

[0180] When the input data is a target text and a target audio segment, the step of performing discrete coding on the input data to obtain discrete coding of the input data includes:

[0181] The target text is subjected to discrete coding processing to obtain the discrete coding content of the text, and the target audio segment is subjected to discrete coding processing to obtain the discrete coding content of the speech.

[0182] In some embodiments of the present application, the target audio segment is input into an audio encoder to obtain the discrete speech encoding content.

[0183] For example, the input data is discretely encoded as {token (target text), token (target audio segment)}.

[0184] S330: Input the discrete code of the input data into the target speech synthesis model to obtain the discrete code of the output speech.

[0185] In some embodiments of the present application, when the input data is a target text, the step of inputting the discrete code of the input data into a target speech synthesis model to obtain the output speech discrete code includes:

[0186] The input data discrete code is input into the target speech synthesis model, and after decoding calculation, the output speech discrete code is obtained.

[0187] The target speech synthesis model in the embodiment of the present application may be an autoregressive model, which has multiple decoders. The multiple decoders are used to calculate the discrete coding of the input data to obtain the discrete coding of the output speech.

[0188] When the input data is a target text and a target audio segment, the step of inputting the discrete code of the input data into a target speech synthesis model to obtain the output speech discrete code includes:

[0189] The text discrete coding content input data is discretely coded and input into the target speech synthesis model, and the speech discrete coding content is input into the target speech synthesis model in the form of prompt words, and after decoding calculation, the output speech discrete coding is obtained.

[0190] In the embodiment of the present application, the model decoding calculation is adjusted by means of voice prompt words, so that the output audio can better have the timbre, ambient sound, and emotional characteristics of the target audio segment.

[0191] like Figure 8 As shown, the target speech synthesis model has a first input entry input1 and a second input entry input2. During the training process, the data entry discrete code needs to be input into the target speech synthesis model, wherein the text discrete code content is input into the first input entry input1, and the speech discrete code content is input into the second input entry input2. After calculation by the target speech synthesis model, the target speech synthesis model outputs the speech discrete code.

[0192] In some embodiments of the present application, the target speech synthesis model is trained by the speech synthesis model training method in any embodiment of the present application.

[0193] S340: Input the output voice discrete code into the voice decoder for decoding to obtain the target output audio

[0194] like Figure 8 As shown, in some embodiments of the present application, the decoding can be implemented using an audio decoder, and after the output speech is discretely encoded and input into the audio decoder, the output target output audio is obtained.

[0195] The speech synthesis method in the embodiment of the present application uses discrete coding to represent audio, which has better context learning ability than traditional Mel-spectrogram. When the audio is generated by audio prompt words, the generated audio can maintain the ambient sound of the audio prompt words and the speaker's emotions.

[0196] In another embodiment, the inventive concept of the present application may also be applied to speech recognition, such as automatic speech recognition (ASR).

[0197] like Fig. 9 As shown, the embodiment of the present application provides a speech recognition method, comprising the following steps:

[0198] S410: Acquire input data, where the input data includes target audio.

[0199] The embodiment of the present application can directly generate a target text based on the input target audio.

[0200] S420: Perform discrete coding processing on the input data to obtain a discrete code of the input data, wherein the discrete code of the input data includes a text discrete code and a voice discrete code, and the voice discrete code is located before the text discrete code.

[0201] In some embodiments of the present application, the text discrete code includes text discrete code content, and the speech discrete code includes speech discrete code content;

[0202] The step of performing discrete coding processing on the input data to obtain discrete coding of the input data includes:

[0203] The target audio is discretely coded to obtain the discrete coded content of the speech, and the discrete coded content of the text is left blank.

[0204] For example, the input data is discretely encoded as {token(target audio), token(NULL)}.

[0205] S430: Input the discrete code of the input data into the target speech recognition model to obtain the discrete code of the output text.

[0206] In some embodiments of the present application, the step of inputting the discrete encoding of the input data into a target speech recognition model to obtain the discrete encoding of the output text includes:

[0207] The input data discrete code is input into the target speech recognition model, and after decoding calculation, the output text discrete code is obtained.

[0208] like Fig.10 As shown, the target speech recognition model has a first input entry input1 and a second input entry input2. During the training process, the data entry discrete code needs to be input into the target speech recognition model, wherein the text discrete code content is input into the second input entry input2, and the speech discrete code content is input into the first input entry input1. After calculation by the target speech recognition model, the target speech recognition model outputs the text discrete code.

[0209] In some embodiments of the present application, the target speech recognition model is trained by the speech recognition model training method in any embodiment of the present application.

[0210] S440: Perform inverse discrete encoding processing on the discrete encoding of the output text to obtain a target output text.

[0211] like Fig.10 As shown, in some embodiments of the present application, the inverse discrete encoding process can be implemented using a phoneme-to-text converter, and after the output text is discretely encoded and input into the phoneme-to-text converter, the output target output text is obtained.

[0212] The speech recognition method in the embodiment of the present application discretizes the input audio and inputs it into the model, so that the corresponding text can be recognized, avoiding the use of Mel spectrum to represent the audio, having better context learning ability, and outputting more accurate text.

[0213] like Fig.11 As shown, the embodiment of the present application provides a speech synthesis model training device 1100, including a training data set acquisition module 1110, a batch data integration module 1120, a discrete coding processing module 1130 and a training module 1140, wherein:

[0214] The training data set acquisition module 1110 is configured to acquire a training data set, the training data set comprising a plurality of data entries, the types of the data entries comprising plain text data entries, plain audio data entries and text-audio pair data entries;

[0215] The batch data integration module 1120 is configured to select multiple data entries from the training data set to generate multiple batch data entry sets, wherein each batch data entry set contains plain text data entries, plain audio data entries, and text-audio pair data entries, and the ratio between the plain text data entries, plain audio data entries, and text-audio pair data entries in the batch data entry set meets a set ratio condition;

[0216] The discrete coding processing module 1130 is configured to perform discrete coding processing on the data entries in the batch data entry set to generate a plurality of data entry discrete codes, wherein each data entry discrete code includes a text discrete code and a voice discrete code, and the text discrete code in the data entry discrete code is located before the voice discrete code;

[0217] The training module 1140 trains the autoregressive model according to the discrete encodings of the multiple data entries in the multiple batch data entry sets to generate a target speech synthesis model.

[0218] like Fig.12 As shown, the embodiment of the present application provides a speech recognition model training device 1200, including a training data set acquisition module 1210, a batch data integration module 1220, a discrete coding processing module 1230 and a training module 1240, wherein:

[0219] The training data set acquisition module 1210 is configured to acquire a training data set, the training data set comprising a plurality of data entries, the types of the data entries comprising plain text data entries, plain audio data entries and text-audio pair data entries;

[0220] The batch data integration module 1220 is configured to select multiple data entries from the training data set to generate multiple batch data entry sets, wherein each batch data entry set includes plain text data entries, plain audio data entries, and text-audio pair data entries, and the ratio between the plain text data entries, plain audio data entries, and text-audio pair data entries in the batch data entry set meets a set ratio condition;

[0221] The discrete coding processing module 1230 is configured to perform discrete coding processing on the data entries in the batch data entry set to generate a plurality of data entry discrete codes, wherein each data entry discrete code includes a text discrete code and a voice discrete code, and the voice discrete code in the data entry discrete code is located before the text discrete code;

[0222] The training module 1240 is configured to train the autoregressive model according to the discrete encodings of the multiple data entries in the multiple batch data entry sets to generate a speech recognition model.

[0223] like Fig.13 As shown, the embodiment of the present application provides a speech synthesis device 1300, including an input data acquisition module 1310, a discrete coding processing module 1320, a model calculation module 1330 and a speech decoding module 1340, wherein:

[0224] The input data acquisition module 1310 is configured to acquire input data, wherein the input data includes a target text;

[0225] The discrete coding processing module 1320 is configured to perform discrete coding processing on the input data to obtain discrete coding of the input data, wherein the discrete coding of the input data includes a text discrete coding and a voice discrete coding, and the text discrete coding is located before the voice discrete coding;

[0226] The model calculation module 1330 is configured to input the discrete code of the input data into the target speech synthesis model to obtain the output speech discrete code;

[0227] The speech decoder module 1340 is configured to perform speech decoding on the output speech discrete code to obtain target output audio.

[0228] like Fig.14 As shown, the embodiment of the present application provides a speech recognition device 1400, including an input data acquisition module 1410, a discrete coding processing module 1420, a model calculation module 1430 and an inverse discrete coding processing module 1440, wherein:

[0229] The input data acquisition module 1410 is configured to acquire input data, wherein the input data includes target audio;

[0230] The discrete coding processing module 1420 is configured to perform discrete coding processing on the input data to obtain discrete coding of the input data, wherein the discrete coding of the input data includes a text discrete coding and a voice discrete coding, and the voice discrete coding is located before the text discrete coding;

[0231] The model calculation module 1430 is configured to input the discrete code of the input data into the target speech recognition model to obtain the discrete code of the output text;

[0232] The de-discrete encoding module 1440 is configured to perform de-discrete encoding processing on the discrete encoding of the output text to obtain a target output text.

[0233] like Fig.15 As shown, a speech synthesis model 1500 of an embodiment of the present application is also shown, which may include a decoder 1510. The speech synthesis model 1500 can be trained by the speech synthesis model training method in any embodiment of the present application, and the speech synthesis model has a first input port input1 for receiving text discrete coding and a second input port input2 for receiving speech discrete coding.

[0234] like Fig.16 As shown, a speech recognition model 1600 of an embodiment of the present application is also shown, which may include a decoder 1610. The speech synthesis model 1600 can be trained by the speech recognition model training method in any embodiment of the present application, and the speech recognition model 1600 has a first input port input1 for receiving speech discrete coding and a second input port inout2 for receiving text discrete coding. And the features of the device, model, system, components, etc. of the embodiment of the present application can refer to the features of the method, steps, etc. of the embodiment of the present application, and the device, model, and system embodiments can combine the features of the method embodiment to obtain a new embodiment, and vice versa, and will not be repeated here.

[0235] In an embodiment of the present application, an electronic device is provided, comprising: a processor and a memory storing a computer program, wherein the processor is configured to execute the speech synthesis model training method of any embodiment and the speech synthesis method of any embodiment when running the computer program.

[0236] Fig.17 A schematic diagram of an electronic device 1700 that can implement a method or implement an embodiment of the present application is shown, and in some embodiments, more or fewer electronic devices than shown may be included. In some embodiments, a single or multiple electronic devices may be used for implementation. In some embodiments, a cloud or distributed electronic device may be used for implementation.

[0237] Fig.17 A schematic diagram of an electronic device 1700 that can be used to implement the method of an embodiment of the present application or to implement an embodiment of the present application is shown. In some embodiments, the number of electronic devices may be more or less than the number shown. In some embodiments, a single or multiple electronic devices may be used for implementation. In some embodiments, cloud or distributed electronic devices may also be used for implementation.

[0238] like Fig.17 As shown, the electronic device 1700 includes a processor 1710 and a memory 1720. The processor is used to execute programs stored in the memory, which can implement the methods, steps or functions described in the above embodiments when executed by a computer. The processor 1710 may include various types of processors, such as a central processing unit (CPU), a graphics processing unit (GPU), a neural network processor (NPU), a digital signal processor (DSP), etc. The processor 1710 and the memory 1720 are interconnected via a bus 1730. An input / output (I / O) interface and the like can also be connected to the bus 1730.

[0239] The systems, devices, modules or units described in the above embodiments may be implemented by a computer or its associated components. The computer may be, for example, a mobile terminal, a smart phone, a personal computer, a laptop computer, a vehicle-mounted human-computer interaction device, a personal digital assistant, a media player, a navigation device, a game console, a tablet computer, a wearable device, a smart TV, an Internet of Things system, a smart home, an industrial computer, a server or a combination thereof.

[0240] Although not shown, in an embodiment of the present application, a storage medium is provided, the storage medium storing a computer program, the computer program being configured to execute the method of any embodiment when executed.

[0241] The storage media in the embodiments of the present application include permanent and non-permanent, removable and non-removable items that can be used to store information by any method or technology. Examples of storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission medium that can be used to store information that can be accessed by a computing device.

[0242] The methods, programs, systems, devices, etc. of the embodiments of the present application may be executed or implemented in a single or multiple networked computers, or may be practiced in a distributed computing environment. In the embodiments of the present specification, in these distributed computing environments, tasks may be performed by remote processing devices connected via a communication network.

[0243] Those skilled in the art should understand that the embodiments of the present specification may be provided as methods, systems or computer program products. Therefore, those skilled in the art may imagine that the functional modules / units or controllers and related method steps described in the above embodiments may be implemented in software, hardware or a combination of software / hardware.

[0244] Unless explicitly stated, the actions or steps of the methods, programs, and embodiments of the present application do not have to be performed in a specific order and can still achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0245] In this document, multiple embodiments of the present application are described, but for the sake of brevity, the description of each embodiment is not exhaustive, and the same or similar features or parts between the embodiments may be omitted. In this document, "one embodiment", "some embodiments", "example", "specific example", or "some examples" are intended to be applicable to at least one embodiment or example according to the present application, rather than all embodiments. The above terms do not necessarily mean to refer to the same embodiment or example. Under non-contradictory emotions, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples.

[0246] The exemplary systems and methods of the present application have been specifically shown and described with reference to the above embodiments, which are merely examples of the best modes for implementing the present systems and methods. It will be appreciated by those skilled in the art that various changes may be made to the embodiments of the systems and methods described herein when implementing the present systems and / or methods without departing from the spirit and scope of the present application as defined in the appended claims.< / text> < / audio> < / audio> < / audio> < / audio> < / audio> < / audio> < / text> < / audio> < / audio> < / audio> < / audio> < / audio> < / text> < / audio> < / audio> < / audio> < / audio> < / audio> < / text>

Claims

1. A speech synthesis model training method, characterized in that: The steps include: Acquire a training data set, the training data set comprising a plurality of data entries, the types of the data entries comprising plain text data entries, plain audio data entries, and text-audio pair data entries; Selecting a plurality of data entries from the training data set to generate a plurality of batch data entry sets, wherein each batch data entry set includes plain text data entries, plain audio data entries, and text-audio pair data entries, and a ratio among the plain text data entries, plain audio data entries, and text-audio pair data entries in the batch data entry set satisfies a set ratio condition; The data items in the batch data item set are discretely coded to generate a plurality of data item discrete codes, wherein each data item discrete code includes a text discrete code and a voice discrete code, and the text discrete code in the data item discrete code is located before the voice discrete code, wherein the text discrete code includes text discrete code content, and the voice discrete code includes voice discrete code content; the data items in the batch data item set are discretely coded to generate a plurality of data item discrete codes, including: In response to the data entry being a plain text data entry, discretely encoding the text content in the plain text data entry, obtaining the text discrete encoding content, and setting the voice discrete encoding content to empty; In response to the data entry being a pure audio data entry, discretely encoding the audio content in the pure audio data entry, obtaining the speech discrete encoding content, and setting the text discrete encoding content to empty; In response to the data entry being a text-audio pair data entry, discretely encoding the text content in the text-audio pair data entry to obtain the text discrete encoding content, and discretely encoding the audio content in the text-audio pair data entry to obtain the voice discrete encoding content; The autoregressive model is trained according to the discrete encodings of multiple data entries in the multiple batch data entry sets to generate a target speech synthesis model, which includes: inputting the discrete encoding content of the text into the first input entrance of the autoregressive model, and inputting the discrete encoding content of the speech into the second input entrance of the autoregressive model.

2. The method according to claim 1, characterized in that The text discrete code also includes a text discrete code start mark and a text discrete code end mark, and the text discrete code content is located between the text discrete code start mark and the text discrete code end mark; the speech discrete code also includes a speech discrete code start mark and a speech discrete code end mark, and the speech discrete code content is located between the speech discrete code start mark and the speech discrete code end mark.

3. The method according to claim 1, characterized in that Discretely encoding the text content to obtain the discretely encoded content of the text, including: The text content is converted into phonemes to generate discrete coding content of the text.

4. The method according to claim 1, characterized in that: Discretely encoding the audio content to obtain the discrete encoding content of the speech, including: The audio content is input into an audio encoder to obtain the discrete speech coding content.

5. A speech synthesis method, characterized in that: The steps include: Acquire input data, wherein the input data includes a target text; Performing discrete coding processing on the input data to obtain a discrete code of the input data, wherein the discrete code of the input data includes a text discrete code and a voice discrete code, and the text discrete code is located before the voice discrete code; Inputting the discrete code of the input data into a target speech synthesis model to obtain a discrete code of the output speech, wherein the target speech synthesis model is trained by the method according to any one of claims 1 to 4; The output voice discrete code is input into the voice decoder for decoding to obtain the target output audio.

6. The method according to claim 5, characterized in that The input data also includes a target audio segment having a set timbre and ambient sound.

7. The method according to claim 5, characterized in that The text discrete code includes text discrete code content, and the speech discrete code includes speech discrete code content; When the input data is a target text, the step of performing discrete coding processing on the input data to obtain a discrete code of the input data includes: Performing discrete coding processing on the target text, obtaining the discrete coding content of the text, and leaving the discrete coding content of the speech blank; When the input data is a target text and a target audio segment, the step of performing discrete coding on the input data to obtain discrete coding of the input data includes: The target text is subjected to discrete coding processing to obtain the discrete coding content of the text, and the target audio segment is subjected to discrete coding processing to obtain the discrete coding content of the speech.

8. The method according to claim 5, characterized in that When the input data is a target text, the step of inputting the discrete code of the input data into a target speech synthesis model to obtain the output speech discrete code includes: Input the discrete code of the input data into the target speech synthesis model, and obtain the output speech discrete code after decoding calculation; When the input data is a target text and a target audio segment, the step of inputting the discrete code of the input data into a target speech synthesis model to obtain the output speech discrete code includes: The text discrete coding content input data is discretely coded and input into the target speech synthesis model, and the speech discrete coding content is input into the target speech synthesis model in the form of prompt words, and after decoding calculation, the output speech discrete coding is obtained.

9. A speech synthesis model training device, characterized in that: It includes a training data set acquisition module, a batch data integration module, a discrete coding processing module and a training module, among which: The training data set acquisition module is configured to acquire a training data set, the training data set comprising a plurality of data entries, the types of the data entries comprising plain text data entries, plain audio data entries and text-audio pair data entries; The batch data integration module is configured to select multiple data entries from the training data set to generate multiple batch data entry sets, wherein each batch data entry set contains plain text data entries, plain audio data entries, and text-audio pair data entries, and the ratio between the plain text data entries, plain audio data entries, and text-audio pair data entries in the batch data entry set meets a set ratio condition; The discrete coding processing module is configured to perform discrete coding processing on the data entries in the batch data entry set to generate multiple data entry discrete codes, wherein each data entry discrete code includes a text discrete code and a voice discrete code, and the text discrete code in the data entry discrete code is located before the voice discrete code, and the discrete coding processing module is further configured so that the text discrete code includes text discrete coding content, and the voice discrete code includes voice discrete coding content; the discrete coding processing on the data entries in the batch data entry set to generate multiple data entry discrete codes includes: in response to the data entry being a pure text data entry, performing discrete coding processing on the text content in the pure text data entry, obtaining the text discrete coding content, and setting the voice discrete coding content to empty; in response to the data entry being a pure audio data entry, performing discrete coding processing on the audio content in the pure audio data entry, obtaining the voice discrete coding content, and setting the text discrete coding content to empty; in response to the data entry being a text audio pair data entry, performing discrete coding processing on the text content in the text audio pair data entry, obtaining the text discrete coding content, and performing discrete coding processing on the audio content in the text audio pair data entry, obtaining the voice discrete coding content; The training module is configured to train the autoregressive model according to the discrete encodings of multiple data entries in the multiple batch data entry sets to generate a target speech synthesis model, and the training module is configured to input the text discrete encoding content into the first input entrance of the autoregressive model and input the speech discrete encoding content into the second input entrance of the autoregressive model.

10. A speech synthesis model, characterized in that: The speech synthesis model is trained by the method according to any one of claims 1 to 4.

11. A speech synthesis device, characterized in that: It includes an input data acquisition module, a discrete coding processing module, a model calculation module and a speech decoder module, wherein: The input data acquisition module is configured to acquire input data, wherein the input data includes a target text; The discrete coding processing module is configured to perform discrete coding processing on the input data to obtain a discrete coding of the input data, wherein the discrete coding of the input data includes a text discrete coding and a voice discrete coding, and the text discrete coding is located before the voice discrete coding; The model calculation module is configured to input the discrete coding of the input data into a target speech synthesis model to obtain an output speech discrete coding, wherein the target speech synthesis model is trained by the method according to any one of claims 1 to 4; The speech decoder module is configured to perform speech decoding on the output speech discrete code to obtain target output audio.

12. An electronic device, characterized in that: The invention comprises a processor and a memory storing a computer program, wherein the processor is configured to execute the method according to any one of claims 1 to 4 when running the computer program.

13. A storage medium, characterized in that: The storage medium stores a computer program, and the computer program is configured to execute the method according to any one of claims 1 to 4 when executed.

Images