Speech synthesis method, speech synthesis device and electronic equipment

Abstract

The invention provides a speech synthesis method, a speech synthesis device and electronic equipment. The method comprises the following steps: acquiring text data and audio data; a text encoder is adopted to encode the text data to obtain discrete codes of the text data, and an audio encoder is adopted to encode the audio data to obtain discrete codes of the audio data; according to the speech language model, the discrete codes of the text data and the discrete codes of the audio data, discrete codes of the target synthetic speech are determined, the discrete codes of the text data and the discrete codes of the audio data jointly form a multi-layer discrete codebook, and time delay exists between any two layers of discrete codes; the speech language model is composed of a large language model and a speech synthesis model; and decoding the discrete code of the target synthetic speech by using a decoder to obtain the target synthetic speech. The problem of poor quality of synthesized speech in the prior art is solved.

Description

Technical Field

[0001] The present application relates to the field of speech synthesis technology, and in particular to a speech synthesis method, a speech synthesis device, a computer-readable storage medium, and an electronic device. Background Art

[0002] In recent years, speech synthesis technology has been widely used in daily life and work, significantly improving efficiency and convenience. As a crucial bridge between human and computer interaction, speech synthesis technology has experienced rapid development in the field of artificial intelligence. With advances in deep learning and natural language processing, modern speech synthesis systems are now capable of generating relatively natural and fluent speech, significantly improving user experience and the widespread adoption of voice applications. However, traditional speech synthesis methods are limited by limited speech libraries and inadequate encoding technologies, resulting in semantic inaccuracies in the generated speech and a lack of naturalness in the output, leading to poor speech quality. Summary of the Invention

[0003] The main purpose of this application is to provide a speech synthesis method, a speech synthesis device, a computer-readable storage medium and an electronic device, so as to at least solve the technical problem of poor quality of synthesized speech in the prior art.

[0004] To achieve the above-mentioned purpose, according to one aspect of the present application, a speech synthesis method is provided, comprising: obtaining text data and audio data; encoding the text data using a text encoder to obtain a discrete encoding of the text data, and encoding the audio data using an audio encoder to obtain a discrete encoding of the audio data; determining the discrete encoding of a target synthesized speech based on a speech language model, the discrete encoding of the text data, and the discrete encoding of the audio data, wherein the discrete encoding of the text data and the discrete encoding of the audio data together constitute a multi-layer discrete codebook, there is a time delay between any two layers of the discrete encoding, and the speech language model consists of a large language model and a speech synthesis model; and decoding the discrete encoding of the target synthesized speech using a decoder to obtain the target synthesized speech.

[0005] Optionally, the discrete coding of the target synthesized speech is determined based on the speech language model, the discrete coding of the text data, and the discrete coding of the audio data, including: using the speech synthesis model to reorganize the discrete coding of the text data and the discrete coding of the audio data to determine a multi-layer discrete codebook, wherein the discrete coding of the text data serves as the first layer of the multi-layer discrete codebook, and the discrete coding of the audio data serves as the second to S layers of the multi-layer discrete codebook; and determining the discrete coding of the target synthesized speech based on the large language model and the sum of the discrete codebooks for each time step in the multi-layer discrete codebook.

[0006] Optionally, the speech synthesis model is used to reorganize the discrete coding of the text data and the discrete coding of the audio data to determine a multi-layer discrete codebook, including: setting the discrete coding of the audio data in the first layer to be delayed by one time step compared with the discrete coding of the text data; and setting the discrete coding of the audio data in adjacent layers to differ by one time step.

[0007] Optionally, determining the discrete coding of the target synthesized speech based on the large language model and the sum of the discrete codebooks for each time step in the multi-layer discrete codebook includes: a determining step of determining the sum of the discrete codebooks for a time step subsequent to the preset time step based on the large language model and the sum of the discrete codebooks for a preset time step in the multi-layer discrete codebook; a looping step of looping the determining step until the subsequent time step is the last time step of the multi-layer discrete codebook; and a determining step of determining multiple independent discrete codebooks for each time step in the multi-layer discrete codebook based on the large language model and the multi-layer discrete codebook for each time step, wherein the multiple independent discrete codebooks constitute the discrete coding of the target synthesized speech.

[0008] Optionally, the text data includes multiple languages, the vocabulary of the text encoder includes at least the multiple languages ​​and punctuation marks, and the text encoder is used to encode the text data to obtain discrete encoding of the text data, including: using a BPE encoder to decompose the text in the text data to obtain multiple sub-texts; according to the vocabulary of the BPE encoder, converting the multiple sub-texts into sub-text discrete encodings, and the multiple sub-text discrete encodings constitute the discrete encoding of the text data.

[0009] Optionally, the access network structure of the audio encoder includes a causal convolution layer and an attention mechanism layer, and the audio encoder is used to encode the audio data to obtain a discrete code of the audio data, including: performing feature extraction on the audio data of each time step according to the causal convolution layer to determine the audio local features of the audio data at each time step; determining the attention weight of the audio local features in each time step according to the attention mechanism layer and the audio local features; obtaining an audio feature vector for each time step according to the audio local features in each time step and the attention weight of the audio local features; and converting multiple audio feature vectors into multiple sub-audio discrete codes according to the database of the audio encoder, and multiple sub-audio discrete codes constitute the discrete code of the text data.

[0010] Optionally, the input sequence of the speech language model includes: an audio data sequence vector, the beginning of a text data sequence, the text of the part to be synthesized, the end of the text data sequence and the end of the audio data sequence.

[0011] To achieve the above-mentioned purpose, according to one aspect of the present application, a speech synthesis device is provided, which is applied to the speech synthesis method, and the speech synthesis device includes: an acquisition module for acquiring text data and audio data; an encoding module for encoding the text data using a text encoder to obtain a discrete encoding of the text data, and encoding the audio data using an audio encoder to obtain a discrete encoding of the audio data; a determination module for determining the discrete encoding of the target synthesized speech based on a speech language model, the discrete encoding of the text data, and the discrete encoding of the audio data, wherein the discrete encoding of the text data and the discrete encoding of the audio data together constitute a multi-layer discrete codebook, there is a time delay between any two layers of the discrete encoding, and the speech language model consists of a large language model and a speech synthesis model; a decoding module for decoding the discrete encoding of the target synthesized speech using a decoder to obtain the target synthesized speech.

[0012] According to another aspect of the present application, a computer-readable storage medium is provided, which includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the above-mentioned speech synthesis method.

[0013] According to another aspect of the present application, an electronic device is provided, comprising: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include a method for executing the above-mentioned speech synthesis method.

[0014] Using the technical solution of this application, text data and audio data for speech synthesis are first obtained. The text data is encoded using a text encoder to obtain a discrete encoding of the text data corresponding to the text data. The audio data is encoded using an audio encoder to obtain a discrete encoding of the audio data corresponding to the audio data. Then, based on the discrete encoding of the text data and the audio data obtained above and a speech language model, a discrete encoding of the target synthesized speech is obtained after the text data and audio data are synthesized. When there is a time delay between any two layers of discrete encoding, fewer discrete encodings are required per time step compared to when there is no time delay. This allows the speech language model to predict multiple layers of codebooks corresponding to the same time step in parallel, speeding up decoding and reducing workload. This, in turn, makes the discrete encoding of the target synthesized speech smoother and improves the stability of the model output. Because the discrete encoding of the text data and the discrete encoding of the audio data are multi-layer codebooks, the discrete encoding of the target synthesized speech contains more information than a single-layer codebook, making the discrete encoding of the target synthesized speech more natural after being decoded by the decoder. Incorporating text data into the audio data avoids the problem of inaccurate recognition of audio data alone, ensuring the stability of the speech synthesis process and improving the quality of the synthesized speech. The invention solves the problem of poor quality of synthesized speech in the prior art. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The drawings that constitute part of this application are used to provide a further understanding of this application. The illustrative embodiments of this application and their descriptions are used to explain this application and do not constitute an improper limitation on this application. In the drawings:

[0016] Figure 1 The following is a hardware structure diagram of a mobile terminal for executing a speech synthesis method according to an embodiment of the present application;

[0017] Figure 2 A flowchart of a speech synthesis method according to an embodiment of the present application is shown;

[0018] Figure 3 A schematic diagram of a design of a delayed discrete codebook provided according to an embodiment of the present application is shown;

[0019] Figure 4 A structural diagram of a speech synthesis device provided according to an embodiment of the present application is shown.

[0020] The above drawings include the following reference numerals:

[0021] 102. Processor; 104. Memory; 106. Transmission device; 108. Input / output device. DETAILED DESCRIPTION

[0022] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in this application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0023] In order to enable those skilled in the art to better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0024] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0025] As introduced in the background, speech synthesis technology, as a crucial bridge for human-computer interaction, has experienced rapid development in the field of artificial intelligence in recent years. With advances in deep learning and natural language processing, modern speech synthesis systems are now capable of generating relatively natural and fluent speech, significantly improving user experience and the widespread adoption of voice applications. However, traditional speech synthesis methods are limited by limited speech libraries and inadequate encoding technologies, resulting in semantic instability in the generated speech, a lack of naturalness in the output, and poor quality.

[0026] For ease of description, some nouns or terms involved in the embodiments of the present application are explained below:

[0027] Speech synthesis (Speech Synthesis) technology, also known as text-to-speech (TTS), is a multidisciplinary technology involving computer science, linguistics, acoustics, etc. It aims to convert text information into natural and fluent speech signals, enabling computers to "speak" like humans.

[0028] Large Language Model (LLM): A large language model (LLM) is a natural language processing model based on deep learning. It has a large number of parameters and a wide range of applications. It is primarily used to understand, generate, and process human language. Trained on massive amounts of text data, LLMs are able to capture the complex patterns and semantic structures of language.

[0029] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application.

[0030] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on a mobile terminal as an example, Figure 1 This is a hardware structure block diagram of a mobile terminal of a speech synthesis method according to an embodiment of the present application. Figure 1 As shown, the mobile terminal may include one or more ( Figure 1 Only one is shown) a processor 102 (the processor 102 may include but is not limited to a microprocessor MCU or a programmable logic device FPGA and other processing devices) and a memory 104 for storing data, wherein the mobile terminal may also include a transmission device 106 and an input and output device 108 for communication functions. It will be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the mobile terminal. Figure 1 More or fewer components than shown, or with Figure 1 Different configurations shown.

[0031] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the speech synthesis method in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, implementing the above-mentioned method. The memory 104 may include a high-speed random access memory and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely located relative to the processor 102, and these remote memories can be connected to the mobile terminal via a network. Examples of the above-mentioned networks include but are not limited to the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof. The transmission device 106 is used to receive or send data via a network. Specific examples of the above-mentioned networks may include a wireless network provided by the mobile terminal's communication provider. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, abbreviated as NIC), which can be connected to other network devices via a base station to communicate with the Internet. In one example, the transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0032] In this embodiment, a speech synthesis method running on a mobile terminal, a computer terminal or a similar computing device is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0033] Figure 2 Flowchart of the speech synthesis method according to the embodiment of the present application. Figure 2 As shown, the method includes the following steps:

[0034] Step S1, obtaining text data and audio data;

[0035] Specifically, text data and audio data are collected as input to the speech language model in the subsequent process. Among them, text data can be any text information that needs to be converted into speech, such as sentences or paragraphs input by the user. Audio data is usually sample audio used to guide the speech synthesis style, such as a recording or a voice clip. The step of obtaining text data can be to directly enter text through the user interface, or to read text information from a file. The text data needs to be preprocessed, such as word segmentation and tokenization, for subsequent encoding operations. The step of obtaining audio data can be to import audio data through a microphone or an audio file. Audio preprocessing includes resampling, noise reduction and normalization to ensure data consistency and improve the processing effect of the encoder.

[0036] Step S2, encoding the text data using a text encoder to obtain a discrete code of the text data, and encoding the audio data using an audio encoder to obtain a discrete code of the audio data;

[0037] Specifically, the text encoder converts text information into digital codes to facilitate model understanding and operation, ensuring the semantic stability and accuracy of the synthesized speech. The audio encoder converts audio signals into discrete codebook tokens, covering a rich semantic and acoustic level, providing rich information support for the synthesized speech.

[0038] Step S3: Determine the discrete coding of the target synthesized speech based on the speech language model, the discrete coding of the text data, and the discrete coding of the audio data, wherein the discrete coding of the text data and the discrete coding of the audio data together constitute a multi-layer discrete codebook, there is a time delay between any two layers of discrete coding, and the speech language model consists of a large language model and a speech synthesis model.

[0039] Specifically, during the early stages of training the speech language model, the input sequence contains text tokens and audio tokens, which are reorganized according to a specific delay strategy. The model predicts a multi-layer audio token sequence consisting of text tokens and audio tokens from the previous time step. Through an autoregressive mechanism, it predicts the sum of tokens from all layers of the current time step. The linear decoding layer then restores the sum to individual tokens, forming a complete multi-layer codebook representation. Through the combined effect of the large language model and the speech synthesis model, the discrete encoding of the target speech is generated based on the discrete encoding of text and audio. The use of multi-layer discrete codebooks improves the naturalness of speech synthesis and thus improves speech quality. There is a time delay between any two layers of discrete encoding, which enables the model to predict the multi-layer codebook corresponding to the same time step in parallel, speeding up decoding.

[0040] Step S4: using a decoder to decode the discrete code of the target synthesized speech to obtain the target synthesized speech.

[0041] Specifically, the decoder recovers the synthesized speech's audio signal through inverse operations based on the discrete code of the target synthesized speech, achieving high-quality speech synthesis. The decoder can employ a MIMI decoder, which receives the discrete code of the target synthesized speech as input and uses inverse signal processing techniques, such as overlap-add and waveform generation networks, to convert the discrete codebook tokens into continuous audio features, ultimately generating synthesized audio at a 24k sampling rate.

[0042] In this embodiment, the text data and audio data for speech synthesis are first obtained. The text data is encoded using a text encoder to obtain a discrete encoding of the text data corresponding to the text data. The audio data is encoded using an audio encoder to obtain a discrete encoding of the audio data corresponding to the audio data. Then, based on the discrete encoding of the text data and the audio data obtained above and a speech language model, a discrete encoding of the target synthesized speech is obtained after the text data and audio data are synthesized. When there is a time delay between any two layers of discrete encoding, fewer discrete encodings are required per time step compared to when there is no time delay. This allows the speech language model to predict multiple layers of codebooks corresponding to the same time step in parallel, speeding up decoding and reducing workload. This, in turn, makes the discrete encoding of the target synthesized speech smoother and improves the stability of the model output. Because the discrete encoding of the text data and the discrete encoding of the audio data are multi-layer codebooks, the discrete encoding of the target synthesized speech contains more information than a single-layer codebook, making the discrete encoding of the target synthesized speech more natural after decoding by the decoder. Incorporating text data in addition to audio data avoids the problem of inaccurate recognition of audio data alone, ensuring the stability of the speech synthesis process and improving the quality of the synthesized speech. The invention solves the problem of poor quality of synthesized speech in the prior art.

[0043] In some embodiments, the text encoder may use Byte-Pair Encoding (BPE) technology to encode the input text data and ultimately generate a text token sequence, which contains at least Chinese and English characters, punctuation marks, and special symbols, with a vocabulary of 152,000.

[0044] In some embodiments, the audio encoder can use the MIMI encoder from the MOSHI model to encode audio data and generate a multi-layered audio token sequence. Each layer of audio token sequence contains specific semantic and acoustic information, with a vocabulary size of 2048, for a total of eight codebook layers. This encoder requires an audio sampling rate of 24,000 Hz, and a network structure that includes causal convolutional layers and an attention mechanism to ensure real-time processing and accurate sequence understanding.

[0045] In some optional implementations, determining the discrete encoding of the target synthesized speech according to the speech language model, the discrete encoding of the text data, and the discrete encoding of the audio data in step S3 includes:

[0046] In step S31, a speech synthesis model is used to reorganize the discrete coding of the text data and the discrete coding of the audio data to determine a multi-layer discrete codebook, wherein the discrete coding of the text data serves as the first layer of the multi-layer discrete codebook, and the discrete coding of the audio data serves as the second to S layers of the multi-layer discrete codebook.

[0047] Specifically, if Figure 3 As shown, according to the speech synthesis model, the discrete coding of the text data and the discrete coding of the audio data, the discrete coding of the text data and the discrete coding of the audio data are recombined to form a multi-layer discrete codebook sequence. Figure 3 There are nine layers in total, with the discrete coding of text data (text token) as the first layer, with a delay of 0, and the discrete coding of audio data (audio token) as the second to ninth layers of the multi-layer discrete codebook, with a delay of 1 for the second layer, a delay of 2 for the third layer, and so on. The discrete coding of audio data (audio token) and the discrete coding of text data (text token) can also have more time steps, which are not specifically limited in this application. Through such a reorganization, not only the stability of the discrete coding output of the synthesized speech is ensured, but the combination of text and audio also enriches the richness of the discrete coding of the synthesized speech converted into speech, making it closer to human speech, making the semantics closer to human expression, and improving the quality of speech.

[0048] In step S32, the discrete encoding of the target synthesized speech is determined based on the large language model and the sum of the discrete codebooks at each time step in the multi-layer discrete codebook. Because a multi-layer discrete codebook is used, each discrete codebook layer covers different dimensions of speech information (such as tone and intonation). This enables the model to more meticulously capture and restore speech details during the synthesis process, improving naturalness.

[0049] Specifically, if Figure 3 As shown, the text token of the first time step of the first layer is used as the input of the large language model, and then the large language model predicts the sum of all layers of discrete codebooks at the second time step. Based on the predicted sum of all layers of discrete codebooks at the second time step, the large language model predicts the sum of all layers of discrete codebooks at the third time step, until the sum of all layers of discrete codebooks at the last time step is predicted. Finally, the linear layer in the large language model restores the sum of multiple layers of codebooks into multiple separate discrete codebook tokens. The linear restoration step can be performed after predicting the sum of all layers of discrete codebooks for each time step, or after predicting the sum of all layers of discrete codebooks for all time steps. This is not specifically limited in this application.

[0050] In some optional embodiments, step S31 uses a speech synthesis model to reorganize the discrete coding of text data and the discrete coding of audio data to determine a multi-layer discrete codebook, including: setting the discrete coding of the first layer of audio data to be one time step later than the discrete coding of the text data; setting the discrete coding of the audio data of adjacent layers to be one time step different.

[0051] Specifically, by combining text tokens and audio tokens with a delay strategy, a natural flow between text and speech information and the parallelism of model predictions are ensured. Text tokens serve as the first layer in a multi-layer codebook sequence, while audio tokens start at the second layer. Each layer is delayed by one time step relative to the previous layer, thus forming a delayed multi-layer discrete codebook sequence. Due to this delay design, the model predicts codebook tokens for all layers at the same time step in parallel, significantly speeding up decoding. At the same time, the delay strategy ensures the rational combination of audio and text data, enabling the generation of natural and fluent synthesized speech even without relying on prompt text.

[0052] In some optional implementations, step S32 determines the discrete code of the target synthesized speech based on the large language model and the sum of the discrete codebooks at each time step in the multi-layer discrete codebook, including:

[0053] Determining step: determining the sum of the discrete codebooks for a time step subsequent to the preset time step according to the large language model and the sum of the discrete codebooks for the preset time step in the multi-layer discrete codebook.

[0054] Specifically, the reorganized multi-layer discrete codebook sequence is autoregressively predicted. The model first calculates the sum of all layer codebooks at the first time step of the input (according to Figure 3 From the above, the input is the text token of the first time step to predict the sum of all layers of codebooks in the next time step.

[0055] Loop step: cyclically execute the determination step until the next time step is the last time step of the multi-layer discrete codebook.

[0056] Specifically, step S321 is repeated until the sequence ends. Due to the use of a multi-layer discrete codebook, the model can simultaneously predict semantic and acoustic information, significantly improving the naturalness and quality of the synthesized speech.

[0057] Determining step: Based on the large language model and the multi-layer discrete codebook for each time step, multiple independent discrete codebooks for each time step in the multi-layer discrete codebook are determined, and the multiple independent discrete codebooks constitute discrete coding of the target synthesized speech.

[0058] Specifically, the model's linear decoding layer converts the sum of the predicted multi-layer codebooks into a sequence of independent discrete codebooks, which form the discrete encoding of the target synthesized speech. This conversion process ensures an efficient mapping from high-dimensional predictions to low-dimensional codebooks, providing accurate input for the subsequent audio decoder and further improving the efficiency and effectiveness of speech synthesis.

[0059] In some optional embodiments, the text data includes multiple languages, and the vocabulary of the text encoder includes at least the multiple languages ​​and punctuation marks. Step S2 encodes the text data using the text encoder to obtain a discrete encoding of the text data, including: decomposing a text in the text data using a BPE encoder to obtain a plurality of subtexts. The plurality of subtexts are converted into subtext discrete encodings according to the vocabulary of the BPE encoder, and the plurality of subtext discrete encodings constitute the discrete encoding of the text data.

[0060] Specifically, the text encoder is a Byte-Pair Encoding (BPE) encoder with a vocabulary of 152,000 characters, including at least Chinese characters, English characters, punctuation marks, and some special symbols. This encoder can handle mixed text in at least Chinese, English, and other languages, ensuring the adaptability and robustness of the speech synthesis system in multilingual scenarios. Furthermore, this encoder can handle text in complex multilingual environments, providing flexible and diverse semantic input for subsequent speech synthesis.

[0061] In some optional embodiments, the access network structure of the audio encoder in step S2 includes a causal convolution layer and an attention mechanism layer, and the audio encoder is used to encode the audio data to obtain a discrete code of the audio data, including: extracting features from the audio data at each time step according to the causal convolution layer, and determining the audio local features of the audio data at each time step; this feature extraction process ensures that the model can only use the audio information at the current moment and before for prediction, avoiding information leakage, and ensuring the efficiency and stability of the model in real-time speech synthesis. Based on the attention mechanism layer and the audio local features, the attention weight of the audio local features in each time step is determined; the attention mechanism can highlight the most representative features in the audio signal and suppress the unimportant parts, so that the model can focus on key acoustic information, thereby improving the naturalness and quality of the synthesized speech. Based on the audio local features in each time step and the attention weights of the audio local features, an audio feature vector for each time step is obtained; based on the audio encoder database, the multiple audio feature vectors are converted into multiple sub-audio discrete codes, and the multiple sub-audio discrete codes constitute the discrete code of the text data. Continuous audio feature vectors are mapped to discrete codebook tokens, providing accurate audio representation for subsequent speech synthesis while ensuring that the model can handle the real-time and streaming processing requirements of audio data.

[0062] Specifically, the speech encoder can adopt the MIMI encoder mentioned in the Moshi model. The encoder's vocabulary codebook has a total of 8 layers, and the vocabulary size of each layer is 2048. The input speech sampling rate is required to be 24000Hz. The overall network structure adopts causal convolution layers and attention mechanism layers, so that the model can smoothly encode and decode the input audio, thereby reducing the delay of the entire system.

[0063] In some embodiments, the Qwen2-0.5b model can be used as the base model of the large language model, with a total of 24 transformer layers. For the TTS model (speech language model) based on LLM, the input sequences during training can be roughly divided into two categories. [BT] and [ET] represent the beginning and end of the text sequence, [BA] and [EA] represent the beginning and end of the speech sequence, prompt_text and prompt_audio represent the text sequence and audio sequence corresponding to the prompt speech, text and audio represent the text and audio of the part to be synthesized, speaker_info represents the speaker vector, then the structure of these two types of input sequences is as follows:

[0064] Structure 1: "[BT], prompt_text, text, [ET], [BA], prompt_audio, audio, [EA]." Works like Vall-E and Fish-Speech concatenate the corresponding sequences of the prompt audio and the audio to be synthesized. The input sequence for inference is: "[BT], prompt_text, text, [ET], [BA], prompt_audio."

[0065] Structure 2: "speaker_info, [BT], text, [ET], [BA], audio, [EA]" compresses the speaker information into a 1-D or 32-D speaker vector, which serves as the conditional vector for the LLM input. During the inference phase, the input sequence can be constructed as: "speaker_info, [BT], text, [ET], [BA]".

[0066] This application uses structure 2 as the input of the large language model. The input sequence of the speech language model includes: an audio data sequence vector, the beginning of the text data sequence, the text of the part to be synthesized, the end of the text data sequence, and the end of the audio data sequence. Since the text sequence and audio sequence corresponding to the prompt voice do not need to be input, the dependence on the text corresponding to the prompt voice can be avoided.

[0067] In order to ensure that the semantic ability of the language model is not seriously forgotten during the training process, in some embodiments, text-level generation training is performed at a ratio of 50% during the training of the speech language model to ensure that the language model's "intelligent understanding" ability at the semantic level will help improve the naturalness and emotional richness of the synthesized speech.

[0068] The following is an introduction to the speech synthesis device provided in the embodiments of the present application.

[0069] Figure 4 Schematic diagram of a speech synthesis device according to an embodiment of the present application. Figure 4 As shown, the device includes: an acquisition module 10 for acquiring text data and audio data; an encoding module 20 for encoding the text data using a text encoder to obtain a discrete encoding of the text data, and encoding the audio data using an audio encoder to obtain a discrete encoding of the audio data; a determination module 30 for determining the discrete encoding of the target synthesized speech according to the speech language model, the discrete encoding of the text data, and the discrete encoding of the audio data, wherein the discrete encoding of the text data and the discrete encoding of the audio data together constitute a multi-layer discrete codebook, there is a time delay between any two layers of discrete encoding, and the speech language model consists of a large language model and a speech synthesis model; a decoding module 40 for decoding the discrete encoding of the target synthesized speech using a decoder to obtain the target synthesized speech.

[0070] In this embodiment, the acquisition module first obtains the text data and audio data for speech synthesis. The encoding module encodes the text data using a text encoder to obtain a discrete code corresponding to the text data, and encodes the audio data using an audio encoder to obtain a discrete code corresponding to the audio data. The determination module then obtains a discrete code for the target synthesized speech after synthesizing the text data and audio data based on the obtained discrete codes for the text data and audio data, as well as the speech language model. The presence of a time delay between any two layers of discrete codes reduces the number of discrete codes per time step compared to a scenario without a time delay. This allows the speech language model to concurrently predict multiple layers of codebooks corresponding to the same time step, reducing workload and streamlining the process of obtaining the discrete code for the target synthesized speech, thereby improving the stability of the model output. The decoding module decodes the discrete code for the target synthesized speech using a decoder to obtain the target synthesized speech. Because the discrete codes for the text data and audio data form a multi-layer codebook, the resulting discrete code for the target synthesized speech contains more information about the synthesized speech than a single-layer codebook, resulting in a more natural sound after decoding. The combination of text data and audio data avoids the problem of inaccurate recognition based on audio data alone, ensures the stability of the speech synthesis process, and improves the quality of the synthesized speech, thus solving the problem of poor quality of synthesized speech in the existing technology.

[0071] An embodiment of the present application provides a computer-readable storage medium, which includes a stored program. When the program is executed, the device where the computer-readable storage medium is located is controlled to execute the speech synthesis method.

[0072] Specifically, the speech synthesis method includes:

[0073] Step S1, obtaining text data and audio data;

[0074] Step S2, encoding the text data using a text encoder to obtain a discrete code of the text data, and encoding the audio data using an audio encoder to obtain a discrete code of the audio data;

[0075] Step S3: determining a discrete code for the target synthesized speech based on the speech language model, the discrete code for the text data, and the discrete code for the audio data, wherein the discrete code for the text data and the discrete code for the audio data together constitute a multi-layer discrete codebook, a time delay exists between any two layers of discrete code, and the speech language model comprises a large language model and a speech synthesis model;

[0076] Step S4: using a decoder to decode the discrete code of the target synthesized speech to obtain the target synthesized speech.

[0077] An embodiment of the present application provides an electronic device, comprising a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, at least the following steps are performed:

[0078] Step S1, obtaining text data and audio data;

[0079] Step S2, encoding the text data using a text encoder to obtain a discrete code of the text data, and encoding the audio data using an audio encoder to obtain a discrete code of the audio data;

[0080] Step S3: determining a discrete code for the target synthesized speech based on the speech language model, the discrete code for the text data, and the discrete code for the audio data, wherein the discrete code for the text data and the discrete code for the audio data together constitute a multi-layer discrete codebook, a time delay exists between any two layers of discrete code, and the speech language model comprises a large language model and a speech synthesis model;

[0081] Step S4: using a decoder to decode the discrete code of the target synthesized speech to obtain the target synthesized speech.

[0082] The electronic devices in this article can be servers, PCs, PADs, mobile phones, etc.

[0083] The present application also provides a computer program product, which, when executed on a data processing device, is adapted to execute a program for initializing at least the following method steps:

[0084] Step S1, obtaining text data and audio data;

[0085] Step S2, encoding the text data using a text encoder to obtain a discrete code of the text data, and encoding the audio data using an audio encoder to obtain a discrete code of the audio data;

[0086] Step S3: determining a discrete code for the target synthesized speech based on the speech language model, the discrete code for the text data, and the discrete code for the audio data, wherein the discrete code for the text data and the discrete code for the audio data together constitute a multi-layer discrete codebook, a time delay exists between any two layers of discrete code, and the speech language model comprises a large language model and a speech synthesis model;

[0087] Step S4: using a decoder to decode the discrete code of the target synthesized speech to obtain the target synthesized speech.

[0088] Obviously, those skilled in the art should understand that the modules or steps of the present application described above can be implemented using a general-purpose computing device, they can be concentrated on a single computing device, or distributed across a network composed of multiple computing devices, they can be implemented using program code executable by the computing device, and thus, they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be performed in a different order than herein, or they can be fabricated into separate integrated circuit modules, or multiple modules or steps can be fabricated into a single integrated circuit module for implementation. Thus, the present application is not limited to any specific combination of hardware and software.

[0089] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0090] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0091] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0092] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0093] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0094] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0095] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0096] The speech synthesis method, speech synthesis device, computer-readable storage medium, and electronic device described above can achieve the following beneficial effects:

[0097] 1) The discrete encoding of text data and audio data in the speech synthesis method of this application uses a multi-layer codebook. Compared to a single-layer codebook, the discrete encoding of the target synthesized speech can provide richer information in the synthesized speech, making the discrete encoding of the target synthesized speech more natural after being decoded by the decoder. The incorporation of text data in addition to audio data avoids the problem of inaccurate recognition based solely on audio data, ensures the stability of the speech synthesis process, and improves the quality of the synthesized speech. This solves the problem of poor synthesized speech quality in the prior art.

[0098] 2) In the speech synthesis method of the present application, there is a time delay between any two layers of discrete coding, which can make the discrete coding of each time step less than that without time delay, so that the speech language model can predict the multi-layer codebook corresponding to the same time step in parallel, which speeds up the decoding speed and reduces the workload, thereby making the process of obtaining the discrete coding of the target synthesized speech smoother and improving the stability of the model output.

[0099] 3) The speech synthesis method of the present application does not input the text content corresponding to the prompt speech during the training process, so that the reasoning process does not require the input of the prompt text, thereby enhancing the robustness of the algorithm.

[0100] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0101] The above description is merely a preferred embodiment of the present application and is not intended to limit the present application. Various modifications and variations are possible for those skilled in the art. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present application shall be included within the scope of protection of the present application.

Claims

1. A speech synthesis method, characterized in that: include: Get text data and audio data; Encoding the text data using a text encoder to obtain a discrete encoding of the text data, and encoding the audio data using an audio encoder to obtain a discrete encoding of the audio data; Determining a discrete code for a target synthesized speech based on a speech language model, a discrete code for the text data, and a discrete code for the audio data, wherein the discrete code for the text data and the discrete code for the audio data together constitute a multi-layer discrete codebook, a time delay exists between any two layers of the discrete codes, and the speech language model comprises a large language model and a speech synthesis model; A decoder is used to decode the discrete code of the target synthesized speech to obtain the target synthesized speech.

2. The speech synthesis method according to claim 1, wherein: Determining a discrete encoding of a target synthesized speech according to a speech language model, the discrete encoding of the text data, and the discrete encoding of the audio data includes: Recombining the discrete coding of the text data and the discrete coding of the audio data using the speech synthesis model to determine a multi-layer discrete codebook, wherein the discrete coding of the text data serves as a first layer of the multi-layer discrete codebook, and the discrete coding of the audio data serves as second to S layers of the multi-layer discrete codebook; The discrete code of the target synthesized speech is determined according to the large language model and the sum of the discrete codebooks at each time step in the multi-layer discrete codebook.

3. The speech synthesis method according to claim 2, wherein: The speech synthesis model is used to recombine the discrete coding of the text data and the discrete coding of the audio data to determine a multi-layer discrete codebook, including: Setting the discrete encoding of the audio data in the first layer to be delayed by one time step compared to the discrete encoding of the text data; The discrete encodings of the audio data of adjacent layers are set to differ by one time step.

4. The speech synthesis method according to claim 2, wherein: Determining a discrete code of a target synthesized speech according to the large language model and the sum of the discrete codebooks at each time step in the multi-layer discrete codebook includes: Determining step: determining the sum of the discrete codebook at a time step subsequent to the preset time step based on the large language model and the sum of the discrete codebook at the preset time step in the multi-layer discrete codebook; Looping step: looping and executing the determining step until the next time step is the last time step of the multi-layer discrete codebook; Determining step: determining a plurality of independent discrete codebooks for each time step in the multi-layer discrete codebook based on the large language model and the multi-layer discrete codebook for each time step, wherein the plurality of independent discrete codebooks constitute discrete coding of the target synthesized speech.

5. The speech synthesis method according to claim 1, wherein: The text data includes multiple languages, the vocabulary of the text encoder includes at least the multiple languages ​​and punctuation marks, and the text encoder is used to encode the text data to obtain a discrete code of the text data, including: Decomposing the text in the text data using a BPE encoder to obtain a plurality of subtexts; According to the vocabulary of the BPE encoder, the plurality of subtexts are converted into subtext discrete codes, and the plurality of subtext discrete codes constitute the discrete code of the text data.

6. The speech synthesis method according to claim 1, wherein: The access network structure of the audio encoder includes a causal convolution layer and an attention mechanism layer. The audio encoder is used to encode the audio data to obtain a discrete code of the audio data, including: Performing feature extraction on the audio data at each time step according to the causal convolutional layer to determine the audio local features of the audio data at each time step; Determining, according to the attention mechanism layer and the local audio features, an attention weight of the local audio features in each of the time steps; Obtaining an audio feature vector for each time step according to the audio local features in each time step and the attention weight of the audio local features; According to the database of the audio encoder, the plurality of audio feature vectors are converted into a plurality of sub-audio discrete codes, and the plurality of sub-audio discrete codes constitute the discrete code of the text data.

7. The speech synthesis method according to claim 1, wherein: The input sequence of the speech language model includes: an audio data sequence vector, the beginning of a text data sequence, the text of the part to be synthesized, the end of the text data sequence and the end of the audio data sequence.

8. A speech synthesis device, characterized in that: The speech synthesis method according to any one of claims 1 to 7, wherein the speech synthesis device comprises: An acquisition module, used to acquire text data and audio data; an encoding module, configured to encode the text data using a text encoder to obtain a discrete encoding of the text data, and to encode the audio data using an audio encoder to obtain a discrete encoding of the audio data; a determination module, configured to determine a discrete code for a target synthesized speech based on a speech language model, a discrete code for the text data, and a discrete code for the audio data, wherein the discrete code for the text data and the discrete code for the audio data together constitute a multi-layer discrete codebook, a time delay exists between any two layers of the discrete codes, and the speech language model comprises a large language model and a speech synthesis model; The decoding module is used to decode the discrete code of the target synthesized speech using a decoder to obtain the target synthesized speech.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the speech synthesis method according to any one of claims 1 to 7.

10. An electronic device, characterized in that: include: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include a method for executing the speech synthesis method described in any one of claims 1 to 7.

Images