Speech generation method based on LLM and electronic equipment

Abstract

The invention provides a voice generation network constructed based on LLM, and the network comprises a reasoning network constructed based on LLM, and the reasoning network is used for carrying out the reasoning calculation of input voice data, so as to generate a text lexical element sequence corresponding to the voice data in an autoregression manner; a generation network adopting a multi-channel network architecture and used for generating a plurality of voice lexical elements corresponding to the text lexical elements in the text lexical element sequence; the plurality of voice lexical elements have voice features of multiple dimensions learned from the voice data; the plurality of voice word elements are used for decoding to generate second voice data serving as voice response corresponding to the first voice data.

Description

TECHNICAL FIELD

[0001] One or more embodiments of the present specification relate to the technical field of artificial intelligence, and in particular, to a voice generation method based on an LLM and an electronic device. BACKGROUND

[0002] In recent years, end-to-end SLMs (Spoken Language Models) have made significant progress, becoming an important milestone in the field of generative artificial intelligence. Such models, by learning discrete speech tokens generated by a speech encoder, endow LLMs (Large Language Models) with the ability to simultaneously understand and generate text and speech, providing strong support for achieving natural and fluent human-computer interaction. In application scenarios such as voice assistants and real-time dialogue agents, which have high requirements for response speed and interaction naturalness, SLMs show great application potential.

[0003] Although existing research has made breakthroughs in the modeling capabilities of SLMs, current open-source SLMs still face a key bottleneck, which is that the response delay is too high, seriously affecting the real-time nature of voice interaction and user experience. This problem is particularly prominent in high-frequency, low-latency dialogue scenarios. SUMMARY

[0004] Therefore, one or more embodiments of the present specification provide technical solutions as follows: According to a first aspect of one or more embodiments of the present specification, a voice generation method based on an LLM is provided, which is applied to a voice generation network constructed based on an LLM; the voice generation network includes an inference network constructed based on an LLM; and a generation network with a multi-channel network architecture; wherein the inference network is used to perform inference calculation on input voice data to generate a sequence of text tokens corresponding to the voice data in a self-recurrent manner; the generation network is used to generate a plurality of speech tokens corresponding to the text tokens in the sequence of text tokens; the plurality of speech tokens have a plurality of dimensions of speech features learned from the voice data; and the method comprises: obtaining input first voice data; inputting the first voice data into the inference network, performing inference calculation on the first voice data by the inference network to generate a sequence of text tokens corresponding to the first voice data in a self-recurrent manner, and outputting the text tokens in the generated sequence of text tokens; inputting the text token into the generation network further to generate, by the generation network, a plurality of speech tokens corresponding to the text token; wherein the plurality of speech tokens are used to decode to generate second speech data as a speech response corresponding to the first speech data.

[0005] Optionally, the inputting the text token into the generation network further to generate, by the generation network, a plurality of speech tokens corresponding to the text token comprises: obtaining the first hidden vector related to the text token generated by the inference network, and obtaining a historical sequence of speech tokens that has been generated by the generation network; inputting the first hidden vector related to the text token and an embedding vector corresponding to the historical sequence of speech tokens into the generation network further to generate, by the generation network, a plurality of speech tokens corresponding to the text token in an autoregressive manner; wherein the embedding vector comprises a vector obtained by vector fusion on token embedding vectors corresponding to each historical speech token in the historical sequence of speech tokens.

[0006] Optionally, before the inputting the first hidden vector related to the text token and the embedding vector corresponding to the historical sequence of speech tokens into the generation network, the method further comprises: vector fusing the embedding vector corresponding to the text token and the first hidden vector related to the text token.

[0007] Optionally, the generation network comprises a speech generator taking the output of the inference network as input, and a multi-layer quantization network taking the output of the speech generator as input; wherein the speech generator is configured to generate a hidden vector related to a plurality of speech tokens corresponding to the text token at a next time step of a time step at which the text token is located; and the multi-layer quantization network respectively maintains a codebook for storing speech features learned from input speech data; and each layer of the quantization network maintains a codebook storing speech features of different dimensions. the inputting the first hidden vector related to the text token and the embedding vector corresponding to the historical sequence of speech tokens into the generation network further to generate, by the generation network, a plurality of speech tokens corresponding to the text token in an autoregressive manner comprises: The first hidden vector related to the text token and the embedding vector corresponding to the historical speech token sequence are further input to the speech generator, so that the speech generator generates a second hidden vector related to a plurality of speech tokens corresponding to the text token at a next time step of a time step where the text token is located in an autoregressive manner based on the first hidden vector and the embedding vector, and the second hidden vector output by the speech generator is input to the multi-layer quantization network respectively, so that the second hidden vector is quantized into a speech token with a speech feature saved in a codebook maintained by the multi-layer quantization network based on the codebook respectively, to obtain a plurality of speech tokens corresponding to the multi-layer quantization network one by one.

[0008] Optionally, the generation network further comprises a multi-level prediction network; wherein a first level prediction network in the multi-level prediction network comprises a predictor taking the output of the speech generator as input; and a multi-layer quantization network taking the output of the predictor as input; other level prediction networks in the multi-level prediction network except the first level prediction network comprise a predictor taking the output of the predictor in the last level prediction network as input; and a multi-layer quantization network taking the output of the predictor as input; The first hidden vector related to the text token and the embedding vector corresponding to the historical speech token sequence are further input to the speech generator, so that the speech generator generates a second hidden vector related to a plurality of speech tokens corresponding to the text token at a next time step of a time step where the text token is located in an autoregressive manner based on the first hidden vector and the embedding vector, and the second hidden vector output by the speech generator is input to the multi-layer quantization network respectively, so that the second hidden vector is quantized into a speech token with a speech feature saved in a codebook maintained by the multi-layer quantization network based on the codebook respectively, to obtain a plurality of speech tokens corresponding to the multi-layer quantization network one by one, comprising: The first hidden vector related to the text token and the embedding vector corresponding to the historical speech token sequence are further input to the speech generator, so that the speech generator generates a second hidden vector related to a plurality of speech tokens corresponding to the text token at a next time step T+1 of a time step T where the text token is located in an autoregressive manner based on the first hidden vector and the embedding vector, and the second hidden vector output by the speech generator is input to the multi-layer quantization network respectively, so that the second hidden vector is quantized into a speech token with a speech feature saved in a codebook maintained by the multi-layer quantization network based on the codebook respectively, to obtain a plurality of speech tokens corresponding to the time step T+1; The second hidden vector output by the speech generator is further input to a first predictor in the first-level prediction network, so that the first predictor further generates a third hidden vector related to a next time step T+2 of the time step T+1 based on the second hidden vector, and the third hidden vector is input to a quantization network of multiple layers in the first-level prediction network respectively, so that the quantization network of multiple layers quantizes the third hidden vector into speech tokens with speech features saved in a codebook maintained by the quantization network of multiple layers respectively, to obtain a plurality of speech tokens corresponding to the time step T+2; The third hidden vector output by the first predictor is further input to a second predictor in a second-level prediction network, so that the second predictor further generates a fourth hidden vector related to a next time step T+3 of the time step T+2 based on the third hidden vector, and the fourth hidden vector is input to a quantization network of multiple layers in the second-level prediction network respectively, so that the quantization network of multiple layers quantizes the fourth hidden vector into speech tokens with speech features saved in a codebook maintained by the quantization network of multiple layers respectively, to obtain a plurality of speech tokens corresponding to the time step T+3; By analogy, the quantization network of multiple layers in the last-level prediction network quantizes the hidden vector output by the predictor in the last-level prediction network into speech tokens with speech features saved in a codebook maintained by the quantization network of multiple layers respectively, to obtain a plurality of speech tokens corresponding to the time step T+1+N; wherein N represents the number of levels of the multi-level prediction network.

[0009] Optionally, the quantization network of multiple layers comprises a classification network of multiple layers; and the codebook maintained by the quantization network comprises a weight matrix used for classification maintained by the classification network of multiple layers. The quantization network of multiple layers quantizes the input hidden vector into speech tokens with speech features saved in a codebook maintained by the quantization network of multiple layers respectively, to obtain a plurality of speech tokens corresponding to the quantization network of multiple layers one by one, comprising: The classification network of multiple layers classifies the input hidden vector based on a weight matrix maintained by the classification network of multiple layers respectively, to obtain a plurality of speech tokens as classification results corresponding to the classification network of multiple layers one by one.

[0010] Optionally, the speech generator and the predictor comprise a neural network based on a transformer architecture; and the classification network of multiple layers comprises a classification head of multiple layers. Optionally, the speech generator and the predictor comprise a decoder part in the transformer architecture.

[0011] Optionally, the speech generation network further comprises a pre-trained decoding network; the decoding network comprises a decoding network based on a multi-codebook architecture. The method further comprises: further inputting the plurality of speech tokens into the decoding network to decode the plurality of speech tokens into second speech data by the decoding network as a speech response corresponding to the first speech data.

[0012] Optionally, the speech generation network further comprises an encoding network; the encoding network is configured to encode speech data samples used for training the speech generation network into a plurality of speech token samples based on a plurality of maintained codebooks as supervision items corresponding to the plurality of speech tokens generated by the generation network.

[0013] Optionally, the decoding network comprises a multi-codebook supported audio codec network; the encoding network comprises an encoder part in the audio codec network; the decoding network comprises a decoder part in the audio codec network.

[0014] Optionally, the multi-codebook supported audio codec network comprises an XY-Tokenizer audio codec network.

[0015] Optionally, the training process of the speech generation network comprises: a first stage of training: removing the inference network or freezing parameters contained in the inference network, pre-training the generation network independently; wherein a pre-training task corresponding to the pre-training comprises a generation task of generating a plurality of speech tokens corresponding to an input text token sample; a second stage of training: removing the generation network or freezing parameters contained in the generation network, fine-tuning the inference network independently; wherein a fine-tuning training task corresponding to the fine-tuning training comprises a generation task of performing inference calculation on an input speech data sample to generate a text token sequence corresponding to the speech data sample in an autoregressive manner; a third stage of training: fine-tuning the speech generation network as a whole; wherein a fine-tuning training task corresponding to the fine-tuning training comprises an end-to-end generation task of generating speech data as a speech response corresponding to an input speech data sample.

[0016] According to a second aspect of one or more embodiments of the present specification, a speech generation network constructed based on LLM is further proposed, comprising: an inference network constructed based on LLM, configured to perform inference calculation on input first speech data to generate a text token sequence corresponding to the first speech data in an autoregressive manner; The generation network adopts a multi-channel network architecture to generate a plurality of speech tokens corresponding to the text tokens in the text token sequence; the plurality of speech tokens have a plurality of dimensions of speech features learned from the first speech data; and the plurality of speech tokens are used to decode the second speech data as the speech response corresponding to the first speech data.

[0017] According to a third aspect of one or more embodiments of the present specification, an electronic device is further provided, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor implements the steps of the method according to any one of the first aspect by running the executable instructions.

[0018] According to a fourth aspect of one or more embodiments of the present specification, a computer-readable storage medium is further provided, which stores computer instructions, and the instructions are executed by a processor to implement the steps of the method according to any one of the first aspect.

[0019] According to a fifth aspect of one or more embodiments of the present specification, a computer program product is further provided, which comprises computer program / instructions, and the computer program / instructions are executed by a processor to implement the steps of the method according to any one of the first aspect.

[0020] In the above embodiments, since the text tokens in the text token sequence generated by the end-to-end speech generation network can be directly modeled into a plurality of speech tokens with a plurality of dimensions of speech features learned from the first speech data, the plurality of speech tokens containing rich speech features can be directly decoded into the second speech data as the speech response corresponding to the input first speech data by a lightweight decoding network, without the need to reconstruct the speech data based on a single speech token. This not only can significantly reduce the delay in generating the speech response for the input first speech data, improve the fluency and response efficiency of the speech interaction, but also can improve the authenticity of the generated speech data. BRIEF DESCRIPTION OF DRAWINGS

[0021] Figure 1 is an exemplary embodiment of a schematic diagram of the architecture of a speech generation system; Figure 2 is an exemplary embodiment of a flowchart of a speech generation method based on LLM; Figure 3 is an exemplary embodiment of a network architecture diagram of a speech generation network constructed based on a pre-trained completed LLM; Figure 4 is an exemplary embodiment of a network architecture diagram of an inference network; Figure 5 is a network architecture diagram of a network provided by an example embodiment for generating a network; Figure 6 is another network architecture diagram of a network provided by an example embodiment for generating a network; Figure 7A is another network architecture diagram of a network provided by an example embodiment for generating a network; Figure 7B is another network architecture diagram of a network provided by an example embodiment for generating a network; Figure 8A is another network architecture diagram of a network provided by an example embodiment for generating a network; Figure 8B is another network architecture diagram of a network provided by an example embodiment for generating a network; Figure 9 is a structural schematic diagram of an electronic device provided by an example embodiment; Figure 10 is a block diagram of a voice generation device based on an LLM provided by an example embodiment. DETAILED DESCRIPTION

[0022] In order for those skilled in the art to better understand the technical solutions in the specification, the technical solutions in the specification will be clearly and completely described below in conjunction with the drawings in the specification. Obviously, the described embodiments are only part of the embodiments of the specification, not all. Based on the embodiments in the specification, all other embodiments obtained by those of ordinary skill in the art without creative labor should be within the scope of protection of the specification.

[0023] The user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the specification are information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation portal for user to choose authorization or refusal.

[0024] In the related art, the SLM mainly adopts the following two technical paradigms in multi-modal modeling: One is the technical paradigm of "original speech multi-modal". Under this technical paradigm, the vocabulary table (also known as the word table) of the LLM will be directly expanded to contain discrete tokens in the speech corpus, so that the LLM can uniformly process text and speech information. However, this technical paradigm usually needs to train the LLM from scratch on a large scale, and the cost of training is relatively high.

[0025] Secondly, there is the technical paradigm of modal alignment. In this paradigm, an independent speech input / output module can be introduced on the basis of a pre-trained LLM through an efficient cross-modal alignment mechanism. This avoids large-scale training of the LLM from scratch and effectively reuses the powerful capabilities of the existing LLM.

[0026] This specification will focus on the more flexible and scalable technical paradigm of "modal alignment" to build SLM.

[0027] However, the mainstream SLM built using the "modal alignment" technical paradigm has made significant progress in achieving joint generation of text and speech, but it still suffers from inference latency, which limits its application in real-time interactive scenarios.

[0028] The delays in these models mainly stem from two key aspects: First, these models typically use an autoregressive approach to generate text and speech words one by one, resulting in an inherent sequential nature in the output process.

[0029] Secondly, while the speech lexical units generated by these models can effectively capture the semantic content of speech, they lack rich audio features (such as limited information on acoustic details), making them difficult to use directly for high-quality speech generation. Therefore, it is necessary to rely on an independent reconstruction model (such as a flow-matching model) to complete the fine modeling from discrete audio lexical units to continuous acoustic representation.

[0030] This process often requires multiple iterative calculations. While it simplifies the speech modeling task of language models to some extent and improves the quality of generated speech, it also introduces a serious computational bottleneck, resulting in high inference overhead and long processing time, which becomes the main source of latency in the entire system.

[0031] Furthermore, the autoregressive generation mechanism, which generates text and speech lexical units one by one, further exacerbates the aforementioned system latency problem.

[0032] For example, since each speech word can only predict the next word after the previous word has been generated, this serial generation method causes the response time to increase linearly with the output length, making it difficult to meet the requirements of low latency and high throughput real-time dialogue.

[0033] Moreover, after reconstructing discrete audio lexical units into continuous acoustic representations through the reconstruction model, the reconstructed continuous acoustic representations usually cannot truly reflect the characteristics of the audio data because the reconstruction process lacks real audio features from the input speech data, resulting in insufficient audio realism.

[0034] Based on this, this specification proposes a novel SLM architecture that adopts the technical paradigm of "modal alignment".

[0035] Under this novel SLM architecture, the SLM can specifically be a speech generation network built on a pre-trained LLM; the speech generation network can specifically include an inference network built on an LLM; and a generation network employing a multi-channel network architecture.

[0036] The aforementioned inference network can be used to perform inference calculations on the input speech data to generate a text word sequence corresponding to the speech data in an autoregressive manner; the aforementioned generation network can be used to generate multiple speech words corresponding to the text words in the aforementioned text word sequence; the multiple speech words have multiple dimensions of speech features learned from the input speech data.

[0037] When training the aforementioned speech generation network or generating speech based on the trained speech generation network, first speech data can be acquired and input into the aforementioned inference network. The inference network performs inference calculations on the first speech data to generate a text word sequence corresponding to the first speech data in an autoregressive manner, and outputs the text words in the generated text word sequence. Then, the text words output by the inference network can be further input into the aforementioned generation network to generate multiple speech words corresponding to the text words. These multiple speech words are used to decode and generate second speech data as a speech response corresponding to the aforementioned first speech data.

[0038] In the above embodiments, since the text words in the text word sequence generated by reasoning and computing on the input first speech data can be directly modeled into multiple speech words with multiple dimensions of speech features learned from the first speech data, a lightweight decoding network can directly decode these multiple speech words containing rich speech features into second speech data as a speech response corresponding to the input first speech data, without the need to reconstruct speech data based on a single speech word. This not only significantly reduces the latency in generating a speech response for the input first speech data and improves the fluency and response efficiency of voice interaction, but also enhances the realism of the generated speech data.

[0039] Figure 1 This is a schematic diagram of the architecture of a speech generation system provided in an exemplary embodiment.

[0040] like Figure 1As shown, the system may include a server 11, a network 12, and several electronic devices, such as a PC (Personal Computer) 13, a mobile phone 14, etc.

[0041] Server 11 can be a physical server containing an independent host, or it can be a virtual server hosted in a host cluster. During operation, server 11 can run a server-side program to implement LLM-based speech generation functions. For example, when server 11 runs the program, it can act as the server for the speech generation system.

[0042] PC23 and mobile phone 14 are just some of the types of electronic devices that users can use. In reality, users can obviously also use electronic devices such as tablets, laptops, PDAs (Personal Digital Assistants), wearable devices (such as smart glasses, smartwatches, etc.), etc., and one or more embodiments in this specification do not limit this. During operation, the electronic device can also run a client-side program to implement LLM-based speech generation functions. For example, when the electronic device runs the program, it can act as a client for the speech generation system. The client-side application of the aforementioned speech generation system can be launched and run on the electronic device. This client-side program can be a native application installed on the electronic device, or it can be a mini-program, quick app, or other similar form. Of course, when using web technologies such as HTML5 or similar, the relevant functions can be implemented through a browser-displayed page. This browser can be a standalone browser application or a browser module embedded in some applications.

[0043] As for the network 12 that enables interaction between electronic devices such as PC13 and mobile phone 14 and server 11, communication can be achieved using either wired or wireless networks, depending on the communication methods supported by the respective electronic devices. This specification does not impose any restrictions on this. For example, PC13 can support both wired and wireless communication, so it can use either wired or wireless networks as needed. Mobile phone 14 typically only supports wireless communication, so it can use a wireless network for communication.

[0044] The technical solution of this specification will be described in detail below with reference to the accompanying drawings.

[0045] Please see Figure 2 , Figure 2 This document presents a flowchart of an LLM-based speech generation method, which can be applied to speech generation networks built on LLM. The method includes the following execution process: Step 202: Obtain the first input voice data; The aforementioned speech generation network can specifically include any type of deep learning network built on top of a pre-trained LLM, employing the modality alignment technique mentioned above. In this case, the constructed deep learning network can be used as an SLM for speech generation tasks.

[0046] Please see Figure 3 , Figure 3 This specification illustrates a network structure diagram of an SLM constructed using a modal alignment technique.

[0047] like Figure 3 As shown, under this novel SLM architecture, the SLM can specifically be a speech generation network built on a pre-trained LLM.

[0048] like Figure 3 As shown, the speech generation network may specifically include an inference network based on LLM; and may also include a generation network employing a multi-channel network architecture.

[0049] The aforementioned inference network, serving as the input processing end of the speech generation network, can specifically be used to perform inference calculations on the input speech data to generate a text word sequence corresponding to the speech data in an autoregressive manner, and output the text words in the generated text word sequence.

[0050] For example, the aforementioned inference network can use an LLM as the inference engine and adopt an autoregressive approach to generate the next text word in time sequence based on the already generated text words.

[0051] The aforementioned generator network, as the output processing end of the speech generation network, can specifically be used to generate multiple speech units corresponding to the text units in the aforementioned text unit sequence.

[0052] Please continue reading Figure 3 In some embodiments, the generative network employing a multi-channel network architecture typically includes multiple levels of generative networks. These multiple levels of generative networks can continue to use the text lexical sequences output by the inference network as input, generating corresponding speech lexical units in parallel. In this case, each level of the generative network can output a speech lexical unit, thereby obtaining multiple speech lexical units that correspond one-to-one with the multiple levels of the generative network.

[0053] It should be noted that the number of levels in the generative network can be flexibly designed based on actual needs. For example, in some embodiments, the generative network may specifically include an 8-level network. In practical applications, if it is desired that the final output speech units of the generative network contain richer speech features, the number of levels in the generative network can be significantly increased; correspondingly, if it is desired that the generative network has better inference performance, the number of levels in the generative network can be appropriately controlled.

[0054] In some embodiments, the aforementioned plurality of speech units may have multiple dimensions of speech features learned from the input speech data.

[0055] Among these multi-dimensional speech features, in addition to the basic semantic features learned from the input speech data, multiple acoustic features can also be included. In practical applications, these acoustic features can specifically include various types of acoustic features that can reflect the acoustic details of the input audio data.

[0056] For example, please continue to see Figure 3 In some embodiments, the speech units output by the first-level generator network in the multi-level generator network may specifically have semantic features learned from the input speech data, while the speech units output by the other levels of generator networks besides the first-level generator network may each have different types of acoustic features learned from the input speech data.

[0057] In this specification, when performing a speech generation task using the aforementioned speech generation network, the first speech data input to the speech generation network can be obtained first.

[0058] It should be noted that, in practical applications, the speech generation task performed using the aforementioned speech generation network can specifically refer to the speech generation task performed during the offline training phase of the speech generation network, or it can refer to the speech generation task performed during the online inference phase based on the trained speech generation network after the network has been trained.

[0059] Correspondingly, if the aforementioned speech generation task is performed during the offline training phase of the speech generation network, the first speech data obtained at this time can specifically be speech data samples used to train the speech generation network.

[0060] If the aforementioned speech generation task is a speech generation task performed during the online inference phase based on the trained speech generation network, then the first speech data obtained at this time can specifically be a speech data sample used for online inference. In other words, the first speech data obtained at this time can serve as an inference sample.

[0061] In other words, the first speech data obtained in step 102 can be used as training samples during the training phase of the speech generation network; and during the online inference phase using the trained speech generation network, it can be used as inference samples.

[0062] Step 204: Input the first speech data into the inference network, and the inference network performs inference calculations on the first speech data to generate a text word sequence corresponding to the first speech data in an autoregressive manner, and output the text words in the generated text word sequence. In this specification, after obtaining the first speech data input to the speech generation network, the first speech data can be input to the inference network, and the inference network can perform inference calculations on the first speech data to generate a text word sequence corresponding to the first speech data in an autoregressive manner, and output the text words in the generated text word sequence. For example, the inference network can generate text words in the text word sequence corresponding to the first speech data one by one in an autoregressive manner and in chronological order, and output the generated text words in chronological order.

[0063] Before inputting the acquired first speech data into the aforementioned inference network, the first speech data can be preprocessed to convert it into a representation suitable for LLM processing.

[0064] Please see Figure 4 , Figure 4 This is a network architecture diagram of an inference network shown in this specification.

[0065] like Figure 4 As shown, in some embodiments, in addition to including an LLM as an inference engine, the above-mentioned inference network may also include a speech encoder and a downsampling adapter before the LLM.

[0066] In this scenario, after acquiring the first speech data, it can be preprocessed using the speech encoder and downsampling adapter to transform it into a continuous representation. This continuous representation is then further input into the LLM (Local Level Module), which performs inference calculations on the first speech data to generate a text lexical sequence corresponding to the first speech data in an autoregressive manner. By processing the first speech data into a continuous representation, this representation can effectively capture the high-quality features of the input audio data.

[0067] For example, in practical applications, the first speech data can be first input into a speech encoder, which encodes the raw audio data into a continuous feature representation rich in semantic and acoustic features. This continuous feature representation output from the speech encoder is then further input into the aforementioned downsampling adapter, which compresses the continuous feature representation in the time dimension, reducing its length so that its length matches that of the continuous feature representation in text form. The compressed continuous feature representation output from the downsampling adapter is then further input into the aforementioned LLM, which performs inference calculations on the continuous feature representation to generate a text word sequence corresponding to the first speech data in an autoregressive manner. For example, the LLM can generate text words in the text word sequence corresponding to the first speech data one by one in chronological order and output the generated text words sequentially in chronological order.

[0068] It should be noted that, in practical applications, both the aforementioned speech encoder and downsampling adapter can be pre-trained speech encoders and downsampling adapters. Alternatively, in practical applications, the aforementioned speech encoder and downsampling adapter can also be non-pre-trained speech encoders and downsampling adapters, and can be trained together with the LLM (Limited Linear Model) as the inference engine, as part of the aforementioned inference network.

[0069] Please continue reading Figure 4 In some embodiments, an upsampling adapter corresponding to the downsampling adapter can also be introduced after the LLM in the inference network described above.

[0070] In this case, the text words in the text word sequence corresponding to the first speech data output by the LLM can be further input into the upsampling adapter, which will then extend the continuous feature representation in the time dimension, increasing its length so that the length of the continuous feature representation can match the length of the continuous feature representation in speech form.

[0071] Step 206: The text lexical units are further input into the generation network to generate multiple speech lexical units corresponding to the text lexical units; wherein, the multiple speech lexical units are used to decode and generate second speech data as a speech response corresponding to the first speech data.

[0072] In this specification, during the process of generating the text word sequence corresponding to the first speech data in an autoregressive manner, the inference network can further input the text words in the already generated text word sequence output by the inference network into the generation network, so that the generation network can generate multiple speech words corresponding to the text words using a multi-channel network architecture.

[0073] In some embodiments, during the process of generating a sequence of text words corresponding to the first speech data in an autoregressive manner, the inference network typically generates a hidden vector (also called a hidden vector or hidden state vector) corresponding to each new text word in chronological order after it generates a new text word. This vector serves as context for generating the next text word.

[0074] For example, the content generated by LLM using an autoregressive approach can be represented as: in, This represents the continuous feature representation input to the LLM (i.e., the compressed continuous feature representation output from the downsampling adapter). This represents the sequence of words in the text generated by LLM; This represents the hidden vector generated by LLM. T represents the length of the input speech data sequence; N represents the length of the text word sequence and hidden vector sequence generated by LLM. For example, This represents the continuous feature representation input to the first time step of the LLM; This represents the text terms generated by LLM corresponding to the first time step; This represents the hidden vector generated by LLM corresponding to the text tokens at the first time step.

[0075] It should be noted that the aforementioned hidden vector is an intermediate feature representation of arbitrary form generated within each layer of the LLM network when processing the input sequence. These intermediate feature representations are typically high-dimensional vector representations rich in semantic information. In practical applications, this hidden vector can provide rich semantic embedding information for subsequent speech generation.

[0076] For example, LLM typically uses the hidden vector output from the last Transformer layer (multi-head attention + feedforward network) in the Transformer architecture to predict the next text terminology.

[0077] In this case, when the text words output by the above-mentioned inference text are further input into the generator network to generate multiple speech words corresponding to the text words, the first hidden vector related to the text words generated by the inference network can be obtained, and the historical speech word sequence generated by the generator network can be obtained; wherein, the historical speech word sequence refers to the sequence of all speech words generated by the generator network in chronological order before this.

[0078] For example, assuming the current time step is T, and the speech lexicon to be generated at time step T, the obtained historical speech lexicon sequence specifically includes the speech lexicons corresponding to each time step from time step 1 to time step T-1 that have already been generated. That is, the obtained historical speech lexicon sequence can be represented as... .

[0079] Then, the first hidden vector associated with the text word and the embedding vector corresponding to the above-mentioned historical speech word sequence can be further input into the generator network so that the generator network can also generate speech words corresponding to the text word in an autoregressive manner.

[0080] For example, the generative network can also generate speech units corresponding to each text unit in the above text unit sequence in an autoregressive manner, and output the generated speech units sequentially in chronological order. In this case, the speech units generated by the generative network corresponding to each text unit in the above text unit sequence can also form a speech unit sequence in chronological order.

[0081] It is important to emphasize that the speech units generated by the generative network corresponding to each text unit in the aforementioned text unit sequence can specifically include multiple speech units with multiple dimensions of speech features learned from the aforementioned first speech data. In other words, the speech unit ultimately generated by the generative network corresponding to any text unit in the aforementioned text unit sequence can specifically be a set of multiple speech units.

[0082] For example, taking the example of a multi-level generative network that generates speech units sequentially in a certain time order, the final speech unit generated by the generative network corresponding to the text unit can be a sequence of speech units output by each level of the generative network in the order of their generation. Of course, in practical applications, the multi-level generative network can also generate speech units in parallel. In this case, the final speech unit generated by the generative network corresponding to the text unit can be a set of speech units generated in parallel by the multi-level generative network.

[0083] It should be noted that the embedding vector corresponding to the aforementioned historical speech word sequence can specifically include a vector obtained by vector fusion of the word embedding vectors corresponding to each historical speech word in the historical speech word sequence.

[0084] For example, in practical applications, the vector obtained by adding the embedding vectors of each historical lexical unit in the historical speech lexical sequence together can be used as the fused vector. Of course, besides vector addition, other methods can also be used for fusion in practical applications; for example, vector concatenation.

[0085] In some embodiments, before further inputting the first hidden vector associated with the aforementioned text lexical units and the embedding vector corresponding to the aforementioned historical speech lexical unit sequence into the generation network, the embedding vector corresponding to the aforementioned text lexical units can also be calculated, and then the embedding vector corresponding to the aforementioned text lexical units and the first hidden vector associated with the aforementioned text lexical units are fused. The specific method of fusion will not be elaborated further. In this way, the aforementioned first hidden vector can be feature-enhanced.

[0086] In some embodiments, when the text words output from the inference text are further input into the generator network to generate multiple speech words corresponding to the text words, in addition to further inputting the first hidden vector related to the text words and the embedding vector corresponding to the historical speech word sequence into the generator network, in practical applications, the embedding vector corresponding to the text words is also used instead of the first hidden vector. The embedding vector corresponding to the text words and the embedding vector corresponding to the historical speech word sequence are further input into the generator network so that the generator network generates multiple speech words corresponding to the text words in an autoregressive manner.

[0087] In some embodiments, the above-described generator network includes a multi-level generator network, specifically including a common speech generator and a multi-level quantization network that takes the output of the common speech generator as input.

[0088] Please see Figure 5 , Figure 5 This is a network architecture diagram of a generated network shown in this specification.

[0089] like Figure 5 As shown, the generative network can specifically include a common speech generator that takes the output of the aforementioned inference network as input; and a multi-layered quantization network that takes the output of the speech generator as input. In other words, the multi-level generative network comprised of this generative network can specifically consist of a common speech generator and a multi-layered quantization network that takes the output of the speech generator as input.

[0090] Specifically, the aforementioned generative network can be a neural network that employs a multi-codebook architecture and, based on the maintained multiple codebooks, quantizes the text lexical units output by the aforementioned inference network into multiple speech lexical units.

[0091] Under this architecture, the aforementioned common speech generator can be used to generate hidden vectors related to multiple speech words corresponding to the text word at the next time step after the time step where the text word is located. Each layer of the quantization network can maintain a codebook to store speech features learned from the input speech data. Each layer of the quantization network can then quantize the input hidden vectors into speech words with the speech features stored in the codebook, based on the maintained codebook.

[0092] In other words, the aforementioned common speech generator, acting as the input processing end, is responsible for generating hidden vectors related to multiple speech words corresponding to the text word at the next time step after the time step where the text word is located. The quantization networks at each layer, acting as the output processing end, are responsible for quantizing the hidden vectors generated by the speech generator into multiple speech words corresponding to the text word based on the maintained codebook.

[0093] It should be noted that the aforementioned common speech generator generates multiple speech vectors corresponding to the text word at the next time step after the time step where the text word is located. Specifically, after the inference network included in the speech generation network outputs a text word, if the time step of that text word is T, the aforementioned common speech generator will continue to generate multiple hidden vectors related to the corresponding speech words at the next time step T+1. In this way, the common speech generator can immediately continue to generate multiple hidden vectors related to the corresponding speech words at the next time step after the inference network outputs a text word, following a streaming processing mode, which helps to reduce the speech generation latency of the system.

[0094] In practical applications, the codebooks maintained by each layer of the quantization network can typically be used as learnable parameters and trained together with the aforementioned speech generation network. Before training begins, a codebook corresponding to different dimensions of speech features can be initialized for each layer of the quantization network. This codebook usually contains several fixed-dimensional vectors as codewords, each codeword being associated with a speech word. The learnable parameters in the vectors within this codebook can be optimized and adjusted along with the learnable parameters in the speech generation network during training. Once the speech generation network is trained, the learnable parameters in the vectors of this codebook are also optimized and adjusted, and these learnable parameters can store the speech features learned from the input speech data samples.

[0095] based on Figure 5 The network architecture shown, when inputting the first hidden vector related to the aforementioned text lexical and the embedding vector corresponding to the aforementioned historical speech lexical sequence into the aforementioned generator network, can specifically input the first hidden vector and the embedding vector corresponding to the aforementioned historical speech lexical sequence into the aforementioned common speech generator. The speech generator can then generate, in an autoregressive manner, a second hidden vector related to multiple speech lexicals corresponding to the text lexical based on the first hidden vector and the embedding vector at the next time step after the time step where the text lexical is located.

[0096] Then, the second hidden vector output by the aforementioned speech generator can be input into the aforementioned multi-layer quantization network. Based on the maintained codebook, the multi-layer quantization network quantizes the second hidden vector into speech units with the speech features stored in the codebook, resulting in multiple speech units that correspond one-to-one with the aforementioned multi-layer quantization network. At this point, the multiple speech units output by the multi-layer quantization network are the multiple speech units corresponding to the aforementioned text units.

[0097] It needs to be emphasized that, Figure 5 The network architecture shown is only the basic network architecture related to the function of the generator network. In practical applications, it can be adjusted based on specific needs. Figure 5 Based on the network architecture of the generated network shown, the network architecture of the generated network is further extended.

[0098] Please see Figure 6 , Figure 6 This is a network architecture diagram of another generated network shown in this specification.

[0099] like Figure 6 As shown, in some embodiments, Figure 5 The generator network shown can also include multi-level prediction networks.

[0100] exist Figure 6 In the illustrated network architecture, the first layer network, consisting of the aforementioned common speech generator and multi-level quantization networks, can serve as the backbone network of the generation network. The multi-level prediction network can then serve as a multi-level extension network corresponding to the backbone network.

[0101] The first-level prediction network in this multi-level prediction network may specifically include a predictor that takes the output of the aforementioned speech generator as input; and a multi-level quantization network that takes the output of the predictor as input.

[0102] The multi-level prediction network, in addition to the first-level prediction network, may specifically include a predictor that takes the output of the predictor in the previous-level prediction network as input; and a multi-layer quantization network that takes the output of the predictor as input.

[0103] Specifically, the predictors in each stage of the prediction network are used to predict the hidden vectors that are generated by the speech generator or the hidden vectors generated by the predictors in the previous stage prediction network, and the hidden vectors that are generated at the next time step. The multi-level quantization networks in each stage of the prediction network retain their function and are used to quantize the input hidden vectors into speech tokens based on the maintained codebook.

[0104] based on Figure 6 The network architecture shown, when inputting the first hidden vector related to the aforementioned text word and the embedding vector corresponding to the aforementioned historical speech word sequence into the aforementioned generator network, can specifically first input the first hidden vector related to the text word and the embedding vector corresponding to the aforementioned historical speech word sequence into the aforementioned common speech generator, so that the speech generator can generate, in an autoregressive manner, a second hidden vector related to multiple speech words corresponding to the text word based on the first hidden vector and the embedding vector at the next time step T+1 after the time step T where the text word is located.

[0105] Then, the second hidden vector output by the speech generator can be input into the aforementioned multi-layered quantization network. The multi-layered quantization network then quantizes the second hidden vector into speech units with the speech features stored in the codebook, based on the maintained codebook, resulting in multiple speech units corresponding to time step T+1. At this point, the multiple speech units corresponding to time step T+1 are the basic speech units corresponding to the text units output by the aforementioned inference network.

[0106] Furthermore, the second hidden vector output by the speech generator can be further input into the first predictor in the first-level prediction network, so that the first predictor can generate a third hidden vector related to time step T+2 based on the second hidden vector at the next time step T+1.

[0107] Then, the third hidden vector can be input into the multi-layer quantization network in the first-level prediction network, so that the multi-layer quantization network can quantize the third hidden vector into speech words with the speech features stored in the codebook based on the maintained codebook, thereby obtaining multiple speech words corresponding to time step T+2; at this time, the multiple speech words corresponding to time step T+2 refer to the next extended speech words in time order of the basic speech words predicted by the first-level prediction network.

[0108] Furthermore, the third hidden vector output by the first predictor can be further input into the second predictor in the second-level prediction network, so that the second predictor can generate a fourth hidden vector related to the higher time step T+3 based on the third hidden vector at the next time step T+3 after time step T+2.

[0109] Then, the fourth hidden vector can be input into the multi-layer quantization network in the second-level prediction network, so that the multi-layer quantization network can quantize the fourth hidden vector into speech words with the speech features stored in the codebook based on the maintained codebook, thus obtaining multiple speech words corresponding to time step T+3. At this time, the multiple speech words corresponding to time step T+3 refer to the next extended speech word predicted in time order by the second-level prediction network based on the extended speech word predicted by the previous-level prediction network.

[0110] Following this pattern, in a streaming processing model, the prediction networks at each level predict subsequent extended speech terms in temporal order, from top to bottom, until the last level's multi-layered quantization network. Each layer quantizes the hidden vector output by the predictor in the last level's network into speech terms with the speech features stored in the codebook, based on a maintained codebook, resulting in multiple speech terms corresponding to time step T+1+N. Here, N represents the number of levels in the multi-level prediction network. In practical applications, the multi-level prediction network can be flexibly configured based on specific circumstances. For example, in practical applications, an empirical value can be determined through experimentation.

[0111] use Figure 6The network architecture shown ultimately outputs speech units corresponding to the inference network, which include one basic speech unit and multiple extended speech units. These basic speech units and multiple extended speech units can be used together as speech units corresponding to the text units output by the inference network.

[0112] In this way, the generator network is required not only to generate the basic speech words corresponding to the text words output by the inference network at time step T+1, but also to predict multiple extended speech words corresponding to the basic speech words at time steps t+2, t+3, etc. This allows multiple speech words to be predicted in a single inference pass. This not only makes more efficient use of limited training data and further reduces the latency during speech generation, but also better captures the local dependencies between speech words, thus improving the speech quality of the final decoded speech data as a speech response.

[0113] For example, in practical applications, the aforementioned extended speech units can often serve as a further supplement to the basic speech units in terms of audio features, enriching the acoustic features of the basic speech units. If we adopt... Figure 5 The network architecture shown, without incorporating the aforementioned multi-level prediction network into the generator network, cannot adequately model the local dependencies between speech units, potentially resulting in the decoded speech data as a speech response lacking some acoustic details. However, by introducing the multi-level prediction network, multiple extended speech units with local dependencies on the basic speech units can be further modeled, ensuring that the final decoded speech data as a speech response possesses rich acoustic details.

[0114] It should be noted that, Figure 5 or Figure 6 The specific type of multilayer quantization network used in this specification is not limited here; it can be flexibly selected in practical applications.

[0115] Please see Figure 7A and 7B , Figure 7A and 7B This is a network architecture diagram of another generated network shown in this specification.

[0116] In one embodiment shown, such as Figure 7A and 7BAs shown, the aforementioned multi-layered quantization network can specifically include multi-layered classification networks; that is, a basic classification network is used to function as the quantization network. In this case, the codebook maintained by the aforementioned multi-layered quantization network can specifically be in the form of a weight matrix maintained by the classification network for classification. In other words, the weight matrix maintained by the classification network for classification is used as the codebook for quantizing the hidden vectors.

[0117] In this context, the classification result of the classification network, which acts as the quantization network, can be used to represent the generated speech lexical units after quantizing the input hidden vector. When the aforementioned multi-layered quantization network quantizes the input hidden vector into speech lexical units with the speech features stored in the maintained codebook, the multi-layered classification network can specifically classify the input hidden vector based on the maintained weight matrix, obtaining multiple speech lexical units that correspond one-to-one with the classification network as classification results.

[0118] In some embodiments, please continue to see Figure 8A and Figure 8B The aforementioned multi-layered quantization network can specifically include multiple layers of classification heads. In other words, the structure of the classification heads is used to act as the classification network. This approach further simplifies the network architecture of the generative network.

[0119] Of course, in practical applications, besides using classification networks as the quantization network mentioned above, other types of quantization networks can also be used. For example, a dedicated quantizer such as VQ-VAE can also be used as the quantization network.

[0120] It should be noted that the network architecture of the aforementioned common speech generator and the predictors in the prediction networks at each level is not specifically limited in this specification. In practical applications, it can be flexibly designed and selected.

[0121] In some embodiments, the aforementioned common speech generator and the predictors in the prediction networks at each level may specifically employ a neural network based on a transformer architecture.

[0122] For example, in some embodiments, since the functions of the aforementioned common speech generator and the predictors in each stage of the prediction network are similar to the decoder part in the transformer architecture, in practical applications, the decoder part of the transformer architecture can be used as the aforementioned common speech generator and the predictors in each stage of the prediction network. In this way, the network architecture of the aforementioned common speech generator and the predictors in each stage of the prediction network can be further simplified.

[0123] In one embodiment shown, the speech generation network may further include a pre-trained decoding network; the decoding network may specifically be a decoding network based on a multi-codebook architecture.

[0124] In this case, multiple speech units output by the generating network can be further input into the decoding network, so that the decoding network can decode the multiple speech units into second speech data, and use the second speech data as the speech response corresponding to the first speech data.

[0125] For example, in practical applications, the above-mentioned decoding network may specifically include an audio decoder and a vocoder. The audio decoder can be responsible for decoding multiple speech tokens output by the generating network into audio spectra, while the vocoder can be responsible for reconstructing the audio spectra into audio data.

[0126] It should be emphasized that the aforementioned multiple speech units possess multiple dimensions of speech features learned from the first input speech data, and thus contain rich speech features; therefore, in practical applications, the aforementioned decoding network can specifically adopt a lightweight decoding network.

[0127] For example, the decoder portion of a general audio codec network that supports multiple codebooks can be used as the aforementioned decoding network.

[0128] In some embodiments, the speech generation network may further include an encoding network; specifically, this encoding network may be a multi-codebook-based encoding network, which can encode the speech data samples used to train the speech generation network into multiple speech word samples based on multiple maintained codebooks. These speech word samples can serve as supervision terms that correspond one-to-one with the multiple speech words generated by the generation network.

[0129] For example, please continue to see 5 or Figure 6 By introducing an encoding network into the aforementioned speech generation network, during the training phase of the speech generation network, see 5 or Figure 6 The output of each path of the quantization network generates a speech lexical sample as a supervision term.

[0130] In some embodiments, the decoding network described above can specifically be an audio codec network supporting multiple codebooks; this audio codec network typically adopts an encoder-decoder architecture. In this case, the encoder portion of the audio codec network can be used as the encoding network, and the decoder portion of the audio codec network can be used as the decoding network. That is, the encoder and decoder portions of the same audio codec network are used as the encoding and decoding networks, respectively. In this way, the network structure of the speech generation network can be effectively simplified.

[0131] The specific types of audio codec networks that support multiple codebooks are not specifically limited in this specification. In practical applications, they can be flexibly selected based on specific needs.

[0132] For example, in one instance, the aforementioned multi-codebook-supporting audio codec network may specifically include the XY-Tokenizer audio codec network.

[0133] After the aforementioned generative network generates multiple speech units corresponding to the aforementioned text units, on the one hand, during the online inference stage based on the trained speech generation network, the second speech data decoded by the aforementioned decoding network can be output as the speech response corresponding to the aforementioned first speech data.

[0134] On the other hand, during the training phase of the speech generation network, the training loss can be further calculated based on the second speech data decoded by the aforementioned decoding network and the speech data samples used as training samples. Based on the calculated training loss, the learnable parameters contained in the speech generation network can be optimized and adjusted.

[0135] The training strategy used in training the speech generation network is not specifically limited in this specification. In practical applications, it can be flexibly selected based on specific training needs.

[0136] For example, in some embodiments, a training strategy that trains the speech generation network in stages can be adopted.

[0137] In this context, the training process for the speech generation network may specifically include: The first stage of training involves removing the aforementioned inference network or freezing the parameters contained in the aforementioned inference network, and then independently pre-training the aforementioned generator network. Specifically, the pre-training task corresponding to this pre-training can include: generating multiple speech lexical units corresponding to the input text lexical units samples.

[0138] The second stage of training involves removing the aforementioned generator network or freezing the parameters contained in the generator network, and independently fine-tuning the aforementioned inference network. Specifically, the fine-tuning training task corresponding to this fine-tuning training may include: performing inference calculations on the input speech data samples to generate text word sequences corresponding to the speech data samples in an autoregressive manner.

[0139] The third stage of training can be fine-tuning the speech generation network as a whole; the fine-tuning training task corresponding to this fine-tuning training can specifically include: an end-to-end generation task of generating speech data as the speech response corresponding to the input speech data sample.

[0140] It should be noted that the order of the first and second phases of training mentioned above can be interchanged.

[0141] By employing this phased training strategy to train the speech generation network, not only can the training effect be improved, but the convergence speed of training the speech generation network can also be significantly reduced.

[0142] This specification does not impose specific restrictions on the training loss used in each of the above training stages. In practical applications, the loss function can be flexibly designed based on the task objective of each training stage. For example, in one scenario, the task objective corresponding to the above three training stages is a generation task. For generation tasks, the cross-entropy loss function can be used to construct the training loss.

[0143] It should be noted that since the above-mentioned generative network adopts a multi-channel network architecture with multiple outputs, in practical applications, a loss function can be constructed for each output. Finally, when calculating the training loss for training the speech generation network, the losses of each output can be summed to obtain the final training loss for training the speech generation network.

[0144] In the above technical solution, since the text words in the text word sequence generated by reasoning and computing on the input first speech data can be directly modeled into multiple speech words with multiple dimensions of speech features learned from the first speech data, a lightweight decoding network can directly decode these multiple speech words containing rich speech features into second speech data as the speech response corresponding to the input first speech data, without the need to reconstruct speech data based on a single speech word. This not only significantly reduces the latency in generating a speech response for the input first speech data and improves the fluency and response efficiency of voice interaction, but also enhances the realism of the generated speech data.

[0145] For example, speech data reconstructed from a single speech unit often fails to accurately reflect the audio characteristics of the audio data because the reconstruction process lacks the authentic speech features from the input speech data. However, if multiple speech units with multi-dimensional speech features learned from the initial input speech data are directly decoded into speech data, the final generated speech data will contain rich and authentic multi-dimensional audio features, thus significantly improving the realism of the generated speech.

[0146] Figure 9 This is a schematic structural diagram of an electronic device provided in an exemplary embodiment. For example... Figure 9 As shown, device 900 mainly consists of a communication interface 902, a user interface 904, a processor 906, and a data storage 908. These components are interconnected and communicate with each other via a system bus, network, or other connection mechanism 910. The communication interface 902 enables device 900 to communicate with other devices, access networks, and transmission networks via analog or digital modulation. For example, the communication interface 902 may include a chipset and antenna for wireless communication with a radio access network or access point. Furthermore, the communication interface 902 can be a wired interface such as Ethernet, Token Ring, or a USB port, or a wireless interface such as Wi-Fi, Bluetooth, Global Positioning System (GPS), or a wide-area wireless interface (e.g., WiMAX or LTE). Of course, the communication interface 902 can also support other forms of physical layer interfaces and standard or proprietary communication protocols. The communication interface 902 may also include multiple physical communication interfaces, such as Wi-Fi, Bluetooth, and wide-area wireless interfaces.

[0147] User interface 904 includes receiving user input and providing output to the user. Therefore, user interface 904 may include input components such as a keypad, keyboard, touch-sensitive or presence-sensitive panel, computer mouse, trackball, joystick, microphone, still camera, and video camera, and output components such as a display screen (which may be combined with a touch-sensitive panel), CRT, LCD, LED, display using DLP technology, printer, and other similar devices known or developed in the future. User interface 904 may also generate auditory output via speakers, speaker jacks, audio output ports, audio output devices, headphones, and other similar devices known or developed in the future. In some embodiments, user interface 904 may include software, circuitry, or other forms of logic capable of transmitting and receiving data from external user input / output devices. Additionally or alternatively, device 900 may support remote access from other devices via communication interface 902 or another physical interface (not shown). User interface 904 may be configured to receive user input, the position and movement of which may be indicated by indicators or cursors described herein. User interface 904 may also be configured as a display device for rendering or displaying text fragments.

[0148] Processor 906 may contain one or more general-purpose processors and / or special-purpose processors.

[0149] Data storage 908 may include one or more volatile and / or non-volatile storage components and may be integrated wholly or partially with processor 906. Data storage 908 may include removable and non-removable components.

[0150] Processor 906 is capable of executing program instructions 918 (e.g., compiled or uncompiled program logic and / or machine code) stored in data storage 908 to perform the various functions described herein. Data storage 908 may contain a non-transitory computer-readable medium on which program instructions are stored, which, when executed by device 900, enable device 900 to perform any methods, processes, or functions disclosed in this specification and / or the accompanying drawings. Processor 906 executing program instructions 918 may result in processor 906 using data 912.

[0151] For example, program instructions 918 may include an operating system 922 (e.g., an operating system kernel, device drivers, and / or other modules) installed on device 900 and one or more application programs 920 (e.g., a browser, social media application, or game application). Similarly, data 912 may include operating system data 916 and application data 914. Operating system data 916 is primarily accessible to the operating system 922, while application data 914 is primarily accessible to one or more application programs 920. Application data 914 may reside in a file system visible or hidden from the user of device 900.

[0152] Application 920 can communicate with operating system 922 through one or more application programming interfaces (APIs). These APIs help application 920 read and / or write application data 914, transmit or receive information via communication interface 902, receive or display information on user interface 904, etc.

[0153] In some terminology, application 920 may be simply referred to as "app". Furthermore, application 920 can be downloaded to device 900 through one or more online app stores or app markets. However, applications can also be installed on device 900 in other ways, such as through a web browser or a physical interface on device 900 (e.g., a USB port).

[0154] Please refer to Figure 10 This specification also proposes an LLM-based speech generation device, which can be applied to applications such as... Figure 9 The device shown implements the technical solution of this specification. The speech generation network based on LLM includes an inference network based on LLM; and a generation network employing a multi-channel network architecture. The inference network performs inference calculations on the input speech data to generate a text word sequence corresponding to the speech data in an autoregressive manner. The generation network generates multiple speech words corresponding to the text words in the text word sequence. The multiple speech words have multiple dimensions of speech features learned from the speech data. This device may include: Module 1001 acquires the first input voice data; The inference module 1002 inputs the first speech data into the inference network, and the inference network performs inference calculations on the first speech data to generate a text word sequence corresponding to the first speech data in an autoregressive manner, and outputs the text words in the generated text word sequence. The generation module 1003 further inputs the text lexical units into the generation network so that the generation network generates a plurality of speech lexical units corresponding to the text lexical units; wherein, the plurality of speech lexical units are used to decode and generate second speech data as a speech response corresponding to the first speech data.

[0155] For ease of description, the above devices are described by dividing them into various modules or units based on their functions. Of course, when implementing one or more of these specifications, the functions of each module or unit can be implemented in the same or different software and / or hardware, or a module that performs the same function can be implemented by a combination of multiple sub-modules or sub-units, etc. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0156] Based on the same concept as the methods described above, this specification also provides an electronic device, including: a processor; a memory for storing processor-executable instructions; wherein the processor performs the steps of the method as described in any of the above embodiments by executing the executable instructions.

[0157] Based on the same concept as the methods described above, this specification also provides a computer-readable storage medium having computer instructions stored thereon that, when executed by a processor, implement the steps of the methods as described in any of the above embodiments.

[0158] Based on the same concept as the methods described above, this specification also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the methods as described in any of the above embodiments.

[0159] What those skilled in the art will understand is: In this specification, the terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, product, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, product, or apparatus. Without further limitation, the presence of additional identical or equivalent elements in a process, method, product, or apparatus that includes said elements is not excluded.

[0160] In this specification, “a,” “an,” and “the” do not specifically refer to the singular, but may also include the plural.

[0161] In this specification, ordinal numbers such as "first," "second," etc., do not necessarily indicate order; they are often used to distinguish between objects. For example, "first server" and "second server" usually refer to two servers. To differentiate between these two servers, they are described as "first server" and "second server." Of course, sometimes these two servers may be the same server.

[0162] In this specification, unless explicitly stated otherwise, "receiving and sending data" does not necessarily mean direct receiving and sending; it can also mean indirect receiving and sending. For example, A receiving data sent by B can be understood as A directly receiving the data sent by B, or it can be understood as A indirectly receiving the data sent by B through other entities such as C. Similarly, B sending data to A can be understood as B sending the data directly to A, or it can be understood as B indirectly sending the data to A through other entities such as C. Here, C can be one entity, or it can be two or more entities.

[0163] In this specification, unless explicitly stated otherwise, the relationships between structures can be direct or indirect. For example, when describing "A is connected to B," unless it is explicitly stated that A and B are directly connected, it should be understood that A can be directly connected to B or indirectly connected to B. Similarly, when describing "A is on top of B," unless it is explicitly stated that A is directly above B (AB is adjacent and A is above B), it should be understood that A can be directly above B or indirectly above B (AB is separated by other elements, and A is above B). And so on.

[0164] This specification uses specific terms to describe embodiments thereof. Terms such as "an embodiment," "one embodiment," and / or "some embodiments" refer to a particular feature, structure, or characteristic associated with at least one embodiment of this specification. Therefore, it should be emphasized and noted that references to "an embodiment," "one embodiment," or "an alternative embodiment" in different locations throughout this specification do not necessarily refer to the same embodiment. Furthermore, those skilled in the art can combine and integrate the different embodiments or examples described herein, as well as the features of those different embodiments or examples, without contradiction.

[0165] Although one or more embodiments of this specification provide method steps as described in the embodiments or flowcharts, it is understood that the order of steps listed in the embodiments or flowcharts is only one of many possible execution orders and does not represent the only execution order. Therefore, when the claims involve method steps, any changes or adjustments to the order of such steps, or the parallelism between steps, are also within the scope of protection of the claims.

Claims

1. An LLM-based speech generation method, applied to a speech generation network constructed based on LLM; the speech generation network includes an inference network constructed based on LLM; and a generation network employing a multi-channel network architecture; wherein, The inference network is used to perform inference calculations on the input speech data to generate a text word sequence corresponding to the speech data in an autoregressive manner; the generation network is used to generate multiple speech words corresponding to the text words in the text word sequence. The plurality of speech lexical units possess multiple dimensions of speech features learned from the speech data; the method includes: Acquire the first input speech data; The first speech data is input into the inference network, which performs inference calculations on the first speech data to generate a text word sequence corresponding to the first speech data in an autoregressive manner, and outputs the text words in the generated text word sequence. The text lexical units are further input into the generation network to generate multiple speech lexical units corresponding to the text lexical units; wherein, the multiple speech lexical units are used to decode and generate second speech data as a speech response corresponding to the first speech data.

2. The method of claim 1, wherein the text lexical units are further input into the generation network to generate a plurality of speech lexical units corresponding to the text lexical units, comprising: Obtain the first hidden vector related to the text lexical generated by the inference network, and obtain the historical speech lexical sequence already generated by the generation network; The first hidden vector associated with the text word and the embedding vector corresponding to the historical speech word sequence are further input into the generation network so that the generation network generates multiple speech words corresponding to the text word in an autoregressive manner. The embedding vector includes a vector obtained by vector fusion of the embedding vectors corresponding to each historical speech word in the historical speech word sequence.

3. The method of claim 2, further comprising, before inputting the first hidden vector associated with the text lexical and the embedding vector corresponding to the historical speech lexical sequence into the generation network: The embedding vector corresponding to the text word and the first hidden vector related to the text word are fused together.

4. The method of claim 2, wherein the generating network comprises a speech generator that takes the output of the inference network as input; and a multi-layer quantization network that takes the output of the speech generator as input; wherein, The speech generator is used to generate hidden vectors related to multiple speech words corresponding to the text words at the next time step after the time step where the text words are located; the multi-layer quantization network maintains codebooks for storing speech features learned from the input speech data; the codebooks maintained by each layer of the quantization network store speech features of different dimensions. The first hidden vector associated with the text lexical and the embedding vector corresponding to the historical speech lexical sequence are further input into the generation network, so that the generation network generates multiple speech lexicals corresponding to the text lexical in an autoregressive manner, including: The first hidden vector associated with the text lexical and the embedding vector corresponding to the historical speech lexical sequence are further input into the speech generator. The speech generator, in an autoregressive manner, generates a second hidden vector associated with multiple speech lexicals corresponding to the text lexical in the next time step after the time step where the text lexical is located, based on the first hidden vector and the embedding vector. The second hidden vectors output by the speech generator are then input into the multi-layer quantization network. The multi-layer quantization network quantizes the second hidden vectors into speech lexicals with the speech features stored in the codebook, based on the maintained codebook, thus obtaining multiple speech lexicals that correspond one-to-one with the multi-layer quantization network.

5. The method of claim 4, wherein the generating network further comprises a multi-level prediction network; wherein, The first-level prediction network in the multi-level prediction network includes a predictor that takes the output of the speech generator as input; and a multi-layer quantization network that takes the output of the predictor as input; the other levels of the prediction network besides the first-level prediction network include predictors that take the output of the predictor in the previous level prediction network as input; and a multi-layer quantization network that takes the output of the predictor as input. The first hidden vector associated with the text lexical and the embedding vector corresponding to the historical speech lexical sequence are further input into the speech generator. The speech generator, using an autoregressive approach, generates second hidden vectors associated with multiple speech lexicals corresponding to the text lexical in the next time step following the time step where the text lexical is located. The second hidden vectors output by the speech generator are then input into the multi-layer quantization network. The multi-layer quantization network quantizes the second hidden vectors into speech lexicals with the speech features stored in the codebook, based on a maintained codebook, resulting in multiple speech lexicals corresponding one-to-one with the multi-layer quantization network, including: The first hidden vector associated with the text word and the embedding vector corresponding to the historical speech word sequence are further input into the speech generator. The speech generator generates a second hidden vector associated with multiple speech words corresponding to the text word in an autoregressive manner at the next time step T+1 after the time step T where the text word is located, based on the first hidden vector and the embedding vector. The second hidden vector output by the speech generator is then input into the multi-layer quantization network. The multi-layer quantization network quantizes the second hidden vector into speech words with the speech features stored in the codebook based on the maintained codebook, thereby obtaining multiple speech words corresponding to the time step T+1. The second hidden vector output by the speech generator is further input into the first predictor in the first-level prediction network. The first predictor generates a third hidden vector related to the time step T+2 based on the second hidden vector at the next time step T+1. The third hidden vector is then input into the multi-layer quantization network in the first-level prediction network. The multi-layer quantization network quantizes the third hidden vector into speech words with the speech features stored in the codebook based on the maintained codebook, thereby obtaining multiple speech words corresponding to the time step T+2. The third hidden vector output by the first predictor is further input into the second predictor in the second-level prediction network. The second predictor generates a fourth hidden vector related to the time step T+3 based on the third hidden vector at the next time step T+2. The fourth hidden vector is then input into the multi-layer quantization network in the second-level prediction network. The multi-layer quantization network quantizes the fourth hidden vector into speech words with the speech features stored in the codebook based on the maintained codebook, thereby obtaining multiple speech words corresponding to the time step T+3. This process continues until the last level of the prediction network, where multiple quantization networks quantize the hidden vectors output by the predictor in the last level of the prediction network into speech words with the speech features stored in the codebook, based on the maintained codebook, thus obtaining multiple speech words corresponding to the time step T+1+N; where N represents the level of the multi-level prediction network.

6. The method of claim 5, wherein the multi-layer quantization network comprises a multi-layer classification network; the codebook maintained by the quantization network comprises a weight matrix maintained by the classification network for classification. The multi-layered quantization network quantizes the input hidden vector into speech lexical units with the speech features stored in the codebook based on the maintained codebook, thereby obtaining multiple speech lexical units that correspond one-to-one with the multi-layered quantization network, including: The multi-layer classification network classifies the input hidden vector based on the maintained weight matrix, resulting in multiple speech lexical units that correspond one-to-one with the multi-layer classification network as classification results.

7. The method of claim 6, wherein the speech generator and the predictor comprise a neural network based on a transformer architecture; and the multi-layer classification network comprises multi-layer classification heads.

8. The method of claim 7, wherein the speech generator and the predictor include a decoder portion of a transformer architecture.

9. The method of claim 5, wherein the speech generation network further comprises a pre-trained decoding network; the decoding network comprises a decoding network based on a multi-codebook architecture; The method further includes: The plurality of speech words are further input into the decoding network, so that the decoding network decodes the plurality of speech words into second speech data, which serves as the speech response corresponding to the first speech data.

10. The method of claim 9, wherein the speech generation network further comprises an encoding network; the encoding network is used to encode speech data samples used to train the speech generation network into multiple speech word samples based on multiple maintained codebooks, as supervision terms corresponding one-to-one with the multiple speech words generated by the generation network.

11. The method of claim 10, wherein the decoding network includes an audio codec network supporting multiple codebooks; the encoding network includes an encoder portion of the audio codec network; and the decoding network includes a decoder portion of the audio codec network.

12. The method of claim 11, wherein the multi-codebook-supporting audio codec network includes the XY-Tokenizer audio codec network.

13. The method of claim 9, wherein the training process for the speech generation network includes: The first stage of training involves removing the inference network or freezing the parameters contained in the inference network, and independently pre-training the generator network; wherein, the pre-training task corresponding to the pre-training includes the task of generating multiple speech words corresponding to the input text word samples. The second stage of training involves removing the generator network or freezing the parameters contained in the generator network, and independently fine-tuning the inference network. The fine-tuning training task corresponding to the fine-tuning training includes performing inference calculations on the input speech data samples to generate text word sequences corresponding to the speech data samples in an autoregressive manner. The third stage of training: fine-tuning training of the speech generation network as a whole; wherein, the fine-tuning training task corresponding to the fine-tuning training includes an end-to-end generation task of generating speech data as speech responses corresponding to the input speech data samples based on the input speech data samples.

14. A speech generation network based on LLM, comprising: An inference network based on LLM is used to perform inference calculations on the input first speech data to generate a text word sequence corresponding to the first speech data in an autoregressive manner. A generative network employing a multi-channel network architecture is used to generate multiple speech words corresponding to text words in the text word sequence; the multiple speech words have multiple dimensions of speech features learned from the first speech data; The plurality of speech lexical units are used to decode and generate second speech data as a speech response corresponding to the first speech data.

15. An electronic device, characterized in that, include: processor; A memory for storing processor-executable instructions; wherein the processor implements the steps of the method as described in any one of claims 1-13 by executing the executable instructions.

16. A computer-readable storage medium, characterized in that, It stores computer instructions that, when executed by a processor, implement the steps of the method as described in any one of claims 1-13.

17. A computer program product, characterized in that, Includes a computer program / instructions that, when executed by a processor, implement the steps of the method as described in any one of claims 1-13.

Images