Speech synthesis streaming reasoning method and system based on large model
By employing a large-model streaming inference method for speech synthesis on resource-constrained devices, asynchronous text encoding, and using a lightweight decoder, the challenge of high-quality, low-latency speech synthesis on resource-constrained devices is solved, achieving low-power, high-efficiency real-time speech synthesis.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- BEIJING NANSHE TECH CO LTD
- Filing Date
- 2026-01-29
- Publication Date
- 2026-04-28
AI Technical Summary
On resource-constrained devices, existing technologies struggle to achieve high-quality, low-latency speech synthesis, especially in scenarios such as the Internet of Things, embedded systems, mobile devices, and wearable devices, where traditional solutions suffer from limitations in computing power, memory capacity, and battery life.
We employ a large-model-based streaming inference method for speech synthesis. By using asynchronous preprocessing, we distribute the computationally intensive text encoding task to offline or coprocessors and use a lightweight adversarial decoder for real-time audio generation. We adopt an encoding-buffering-streaming decoding architecture to achieve frame-by-frame output.
High-quality speech synthesis, millisecond-level first-frame latency, and real-time interactive experience were achieved on low-power devices, reducing the instantaneous computing power and memory requirements of the device and meeting the needs of real-time interaction.
Smart Images
Figure CN121938346A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of speech synthesis, and in particular to a streaming inference method and system for speech synthesis based on a large model. Background Technology
[0002] In recent years, with the rapid development of deep learning and artificial intelligence technologies, neural network-based speech synthesis technology has made significant progress, especially end-to-end neural network speech synthesis models, which can generate high-quality speech that closely resembles natural human voices. These models are typically trained on large-scale, high-quality speech datasets, directly mapping high-fidelity speech waveforms or acoustic features through a deep understanding of the input text. However, these advanced models, especially large models that can generate highly expressive and natural speech, often have a huge number of parameters and high computational complexity, placing high demands on computing hardware (such as GPUs / TPUs) in terms of computing power, memory, and power consumption.
[0003] While traditional cloud-based speech synthesis solutions can provide high-quality speech, their reliance on stable network connections introduces unavoidable network latency, making it difficult to meet the needs of scenarios requiring real-time interaction, high privacy, and sensitivity to network conditions (such as weak network environments). This is especially true in resource-constrained edge computing environments such as IoT, embedded systems, mobile devices, and wearable devices, where computing power, memory capacity, and battery life are all very limited. Directly deploying large speech synthesis models onto such devices often faces challenges such as slow inference speed, high power consumption, excessive memory usage, and even failure to run.
[0004] To achieve high-quality, low-latency speech synthesis on resource-constrained devices, the industry has explored various technical approaches. Model lightweighting techniques, such as knowledge distillation, pruning, and quantization, can compress model size and reduce computational load to some extent, but often come at the cost of some speech quality loss. Furthermore, for ultra-large-scale models, the lightweighted size and computational requirements may still exceed the capabilities of many low-power devices. Another approach is to adopt a "cloud-based collaborative" or "edge computing" architecture to distribute some computational tasks, but this still cannot completely eliminate network latency and dependencies. Summary of the Invention
[0005] Purpose of the invention: The purpose of this invention is to provide a streaming inference method and system for speech synthesis based on a large model, which enables high-quality, low-latency speech synthesis on low-power, resource-constrained devices.
[0006] Technical solution: The present invention provides a streaming inference method for speech synthesis based on a large model, comprising the following steps:
[0007] Step 1: Obtain the raw text data input by the user, use the trained large language model as a text encoder to generate a vector sequence and store it in the vector storage area;
[0008] Step 2: When the vector sequence in the vector storage area reaches the preset length, read this preset length vector sequence from the vector storage area and send it to the audio decoder to generate a frame of audio data.
[0009] Step 3: Repeat step 2 to generate audio data frame by frame until all the original text data is converted into an audio data stream.
[0010] Furthermore, the vector storage area is a first-in-first-out circular buffer containing several consecutive slots for sequentially storing the vector sequence.
[0011] Furthermore, in step 1, the vector sequence is stored in the vector storage area by writing the pointer, and in step 2, the vector sequence is read out from the vector storage area by reading the pointer.
[0012] Furthermore, the text encoder is a large language model based on the Transformer architecture.
[0013] Furthermore, the audio decoder is an audio decoding model based on generative adversarial networks or conditional adversarial generative networks.
[0014] The present invention discloses a large-model-based speech synthesis streaming inference system, comprising:
[0015] A text encoder is used to generate a vector sequence based on the user's input raw text data, using a trained large language model as the text encoder.
[0016] A vector storage area is used to store the vector sequence;
[0017] An audio decoder is used to read a vector sequence of a preset length from a vector storage area in a loop, generate audio data frame by frame, until all the original text data is converted into an audio data stream.
[0018] Furthermore, the vector storage area is a first-in-first-out circular buffer containing several consecutive slots for sequentially storing the vector sequence;
[0019] The text encoder stores the vector sequence into the vector storage area by writing the pointer, and the audio decoder reads the vector sequence from the vector storage area by reading the pointer.
[0020] Furthermore, it also includes a streaming control logic unit for managing the read and write status of the vector storage area and controlling the start and stop of the audio decoder;
[0021] When the streaming control logic unit detects that a vector sequence has been stored in the vector storage area and the length has reached the preset length, it starts the audio decoder and instructs the audio decoder to retrieve the vector sequence from the current read pointer position. After retrieval, the read pointer moves forward one position.
[0022] Furthermore, after the text encoder generates a vector sequence of all the original text data and stores it in the vector storage area, the text encoder sends an end bit signal to the streaming control logic unit.
[0023] When the audio decoder has consumed all vector sequences in the vector storage area and the streaming control logic unit receives the end bit signal, it determines that the speech synthesis task is complete and controls the audio decoder to stop.
[0024] Furthermore, the text encoder is a large language model based on the Transformer architecture; the audio decoder is an audio decoding model based on generative adversarial networks or conditional adversarial generative networks.
[0025] Beneficial effects: Compared with the prior art, the advantages of the present invention are: (1) By asynchronously preprocessing the computationally intensive large model encoding task, the device only needs to run a lightweight adversarial decoder for real-time generation, which greatly reduces the demand for instantaneous computing power, memory and power consumption of the device, and enables high-quality speech synthesis function to be deployed locally on resource-constrained devices such as embedded and wearable devices. (2) The present invention adopts a pipeline architecture of "encoding-buffering-streaming decoding". Audio generation does not need to wait for the full text encoding to be completed. As long as there is data in the buffer, the first frame of speech can be output immediately, realizing millisecond-level first frame delay and real-time interactive experience of "speaking and playing at the same time", completely eliminating the waiting delay in the traditional solution. Attached Figure Description
[0026] Figure 1 This is a flowchart of the large-model-based speech synthesis streaming inference method of the present invention.
[0027] Figure 2 This is a block diagram of the speech synthesis streaming inference system based on a large model according to the present invention. Detailed Implementation
[0028] This invention utilizes a pre-trained large-scale text encoding model (such as a large language model) to efficiently compress long text input by the user into a compact yet information-rich encoder vector sequence, which is then stored in a vector storage area for subsequent streaming access. In the speech generation stage, a lightweight adversarial generation mode is employed, reading vectors frame by frame from the vector storage area and decoding them in real time to generate corresponding audio data frames. This "pre-encoding + streaming decoding" architecture allows the text understanding part, which demands the most computational resources, to be completed offline or through preprocessing, or the computation of the encoding model to be distributed to a more powerful coprocessor. The real-time audio generation part is implemented through an efficient and lightweight adversarial decoder, greatly reducing the demand on the device's instantaneous computing power and memory. This enables frame-by-frame speech synthesis with extremely low latency on low-power devices, meeting the needs of real-time interactive applications.
[0029] The technical solution of the present invention will be further described below with reference to the accompanying drawings.
[0030] like Figure 1 As shown, the speech synthesis streaming inference method includes the following steps.
[0031] Step 1: Obtain the raw text data input by the user, use the trained large language model as a text encoder to generate a vector sequence and store it in the vector storage area;
[0032] Step 2: When the vector sequence in the vector storage area reaches the preset length, read this preset length vector sequence from the vector storage area and send it to the audio decoder to generate a frame of audio data.
[0033] Step 3: Repeat step 2 to generate audio data frame by frame until all the original text data is converted into an audio data stream.
[0034] Specifically, the vector storage area is a first-in-first-out circular buffer containing several consecutive slots for sequentially storing the vector sequence. In step 1, the vector sequence is stored in the vector storage area using a write pointer, and in step 2, the vector sequence is read out of the vector storage area using a read pointer.
[0035] Specifically, the text encoder is a large language model based on the Transformer architecture. It utilizes the deep semantic encoding capabilities of the large model to generate vectors rich in prosodic information, which serve as strong conditional inputs to the lightweight decoder. This allows the final synthesized speech to approach the level of large cloud-based models in terms of naturalness, clarity, and expressiveness, overcoming the quality degradation inherent in traditional lightweight solutions. The audio decoder is an audio decoding model based on generative adversarial networks or conditional generative adversarial networks.
[0036] like Figure 2As shown, the speech synthesis streaming inference system includes a text encoder, a vector storage area (decoder buffer), an audio decoder based on an adversarial generative network, and a streaming control logic unit. The encoding and decoding modules are separated through a standardized vector interface, allowing for flexible configuration of the encoder (local / cloud / collaborative) and decoder (timbre / language) according to device capabilities, facilitating system adaptation to different scenarios and subsequent upgrades. The functions and operating methods of each module are described below.
[0037] (1) Text encoder based on large model
[0038] The text encoder is responsible for receiving raw text data input by the user. This text has no length limit and can be a short instruction or a long paragraph.
[0039] The text encoder uses a pre-trained large language model as its core encoder, such as a Transformer-based model. This model possesses powerful contextual understanding and semantic representation capabilities. Its operation is as follows:
[0040] First, the input text undergoes standardized preprocessing (such as word segmentation and adding prosodic markers).
[0041] The processed text sequence is then input into a large model. Instead of using the model's final output, the hidden states of intermediate layers or feature sequences processed by specific adapter / projection layers are extracted. These feature vectors contain rich linguistic information (such as phonemes, prosody, stress, intonation, etc.).
[0042] The text encoder ultimately outputs an encoder vector sequence E = [e1,e2, ..., eT] corresponding to the temporal sequence of the input text, where T is the sequence length, and each vector et is a high-dimensional dense vector that is a strong conditional signal for subsequent audio generation.
[0043] Specifically, the text encoder leverages prior knowledge gained from training large models on massive corpora to map text of arbitrary length to a highly compressed and semantically clear continuous vector space. This step is computationally intensive but can be performed asynchronously. For example, it can be processed when the device is idle, by a dedicated coprocessor on the device (such as an NPU), or even by a cloud service, with the results stored in a local cache.
[0044] (2) Vector storage area (decoder cache)
[0045] The vector storage area, as a first-in-first-out (FIFO) circular buffer, is used to temporarily store the encoder vector sequence E generated by the text encoder.
[0046] Specifically, the vector storage area is a fixed-size buffer region allocated in memory, organized into multiple contiguous "slots". The text encoder writes the generated vector sequence sequentially and cyclically into these slots. The audio decoder module reads vectors from the buffer's read pointer as needed.
[0047] Specifically, the read and write pointers are managed by a streaming control logic unit to ensure that data overwriting or reading empty data does not occur when there are differences in read and write speeds. This design implements a producer-consumer model. The encoder, acting as a "producer," may "produce" the entire vector sequence at once or in batches and store it in the buffer. The decoder, acting as a "consumer," "consumes" these vectors from the buffer at a constant or variable audio frame rate. The buffer smooths the data flow between two asynchronous, different-rate processing modules.
[0048] Specifically, the vector storage buffer mechanism decouples the encoding and decoding rates, allowing them to be executed asynchronously and improving resource utilization; streaming generation ensures a stable audio output frame rate and predictable system response, avoiding stuttering or interruptions.
[0049] (3) Audio decoder based on adversarial generative network
[0050] The audio decoder is a lightweight generative model that runs locally on the device. Each time, it retrieves one or a small segment of encoder vectors from the current read position in the vector storage as conditional input, ultimately generating a high-quality audio data stream frame by frame.
[0051] The audio decoder employs a carefully designed and compressed generative adversarial network (GAN) or conditional generative adversarial network (CGA) as its core. The generator G is typically a lightweight convolutional neural network or recurrent neural network.
[0052] The input to generator G is the encoder vector et corresponding to the current time t, and optional hidden states from the generation results of the previous time step (to maintain temporal coherence). The output of generator G is a sample of the original speech waveform directly corresponding to the current frame or a low-dimensional acoustic feature (such as a frame of Mel spectrogram). If the output is an acoustic feature, it can be connected to an efficient speech synthesizer (such as a lightweight vocoder) to convert it into a waveform.
[0053] The audio decoder includes a discriminator D, which distinguishes between real speech frames and generator-synthesized speech frames during model training, thereby driving the generator to improve generation quality. During the inference (deployment) phase, only the generator G needs to be used, further reducing runtime computational overhead.
[0054] Specifically, the audio decoder employs adversarial training, enabling the generator to learn the data distribution for generating realistic audio frames given text semantic vectors. Because it processes only the conditional vectors corresponding to one (or a few) frames at a time, rather than the entire long sequence, the model can be made very lightweight, with extremely fast inference speed and small memory footprint, making it ideal for real-time operation on low-power devices.
[0055] (4) Stream control logic unit
[0056] The streaming control logic unit coordinates the entire streaming generation process, manages the read / write status of the vector storage area, and controls the start and stop of the audio decoder. Its workflow is as follows:
[0057] 1) Initialization: When the system starts, the vector storage area is cleared and the read / write pointers are reset.
[0058] 2) Text Reception and Encoding Startup: Upon receiving user text, the text encoder is immediately triggered (or scheduled) to begin operation. The vector sequence generated by the text encoder is then written to the vector storage area.
[0059] 3) Streaming generation startup: Once there is available data in the vector storage area (e.g., the vectors of the first few frames have been written), the streaming control logic immediately starts the audio decoder.
[0060] 4) Loop Reading and Generation: The control logic instructs the audio decoder to retrieve an encoder vector from the current read pointer position in the vector storage area; the audio decoder uses this vector to generate a frame of audio data and outputs it, and finally the read pointer moves forward one position.
[0061] 5) Continuity judgment and loop: The control logic determines whether it has received the "end bit" signal. This "end bit" signal comes from the text encoder. After the entire vector sequence of the text is written into the buffer, the encoder will send a special "sequence end" flag to the control logic.
[0062] If the end bit is not received, it indicates that the text encoding may not be complete, or there are still subsequent vectors to be generated in the buffer. The control logic returns to step 4) to continue fetching data from the buffer and decoding to generate the next frame of audio. This process is continuous and real-time, achieving a pipeline effect of "encoding and generating simultaneously," which greatly reduces the latency of the first frame.
[0063] 6) Completion of Generation: When the audio decoder has consumed all vectors in the buffer (the read pointer catches up with the write pointer), and the control logic confirms that it has received the "end bit" signal from the text encoder, the speech synthesis task is considered complete. The control logic stops the audio decoder and resets the system state to prepare for the next task.
Claims
1. A streaming inference method for speech synthesis based on a large model, characterized in that, Includes the following steps: Step 1: Obtain the raw text data input by the user, use the trained large language model as a text encoder to generate a vector sequence and store it in the vector storage area; Step 2: When the vector sequence in the vector storage area reaches the preset length, read this preset length vector sequence from the vector storage area and send it to the audio decoder to generate a frame of audio data. Step 3: Repeat step 2 to generate audio data frame by frame until all the original text data is converted into an audio data stream.
2. The speech synthesis streaming inference method based on a large model according to claim 1, characterized in that, The vector storage area is a first-in-first-out circular buffer containing several consecutive slots for sequentially storing the vector sequence.
3. The speech synthesis streaming inference method based on a large model according to claim 2, characterized in that, In step 1, the vector sequence is stored in the vector storage area by writing the pointer, and in step 2, the vector sequence is read out from the vector storage area by reading the pointer.
4. The speech synthesis streaming inference method based on a large model according to claim 1, characterized in that, The text encoder is a large language model based on the Transformer architecture.
5. The speech synthesis streaming inference method based on a large model according to claim 1, characterized in that, The audio decoder is an audio decoding model based on generative adversarial networks or conditional adversarial generative networks.
6. A speech synthesis streaming inference system based on a large model, characterized in that, include: A text encoder is used to generate a vector sequence based on the user's input raw text data, using a trained large language model as the text encoder. A vector storage area is used to store the vector sequence; An audio decoder is used to read a vector sequence of a preset length from a vector storage area in a loop, generate audio data frame by frame, until all the original text data is converted into an audio data stream.
7. The speech synthesis streaming inference system based on a large model according to claim 6, characterized in that, The vector storage area is a first-in-first-out circular buffer containing several consecutive slots for sequentially storing the vector sequence; The text encoder stores the vector sequence into the vector storage area by writing the pointer, and the audio decoder reads the vector sequence from the vector storage area by reading the pointer.
8. The speech synthesis streaming inference system based on a large model according to claim 7, characterized in that, It also includes a streaming control logic unit for managing the read and write status of the vector storage area and controlling the start and stop of the audio decoder; When the streaming control logic unit detects that a vector sequence has been stored in the vector storage area and the length has reached the preset length, it starts the audio decoder and instructs the audio decoder to retrieve the vector sequence from the current read pointer position. After retrieval, the read pointer moves forward one position.
9. The speech synthesis streaming inference system based on a large model according to claim 6, characterized in that, After the text encoder generates a vector sequence of all the original text data and stores it in the vector storage area, the text encoder sends an end bit signal to the streaming control logic unit. When the audio decoder has consumed all vector sequences in the vector storage area and the streaming control logic unit receives the end bit signal, it determines that the speech synthesis task is complete and controls the audio decoder to stop.
10. The large-model-based speech synthesis streaming inference system according to claim 6, characterized in that, The text encoder is a large language model based on the Transformer architecture; the audio decoder is an audio decoding model based on generative adversarial networks or conditional adversarial generative networks.