A big model-based bidirectional streaming speech synthesis system and method
By using a bidirectional streaming speech synthesis system based on a large model, low-latency, highly flexible, and low-cost speech synthesis is achieved, solving the problems of dual latency and high coupling in traditional systems. It provides an efficient and natural voice interaction experience and is suitable for scenarios such as intelligent customer service, smart speakers, and in-vehicle voice systems.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-05-08
- Publication Date
- 2026-07-03
AI Technical Summary
Existing technologies cannot simultaneously meet the commercial demands for low latency, high flexibility, low cost, high synthesis quality, and strong contextual coherence. Traditional speech synthesis systems suffer from problems such as dual latency, low resource utilization, high training and deployment costs, excessive coupling, and difficulty in ensuring contextual consistency.
A bidirectional streaming speech synthesis system based on a large model is adopted. Through module decoupling and full-process bidirectional streaming processing, it can generate text and synthesize speech at the same time. It adopts a streaming text semantic extraction module, an audio token generation module, a streaming Code2Wav module and a context management unit to ensure the naturalness of speech and the consistency of context, reduce interaction latency and support easy replacement of timbre and lightweight deployment.
It achieves millisecond-level interaction latency, decoupled architecture, full-process streaming processing, context traceability, and easy voice replacement, reducing training and deployment costs, improving system customization flexibility, ensuring semantic, tone, and voice consistency in multi-turn dialogues and long text synthesis, and has high resource utilization. It is suitable for scenarios such as intelligent customer service, smart speakers, and in-vehicle voice systems.
Smart Images
Figure CN122337178A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of artificial intelligence technology, and in particular, to the field of speech synthesis technology; specifically, it relates to a bidirectional streaming speech synthesis system and method based on a large model. Background Technology
[0002] Voice interaction technology is one of the most user-friendly applications of artificial intelligence, and it has been widely used in scenarios such as intelligent customer service, smart speakers, in-vehicle voice systems, human-computer dialogue robots, and smart home control. Traditional voice interaction systems generally adopt a three-stage serial architecture of Automatic Speech Recognition (ASR), Natural Language Understanding (NLP), and Text-to-Speech (TTS: Text-to-Speech, a technology that converts text information into audible speech signals). This architecture can complete the basic "hear-understand-speak" closed loop, but due to the inherent defects of traditional NLP models, such as weak semantic understanding, shallow logical reasoning, and poor contextual memory, the naturalness, intelligence, and human-likeness of the interaction have been difficult to improve for a long time.
[0003] With the rapid iteration and widespread adoption of Large Language Model (LLM) technology (an AI model with powerful semantic understanding and text generation capabilities), models such as Qwen3, Llama3, and DeepSeek-LLM, with their powerful semantic understanding, long text modeling, multi-turn dialogue management, logic generation, and emotional expression capabilities, have completely replaced traditional NLP modules and become the core driving engine of the next generation of AI voice interaction systems. Large models, with their powerful semantic understanding, long text context modeling, multi-turn dialogue management, emotional expression, and logic generation capabilities, enable AI responses to be more accurate, more human-like, and more logical. However, this also places more stringent engineering requirements on the backend speech synthesis process, demanding low latency, high coherence, and high flexibility.
[0004] To reduce speech synthesis latency, the industry has proposed two main technical approaches, but both have insurmountable drawbacks, mainly manifested in the following ways:
[0005] Category 1: Traditional non-end-to-end TTS solutions: This approach maintains a serial execution flow where the large model first generates the complete text, and only after all the text is output does the TTS synthesis begin. Its main drawback is:
[0006] (1) The double delay has a response speed problem: users must wait for the large model to complete the generation of the whole text before they can wait for the TTS to complete the audio synthesis. The total delay is equal to the text generation delay plus the speech synthesis delay, resulting in obvious interaction stutter. Users have to endure the double delay of text generation and speech synthesis, which seriously affects the voice interaction experience.
[0007] (2) Difficulty in achieving streaming output: In scenarios involving long texts and multi-turn dialogues, the waiting time increases exponentially, and the user experience deteriorates sharply;
[0008] (3) Low resource utilization: The large model and the TTS module run in sequence, resulting in empty computing resources, inability to work in parallel, low resource utilization and insufficient real-time performance.
[0009] The second type: End-to-end integrated large-scale model speech generation solution: This approach integrates ASR, LLM, and TTS into a single ultra-large model. The model directly outputs speech signals or speech tokens (token: the smallest semantic unit that the model can recognize, divided into text tokens and speech tokens, which are digital representations of text and speech, respectively) while generating text tokens, attempting to shorten the link and reduce latency from an architectural perspective. However, in actual implementation, it has the following fatal shortcomings:
[0010] (1) High training and deployment costs: It requires a large amount of “speech-text” strongly paired labeled data, and the data collection, cleaning and labeling cycle is long and expensive, making it difficult to use commercially;
[0011] (2) Extremely high computing power threshold: The model has a huge number of parameters, and both training and inference require high-end computing power clusters, which are difficult for small and medium-sized enterprises to deploy;
[0012] (3) High coupling and poor flexibility: The end-to-end solution model is highly coupled. Changing the timbre, adjusting the tone, modifying the speech rate, fixing bugs, and adapting to the scene all require retraining the entire model (full retraining), which results in poor flexibility and inability to iterate quickly.
[0013] (4) Uncontrollable maintenance costs: Once the model is released, it is difficult to quickly respond to the personalized needs of the business side and cannot meet the needs of frequent changes in commercial projects;
[0014] (5) Contextual consistency is difficult to guarantee: When generating long texts in a streaming manner, there are easy to have broken tone, drifting tone, and inconsistent emotions.
[0015] In summary, none of the existing technologies can simultaneously meet the commercial requirements of low latency, high flexibility, low cost, high synthesis quality, and strong contextual coherence. Summary of the Invention
[0016] Therefore, the purpose of this invention is to propose a bidirectional streaming speech synthesis system and method based on a large model. Through module decoupling and full-process bidirectional streaming processing (streaming processing: data is not input all at once, but is input segment by segment / block by block and processed in real time with incremental output), it can generate text, synthesize speech, and output audio simultaneously. While ensuring the naturalness of speech and consistency with context, the interaction latency is reduced to the millisecond level. It achieves bidirectional streaming speech synthesis with a decoupled architecture, full-process streaming, context traceability, easy timbre replacement, and lightweight deployment. At the same time, it greatly improves the system customization flexibility and reduces training and deployment costs. It is suitable for low-latency streaming speech synthesis in AI voice interaction scenarios driven by large models.
[0017] This invention provides a bidirectional streaming speech synthesis system based on a large model, comprising: a system configuration and parameter scheduling unit, a streaming text semantic extraction module, an audio token generation module, a streaming Code2Wav module, a context management unit, and an audio output and splicing optimization unit;
[0018] The system configuration and parameter scheduling unit is used to configure text segmentation semantic threshold, speech token generation granularity, streaming audio output time slice parameters, System Prompt (system settings, basic instructions provided for large models, defining the model's working role, dialogue rules, etc.), semantic integrity rules, and dialogue context management rules.
[0019] The streaming text semantic extraction module is used to receive streaming text output by the large model in real time, intelligently divide it into blocks according to semantic integrity rules, and extract the semantic features of the current text block (semantic features: digital features obtained after encoding the text, including information such as the semantics, sentiment, and tone of the text) and the high-level semantic features of the historical text.
[0020] The audio token generation module has a built-in DT LLM large model inference unit, which is used to concatenate the System Prompt, historically generated voice tokens, historical text semantic features, and the text token of the current text block into an input sequence, and generate voice tokens incrementally in a streaming manner.
[0021] Specifically, the DT LLM large model inference unit performs segment-by-segment / token-by-token inference based on the concatenated input sequence, incrementally generating the speech token corresponding to the current text block without waiting for the complete text sentence input. The speech token contains key speech synthesis information such as pronunciation, intonation, speech rate, and timbre.
[0022] The streaming Code2Wav module is a time-slice driven streaming vocoder (vocoder: a module that converts the digital representation of speech into an actual audio waveform), used to decode the speech token into an audio waveform block by block according to a preset time slice and output it in real time.
[0023] The context management unit is used to store and dynamically update historically generated speech tokens and historical text semantic features to ensure the contextual consistency of synthesized speech;
[0024] The audio output and splicing optimization unit is used for real-time output of audio blocks, smooth splicing between blocks, and optimization of the smoothness of the ending. All audio blocks are spliced at the output end to form a complete streaming speech.
[0025] Furthermore, the streaming text semantic extraction module executes the semantic integrity rules upon which the intelligent text segmentation is based, and makes joint decisions based on the semantic threshold of text segmentation, contextual information, dialogue sentiment, tone features, and the ability of speech synthesis to be independently synthesized, so that each text segment is semantically complete, can be independently synthesized, and is not broken.
[0026] Furthermore, the input to the audio token generation module is a sequence of four types of information: System Prompt, historically generated voice tokens, historical text high-level semantic features, and the text token of the current text block, which are used to maintain consistency in timbre, tone, emotion, and semantics throughout the process.
[0027] Furthermore, the streaming Code2Wav module uses millisecond-level (preferably 40ms-80ms) time slice parameters to segment, decode, and output the voice token block by block, thereby generating, synthesizing, and outputting a block.
[0028] To ensure a natural and coherent listening experience, the system takes the following measures during conversion output: whenever an audio waveform block is synthesized, the system will immediately play or output it, allowing users to hear the sound even before the text sentence is fully generated.
[0029] Furthermore, the audio token generation module employs two streaming generation granularities: segment-by-segment reasoning and token-by-token reasoning, to generate speech information that includes pronunciation, intonation, speech rate, and timbre.
[0030] The generation granularity of speech tokens affects the step size of the large language model (DT LLM) during streaming inference. The system supports "segment-by-segment" or "token-by-token" inference to adapt to different latency requirements and hardware performance scenarios.
[0031] This invention also provides a bidirectional streaming speech synthesis method based on a large model, applied to the bidirectional streaming speech synthesis system based on a large model as described above, comprising the following steps:
[0032] S1. Construct a bidirectional streaming speech synthesis system and complete the configuration of text block semantic threshold, speech token generation granularity, streaming audio output time slice parameters, semantic integrity rules, context management rules and System Prompt.
[0033] S2. Receive the streaming text output by the large model in real time, divide it into semantically independent and complete text blocks according to the semantic integrity rules, and extract the semantic features of the current text block and the high-level semantic features of the historical text.
[0034] S3. Concatenate the System Prompt (system settings), historically generated voice tokens, historical text semantic features, and the text token of the current text block into an input sequence, and generate the corresponding voice tokens incrementally through the DT LLM large model.
[0035] Specifically, the splicing process follows a full-process streaming approach, which is mainly characterized by the following features:
[0036] Serialization integration: The system linearly concatenates the above four types of information in a specific order (usually "instruction → historical background → current input") to form a streaming input sequence that can be understood by the large model.
[0037] Dynamic Updates (Incremental Concatenation): The concatenation method in this invention is not static. As the dialogue progresses, the system uses a context synchronization management mechanism to synchronize each newly generated voice token and text features to the system context library in real time. When processing the next text block, the previous "current" content is automatically converted into "historical" content to participate in the new concatenation.
[0038] Cross-modal fusion: This method combines textual information (Token), speech information (Token), and abstract semantic features (digital features). This approach breaks through the limitations of traditional text generation, enabling models to reason based on the dual context of text and speech.
[0039] Through this detailed splicing method, the system achieves the following effects:
[0040] The reasoning basis is formed by using the concatenated total sequence as input to the DT LLM, enabling incremental reasoning on a segment-by-segment or token-by-token basis.
[0041] Ensure consistency: Ensure that the model fully considers previous pronunciation, intonation, speech rate and timbre information when generating the current speech token, so as to avoid tone breaks or timbre drift when synthesizing long texts.
[0042] The streaming incremental generation eliminates the need to wait for a large model to output a complete text sentence, enabling simultaneous text generation and speech synthesis.
[0043] S4, the streaming Code2Wav module decodes the voice token into audio waveforms block by block according to preset time slices, outputs them in real time and splices them smoothly.
[0044] The streaming decoding and synthesis of this invention is a key conversion step in realizing "code-to-waveform" conversion. It utilizes Code2Wav [a vocoder technology that converts speech tokens (encoded data) into audio waveforms (Wav)] to decode the speech tokens (digital representations of speech) into actual audio waveforms (Wav). Adopting a block-by-block decoding and real-time output principle, it does not wait for all tokens to be generated but synthesizes them piece by piece. The user hears the speech earlier than with traditional whole-sentence synthesis schemes, reducing interaction latency to milliseconds.
[0045] S5. Process subsequent streaming text in a loop. After the text generation is completed, optimize and complete the ending audio to complete the bidirectional streaming speech synthesis.
[0046] After the text generation is complete, the system will optimize and splice the last audio block to ensure the integrity and fluency of the overall speech.
[0047] Further, step S2 includes:
[0048] S21. Receive streaming text output from large models in real time. The text flows in incrementally, word by word, and token by token, rather than being input as a whole sentence at once.
[0049] S22. Based on semantic integrity rules and preset semantic thresholds, the streaming text is dynamically divided into multiple semantically independent and complete text chunks.
[0050] S23. Perform tokenizer segmentation on the text block and convert it into a text token sequence;
[0051] S24. Encode the text token using a text encoder to obtain the semantic feature vector of the current text block;
[0052] S25. Extract high-level semantic features of historical text from the context management unit, including contextual information, dialogue sentiment, tone features, and logical connection features.
[0053] Furthermore, step S3 also includes: synchronously updating the generated voice token and text features to the context management unit for consistency constraints of the next text block.
[0054] The bidirectional streaming speech synthesis method based on a large model in this invention supports complete decoupling between the large model and the speech synthesis module. The large model can be replaced independently, the timbre can be changed independently, and the speech rate and intonation can be adjusted independently without retraining the entire system.
[0055] This invention completely decouples large-scale model inference from speech synthesis, with each module trained, updated, and deployed independently. It features streaming text input and streaming audio output, with end-to-end incremental processing, eliminating waiting and blocking. Text is segmented based on semantic integrity rules to ensure uninterrupted and natural synthesized speech. Historical text features and historically generated speech tokens are used as input, creating strong contextual relationships and maintaining consistency in tone, timbre, and emotion throughout. Streaming Code2Wav: based on time-slice block-by-block decoding, it generates, synthesizes, and outputs a block simultaneously, achieving low latency.
[0056] Preferably, the bidirectional streaming speech synthesis system based on large models of the present invention is compatible with almost all mainstream large models such as Qwen3, Llama3, and DeepSeek, and can be embedded in intelligent customer service, smart speakers, in-vehicle voice, humanoid robots, speech multimodal large model systems, and real-time dialogue hardware and software devices. It has strong compatibility and is applicable to a wide range of scenarios.
[0057] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the bidirectional streaming speech synthesis method based on a large model as described above.
[0058] The present invention also provides a computer device, the computer device including a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the bidirectional streaming speech synthesis method based on a large model as described above.
[0059] Compared with the prior art, the beneficial effects of the present invention are as follows:
[0060] The bidirectional streaming speech synthesis system and method based on a large model provided by this invention adopts full-process bidirectional streaming processing, eliminating the need to wait for the large model to generate complete text. It achieves simultaneous text generation and speech synthesis, with users only experiencing extremely short single-text block processing latency (40ms-80ms). This effectively solves the double latency problem of traditional solutions, achieving near real-time voice interaction and significantly reducing latency compared to traditional TTS. The large model, speech token generation, and vocoder are completely decoupled, with each module working independently. If it is necessary to change the synthesized timbre, adjust the tone and speed, or fix bugs in the speech synthesis module, only the corresponding module needs to be modified / fine-tuned. Changing the timbre, adjusting the tone, fixing bugs, and customizing scenarios do not require retraining the entire system, reducing maintenance costs. It also eliminates the need for massive amounts of speech-text data. This paired data trains a single model, and each module can be trained independently and optimized on demand, significantly reducing computing power requirements. During deployment, module parameters can be flexibly adjusted according to the scenario, greatly reducing training and deployment costs and making it easier to implement in real-world projects. Through dual constraints of historically generated voice tokens and historical text features, the context is highly consistent, resulting in high synthesis quality. This ensures the consistency of semantics, tone, and timbre in speech during multi-turn dialogues and long text synthesis, avoiding the semantic fragmentation problem of streaming synthesis. There is no drift or abruptness. At the same time, relying on the strong semantic understanding capabilities of the large model, the synthesized speech is more natural and expressive. The large model and TTS are processed in parallel, with no idle computing resources, high resource utilization, and high inference efficiency, supporting high concurrency and large-scale deployment. The system framework is compatible with mainstream large language models such as Qwen3 and Llama3, and can be directly applied to various AI voice interaction scenarios such as intelligent customer service, smart speakers, and human-computer dialogue robots, with broad prospects for promotion and application. Attached Figure Description
[0061] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention.
[0062] In the attached diagram:
[0063] Figure 1 A system framework diagram for implementing bidirectional streaming voice interaction in an application example of the present invention;
[0064] Figure 2 This is a technical framework diagram of a bidirectional streaming speech synthesis system based on a large model, according to an embodiment of the present invention.
[0065] Figure 3 This is a schematic diagram of the configuration of a computer device according to an embodiment of the present invention. Detailed Implementation
[0066] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of systems and products consistent with some aspects of this disclosure as detailed in the appended claims.
[0067] The terminology used in this disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The singular forms “a,” “the,” and “the” as used in this disclosure and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.
[0068] It should be understood that although the terms first, second, third, etc., may be used in this disclosure to describe various information, such information should not be limited to these terms. These terms are used only to distinguish information of the same type from one another. For example, without departing from the scope of this disclosure, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."
[0069] The embodiments of the present invention will be described in further detail below.
[0070] Example
[0071] This invention provides a bidirectional streaming speech synthesis system based on a large model, comprising: a system configuration and parameter scheduling unit, a streaming text semantic extraction module, an audio token generation module, a streaming Code2Wav module, a context management unit, and an audio output and splicing optimization unit.
[0072] The modules mentioned above achieve bidirectional communication through a standardized streaming data interface, and work together to complete the entire process of bidirectional streaming speech synthesis, including text stream input, semantic segmentation, feature extraction, streaming generation of speech tokens, streaming decoding, and real-time audio output.
[0073] The system configuration and parameter scheduling unit is responsible for system initialization, parameter configuration, rule loading, and execution scheduling. Configuration content includes:
[0074] Text chunk semantic threshold: used to quantitatively determine whether a text chunk has reached a semantically complete and synthesizable state;
[0075] Voice token generation granularity: Supports generation segment by segment or token by token;
[0076] Streaming audio output time slice parameters: preferably 40ms-80ms, used to control voice token segmentation and audio output granularity;
[0077] System Prompt instruction set: Defines model roles, speech styles, and dialogue rules;
[0078] Semantic integrity rules: guide intelligent text segmentation, ensuring that the segmented blocks are independent and can be synthesized without breaks;
[0079] Dialogue context management rules: strategies for storing, updating, and retrieving historically generated voice tokens and historical text features.
[0080] The System Prompt comprises two main dimensions:
[0081] (1) Model working roles: such as intelligent customer service, voice assistant, companion robot, in-vehicle assistant, etc.;
[0082] (2) Dialogue rules: including tone of voice, emotional expression, speech rate constraints, pronunciation norms, and context preservation rules.
[0083] The streaming text semantic extraction module is used to receive incremental streaming text output from large models in real time. It completes real-time reception of streaming text character by character / token by token, intelligent text segmentation based on semantic integrity rules and semantic thresholds, text tokenizer segmentation and encoding, semantic feature extraction of the current text block, and high-level semantic feature extraction of historical text. The output data includes: the text token sequence of the current text block, the semantic feature vector of the current text block, and the semantic feature vector of the historical context.
[0084] The audio token generation module incorporates a DT LLM (DeepThinkingLargeLanguageModel) inference unit. Taking four types of information sequences as input, it performs multi-dimensional information serialization and concatenation, streaming incremental inference, and generates voice tokens segment by segment / token by token, with context-synchronized updates. Voice tokens contain all key information related to speech synthesis, including pronunciation, intonation, speech rate, volume, timbre, and emotion.
[0085] The streaming Code2Wav module is a time-slice driven streaming vocoder responsible for converting voice tokens into playable audio waveforms. It performs the following steps: segmenting voice tokens into time slices; decoding each slice to generate an audio waveform; smoothing transitions between slices; and real-time audio stream output. The optimal time slice parameter is 40ms-80ms, balancing low latency and auditory continuity.
[0086] The context management unit is used to store historical generated voice token sequences, store historical text high-level semantic feature vectors, dynamically maintain the context window, and provide consistency constraint inputs for the audio token generation module.
[0087] The audio output and splicing optimization unit is used for real-time playback / network streaming of audio blocks; inter-block overlap and smooth splicing to eliminate pops, gaps, and abrupt pauses; end segment completion, mute trimming, and smoothness enhancement; synthesis quality monitoring and anomaly compensation.
[0088] Figure 2 This paper illustrates the technical framework of a bidirectional streaming speech synthesis system based on a large model, according to an embodiment of the present invention.
[0089] This invention also provides a bidirectional streaming speech synthesis method based on a large model, applied to the bidirectional streaming speech synthesis system based on a large model as described above, comprising the following steps:
[0090] S1. System construction and global parameter initialization, including:
[0091] Build a bidirectional streaming speech synthesis system that includes a streaming text semantic extraction module, an audio token generation module, a streaming Code2Wav module, a context management unit, a configuration unit, and an audio optimization unit;
[0092] Configure the following parameters:
[0093] Text segmentation semantic threshold;
[0094] Voice token generation granularity (segment-by-segment / token-by-token);
[0095] Streaming audio output time slice parameters (40ms-80ms);
[0096] Load the System Prompt and define the model's role and voice style;
[0097] Load semantic integrity rules;
[0098] Load the dialogue context management rules.
[0099] S2, streaming text reception and semantic extraction processing, including:
[0100] S21. Receive streaming text output from large models in real time. The text flows in incrementally, word by word, and token by token, rather than being input as a whole sentence at once.
[0101] S22. Based on semantic integrity rules and preset semantic thresholds, the streaming text is dynamically divided into multiple semantically independent and complete text chunks.
[0102] S23. Perform tokenizer segmentation on the text block and convert it into a text token sequence;
[0103] S24. Encode the text token using a text encoder to obtain the semantic feature vector of the current text block;
[0104] S25. Extract high-level semantic features of historical text from the context management unit, including contextual information, dialogue sentiment, tone features, and logical connection features.
[0105] S3, Streaming Audio Token Generation, including:
[0106] S31. Model Input Serialization and Concatenation: The four parts, System Prompt, historical generated speech tokens, historical text high-level semantic features, and current text block text token, are linearly concatenated in a fixed order to form a unified DT LLM input sequence.
[0107] S32 and DT LLM perform streaming incremental inference based on the concatenated input, without waiting for the complete sentence to be generated, and generate the speech token corresponding to the current text block segment by segment or token by token.
[0108] S33, Context Synchronization Management: The generated voice token and text features are written into the context management unit in real time to provide consistency constraints for the next piece of text processing.
[0109] S4, streaming speech synthesis and real-time audio output, including:
[0110] S41, The streaming Code2Wav module divides the voice token into voice token blocks according to preset time slice parameters;
[0111] S42. Decode block by block, converting the voice token into continuous audio waveform data;
[0112] S43. After each audio waveform is synthesized, it is immediately output / played out, realizing "generating one, synthesizing one, and outputting one".
[0113] S44. Perform smooth splicing on adjacent audio blocks to ensure a continuous listening experience without gaps, pops, or abrupt pauses.
[0114] S5, cyclic streaming processing and cleanup optimization, including:
[0115] S51. Repeat steps S2-S4 to continue processing the streaming text output by the large model.
[0116] S52. After the large model text is generated, the last audio block is finished, muted, and smoothness is optimized.
[0117] S53. Output complete streaming speech to complete this two-way streaming speech synthesis.
[0118] Semantic integrity rules are a key innovation in achieving "flowing sentence continuity" in this invention, and their determination criteria include:
[0119] Semantic threshold determination: When the semantic saturation of a text fragment reaches a preset threshold, segmentation is triggered;
[0120] Contextual logical dependence: Combine the preceding semantics to avoid segmentation between logical connectors and modifying relationships;
[0121] Emotional unit integrity: Ensure that emotional expression is complete and not interrupted midway;
[0122] Speech synthesis must be able to synthesize independently: the segments must be able to synthesize speech independently, without ambiguity or fragmentation.
[0123] By using this semantic integrity rule, the embodiments of the present invention effectively solve the problems of "hard segmentation, hard pause, and semantic fragmentation" in traditional streaming TTS while maintaining extremely low latency.
[0124] This invention, for the first time, employs cross-modal input of four types of information: commands, historical speech, historical text, and current text, as detailed below:
[0125] Command layer: System Prompt has fixed roles, styles, and tones;
[0126] Historical speech layer: The generated speech tokens ensure continuity of timbre and intonation;
[0127] Historical text layer: Contextual semantic features ensure logical coherence;
[0128] Current text layer: Content to be synthesized in real time.
[0129] This mechanism avoids problems such as tone breaks, voice drift, emotional abrupt changes, and semantic incoherence in long texts / multi-turn dialogues.
[0130] This invention transforms the traditional Code2Wav into a time-slice-driven, low-latency streaming vocoder: voice tokens are segmented into 40ms-80ms time slices; decoding and output are performed block by block with no waiting time; inter-block overlap is smooth, eliminating splicing artifacts; and real-time playback, network streaming, and local caching are supported. Users hear the voice 300ms-2000ms earlier than with traditional TTS, resulting in an interactive experience close to real-life conversation.
[0131] This invention completely decouples the large model from speech synthesis, allowing for independent replacement of the LLM, independent change of timbre, and independent adjustment of speech rate; it eliminates the need to train an end-to-end giant model; it requires no massive speech-text pairing data; and it is lightweight in deployment, making it easy to implement in real-world projects.
[0132] Application examples
[0133] The bidirectional streaming speech synthesis system based on a large model of the present invention includes the following specific application examples in practical application scenarios:
[0134] Application Example 1: Intelligent Customer Service Scenario (Single-Turn Short Text):
[0135] User speech is converted into text by ASR and then fed into the large model;
[0136] Large models output reply text word by word in a streaming manner;
[0137] The streaming text semantic extraction module segments phrase blocks according to semantic thresholds;
[0138] The audio token generation module generates voice tokens block by block;
[0139] Streaming Code2Wav is decoded and output in 60ms time slices;
[0140] Users hear the speech 3-5 words after the large model outputs, with an overall end-to-end latency of <150ms.
[0141] Application Example 2: Long Text Broadcast (Multiple Rounds / Large Responses):
[0142] The large model continuously outputs long text streams;
[0143] The system is divided into blocks according to semantic integrity rules;
[0144] The context management unit retains all historically generated voice tokens and semantic features throughout the process;
[0145] The synthesized speech maintains a consistent tone, stable timbre, and no breaks throughout.
[0146] The ending is automatically optimized and completed, resulting in a smooth and natural flow.
[0147] Application Example 3: Quickly change timbre (without retraining the model):
[0148] Keep DT LLM unchanged;
[0149] Only replace the timbre model of the streaming Code2Wav;
[0150] The new tone is output immediately after restarting, and the whole process takes less than 10 seconds, requiring no training or data.
[0151] Application Example 4: Accessing a large-scale multimodal speech model:
[0152] The system of this invention is embedded as an independent DT-TTS module within a multimodal speech model architecture, realizing a full-link bidirectional streaming (e.g., speech input, multimodal model, streaming text, DT-TTS, and real-time speech output) process. Figure 1 As shown in the image, it has low latency and natural interaction.
[0153] This invention can be deployed on cloud servers, edge computing devices, and edge chips (Android / Linux / embedded systems), supporting real-time voice streaming, local offline synthesis, and low-power device operation. It is ready for commercialization and can be widely applied to: AI intelligent assistants, intelligent customer service systems, in-vehicle voice interaction, humanoid robot / robot dog voice interaction, smart hardware / speakers, educational companion robots, and a large-scale multimodal voice ecosystem, possessing broad market prospects and scalability.
[0154] This invention also provides a computer device. Figure 3 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present invention; see the accompanying drawings. Figure 3 As shown, the computer device includes: an input device 23, an output device 24, a memory 22, and a processor 21; the memory 22 is used to store one or more programs; when the one or more programs are executed by the one or more processors 21, the one or more processors 21 implement the bidirectional streaming speech synthesis method based on a large model as provided in the above embodiments; wherein the input device 23, the output device 24, the memory 22, and the processor 21 can be connected via a bus or other means. Figure 3 Taking the example of a connection between China and Israel via a bus.
[0155] The memory 22, as a read / write storage medium for a computing device, can be used to store software programs and computer-executable programs, such as the program instructions corresponding to the bidirectional streaming speech synthesis method based on a large model as described in this embodiment of the invention. The memory 22 may mainly include a program storage area and a data storage area. The program storage area may store the operating system and at least one application program required for a function; the data storage area may store data created based on the use of the device. Furthermore, the memory 22 may include high-speed random access memory and non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 22 may further include memory remotely located relative to the processor 21, and these remote memories can be connected to the device via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.
[0156] Input device 23 can be used to receive input digital or character information, and generate key signal inputs related to user settings and function control of the device; output device 24 may include display devices such as a display screen.
[0157] The processor 21 executes various functional applications and data processing of the device by running software programs, instructions and modules stored in the memory 22, thereby realizing the above-mentioned bidirectional streaming speech synthesis method based on a large model.
[0158] The computer equipment provided above can be used to execute the bidirectional streaming speech synthesis method based on a large model provided in the above embodiments, and has corresponding functions and beneficial effects.
[0159] This invention also provides a storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to perform the bidirectional streaming speech synthesis method based on a large model as provided in the above embodiments. The storage medium can be any type of memory device or storage device, including: mounting media such as CD-ROM, floppy disk, or magnetic tape; computer system memory or random access memory such as DRAM, DDRRAM, SRAM, EDORAM, Rambus RAM, etc.; non-volatile memory such as flash memory, magnetic media (e.g., hard disk or optical storage); registers or other similar types of memory elements; the storage medium may also include other types of memory or combinations thereof; furthermore, the storage medium may reside in a first computer system in which the program is executed, or it may reside in a different second computer system connected to the first computer system via a network (such as the Internet); the second computer system can provide program instructions to the first computer for execution. The storage medium includes two or more storage media that may reside in different locations (e.g., in different computer systems connected via a network). The storage medium may store program instructions (e.g., specifically implemented as a computer program) executable by one or more processors.
[0160] Of course, the computer-executable instructions provided in the embodiments of the present invention are not limited to the bidirectional streaming speech synthesis method based on a large model as described in the above embodiments, but can also execute related operations in the bidirectional streaming speech synthesis method based on a large model provided in any embodiment of the present invention.
[0161] The technical solution of the present invention has been described above with reference to the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will all fall within the scope of protection of the present invention.
[0162] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. A large model-based bidirectional streaming speech synthesis system, characterized by, include: System configuration and parameter scheduling unit, streaming text semantic extraction module, audio token generation module, streaming Code2Wav module, context management unit, audio output and splicing optimization unit; The system configuration and parameter scheduling unit is used to configure text block semantic threshold, voice token generation granularity, streaming audio output time slice parameters, System Prompt, semantic integrity rules, and dialogue context management rules. The streaming text semantic extraction module is used to receive streaming text output by a large model in real time, intelligently segment it according to semantic integrity rules, and extract the semantic features of the current text block and the high-level semantic features of historical text. The audio token generation module has a built-in DT LLM large model inference unit, which is used to concatenate the System Prompt, historically generated voice tokens, historical text semantic features, and the text token of the current text block into an input sequence, and generate voice tokens incrementally in a streaming manner. The streaming Code2Wav module is a time-slice driven streaming vocoder, used to decode the voice token into audio waveforms block by block according to preset time slices and output them in real time. The context management unit is used to store and dynamically update historically generated speech tokens and historical text semantic features to ensure the contextual consistency of synthesized speech; The audio output and splicing optimization unit is used for real-time output of audio blocks, smooth splicing between blocks, and optimization of the smoothness of the ending.
2. The large model-based bidirectional streaming speech synthesis system of claim 1, wherein, The streaming text semantic extraction module executes the semantic integrity rules on which the intelligent text segmentation is based. It makes joint decisions based on the semantic threshold of text segmentation, context information, dialogue sentiment, tone features, and the ability of speech synthesis to be independently synthesized, so that each text segment is semantically complete, can be independently synthesized, and is not broken. 3.The large model based bidirectional streaming speech synthesis system of claim 1, wherein, The audio token generation module takes as input four types of information concatenation sequences: System Prompt, historically generated voice tokens, historical text high-level semantic features, and the text token of the current text block, which are used to maintain consistency in timbre, tone, emotion, and semantics throughout the process.
4. The large model-based bidirectional streaming speech synthesis system of claim 1, wherein, The streaming Code2Wav module uses millisecond-level time slice parameters to segment, decode, and output voice tokens block by block, thereby generating, synthesizing, and outputting a block.
5. The large model-based bidirectional streaming speech synthesis system of claim 1, wherein, The audio token generation module employs two streaming generation granularities: segment-by-segment reasoning and token-by-token reasoning, to generate speech information that includes pronunciation, intonation, speech rate, and timbre.
6. A method for large model based bidirectional streaming speech synthesis, applied to the large model based bidirectional streaming speech synthesis system of any one of claims 1-5, characterized in that, Includes the following steps: S1. Construct a bidirectional streaming speech synthesis system and complete the configuration of text block semantic threshold, speech token generation granularity, streaming audio output time slice parameters, semantic integrity rules, context management rules and System Prompt. S2. Receive the streaming text output by the large model in real time, divide it into semantically independent and complete text blocks according to the semantic integrity rules, and extract the semantic features of the current text block and the high-level semantic features of the historical text. S3. Concatenate the System Prompt, historically generated speech tokens, historical text semantic features, and the text token of the current text block into an input sequence, and generate the corresponding speech tokens incrementally through the DT LLM large model. S4, the streaming Code2Wav module decodes the voice token into audio waveforms block by block according to preset time slices, outputs them in real time and splices them smoothly. S5. Process subsequent streaming text in a loop. After the text generation is completed, optimize and complete the ending audio to complete the bidirectional streaming speech synthesis.
7. The big model based bidirectional streaming speech synthesis method of claim 6, wherein, Step S2 includes: S21. Receive streaming text output from large models in real time. The text flows in incrementally, word by word, and token by token, rather than being input as a whole sentence at once. S22. Based on semantic integrity rules and preset semantic thresholds, the streaming text is dynamically divided into multiple semantically independent and complete text blocks; S23. Perform tokenizer segmentation on the text block and convert it into a text token sequence; S24. Encode the text token using a text encoder to obtain the semantic feature vector of the current text block; S25. Extract high-level semantic features of historical text from the context management unit, including contextual information, dialogue sentiment, tone features, and logical connection features.
8. The big model based bidirectional streaming speech synthesis method of claim 6, wherein, Step S3 also includes: synchronously updating the generated voice token and text features to the context management unit for consistency constraints of the next text block.
9. A computer readable storage medium having stored thereon a computer program, characterized in that, When the program is executed by the processor, it implements the steps of the bidirectional streaming speech synthesis method based on a large model as described in any one of claims 6-8.
10. A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the bidirectional streaming speech synthesis method based on a large model as described in any one of claims 6-8.