Real-time processing method, device, gateway equipment and system for converting streaming text into voice of AI (Artificial Intelligence) large model

Abstract

The invention provides an AI large model streaming text-to-voice real-time processing method, device, gateway equipment and system, and relates to the technical field of artificial intelligence. According to the method, an OpenResty gateway technology, SSE streaming communication, text fragmentation and parallel text-to-voice processing are organically combined, and an SSE data stream returned by a large language model is analyzed and split in real time and a text-to-voice service is called in parallel in a gateway layer, so that an assembly line parallel processing mode of text generation and voice synthesis is realized; the problems of high delay, low resource utilization rate and poor user experience in a long text scene are relieved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method, apparatus, gateway device, and system for real-time processing of streaming text-to-speech using large AI models. Background Technology

[0002] In recent years, the field of artificial intelligence has undergone a profound technological revolution, with the most notable breakthroughs in Large Language Models (LLMs). From the BERT model in 2018, to GPT-3 in 2020, then ChatGPT at the end of 2022, and numerous other models such as DeepSeek and Tongyi Qianwen, large language models have become a core technology in natural language processing. These models, based on the Transformer architecture, have achieved powerful language understanding and generation capabilities through pre-training on massive amounts of text data.

[0003] Large language models work by generating text content token by token in an autoregressive manner. In practical applications, when a user poses a question or request to the model, the model predicts and generates the next most likely word based on the input context. This token-by-token generation characteristic provides a natural technical foundation for streaming output. To improve user experience, current large language model services generally adopt streaming response technology, which continuously pushes generated content to the client during text generation, rather than waiting for all content to be generated and then returning it all at once. This approach allows users to see the model's thinking and generation process in real time, greatly reducing waiting anxiety and enhancing the immediacy and immersion of the interaction.

[0004] In practical business applications, large language models are widely used in various fields such as intelligent voice customer service systems, online interviews, online education, content creation tools, intelligent question-and-answer assistants, and code generation tools. Especially in scenarios requiring the generation of long text content, such as report writing, article creation, and tutorial writing, large language models have demonstrated outstanding capabilities. However, as the length of generated content increases, how to efficiently convert this text content into speech output has become a pressing technical challenge. Summary of the Invention

[0005] The purpose of this invention is to provide a method, apparatus, gateway device and system for real-time processing of streaming text to speech in AI large-scale models, so as to alleviate the problems of high latency, low resource utilization and poor user experience in long text scenarios.

[0006] In a first aspect, the present invention provides a real-time method for streaming text-to-speech processing of large AI models, applied to gateway devices deployed with the OpenResty platform. The real-time method for streaming text-to-speech processing of large AI models includes: It receives a question request from the client and forwards the question request to the Large Language Model Service, which is used to return the generated text content by sending an event SSE data stream protocol through the server. Intercept and parse the SSE data stream returned by the large language model service in real time to extract streaming text data; The streaming text data is segmented to obtain at least one text fragment. For each text segment, the text-to-speech service is invoked in parallel to generate an audio segment; The audio clips returned by the text-to-speech service are pushed to the client in real time.

[0007] In an optional implementation, the server-sent event SSE data stream returned by the large language model service is intercepted and parsed in real time to extract streaming text data, including: Intercept the SSE data stream returned by the large language model service in real time through the body_filter hook; Parse the SSE data stream protocol on the SSE data stream; The parsed streaming text data is stored in the session data buffer.

[0008] In an optional implementation, the streaming text data is segmented to obtain at least one text fragment, including: The intelligent fragmentation algorithm, which employs a multi-level fragmentation strategy, splits streaming text data into one or more text segments. The multi-level fragmentation strategy includes a first priority strategy, a second priority strategy, and a third priority strategy. The first priority strategy fragments the text when a sentence terminator is detected. The second priority strategy fragments the text at secondary punctuation marks when the cumulative text length exceeds a preset first threshold and no sentence terminator is detected. The third priority strategy forces fragmentation when the cumulative text length exceeds a preset second threshold. The second threshold is greater than the first threshold.

[0009] In an optional implementation, the intelligent sharding algorithm is further configured as follows: For the initial text fragment obtained through the first priority strategy, it is determined whether the length of the initial text fragment is less than the preset minimum length threshold. If it is less than the minimum length threshold, the initial text fragment is buffered and merged with the subsequent text data. Then, ignoring the sentence termination character at the merging point, the merged text is fragmented until the length of the fragmented text reaches the minimum length threshold.

[0010] In optional implementations, the AI ​​large-model streaming text-to-speech real-time processing method further includes: By monitoring the average response time and current load of the text-to-speech service, the sharding threshold in the intelligent sharding algorithm is dynamically adjusted. The sharding threshold includes one or more of the first threshold, the second threshold, and the minimum length threshold.

[0011] In optional implementations, the AI ​​large-model streaming text-to-speech real-time processing method further includes: During the process of segmenting streaming text data, at least one of the following processes is performed on the streaming text data: data desensitization, compliance check, and semantic caching; During the process of calling the text-to-speech service, fault tolerance is performed, which includes at least one of the following: implementing an exponential backoff retry mechanism for failed text-to-speech service calls for individual text segments; setting a timeout for text-to-speech service calls and performing degradation processing after the timeout; and pushing preset background audio to the client to maintain playback continuity when the text-to-speech service is unavailable or the call fails.

[0012] Secondly, the present invention provides a real-time AI large-scale model streaming text-to-speech processing device, applied to a gateway device deployed with the OpenResty platform. The AI ​​large-scale model streaming text-to-speech processing device includes: The forwarding module is used to receive problem requests from clients and forward them to the large language model service. The large language model service is used to return the generated text content by sending an event SSE data stream protocol through the server. The parsing module is used to intercept and parse the SSE data stream returned by the large language model service in real time to extract streaming text data; The fragmentation module is used to process streaming text data into fragments to obtain at least one text segment. The calling module is used to call the text-to-speech service in parallel for each text segment to generate an audio segment; The push module is used to push audio clips returned by the text-to-speech service to the client in real time.

[0013] Thirdly, the present invention provides a gateway device, including a memory and a processor. The memory stores a computer program that can run on the processor. The gateway device is deployed with the OpenResty platform. When the processor executes the computer program, it implements the AI ​​large model streaming text-to-speech real-time processing method of any of the foregoing embodiments.

[0014] Fourthly, the present invention provides an AI large-scale model streaming text-to-speech real-time processing system, including the gateway device of the aforementioned embodiments, and also including a large language model service and a text-to-speech service respectively connected to the gateway device.

[0015] Fifthly, the present invention provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, performs the AI ​​large-scale model streaming text-to-speech real-time processing method according to any of the foregoing embodiments.

[0016] This invention provides a real-time method, apparatus, gateway device, and system for AI large-scale model streaming text-to-speech processing. The method is applied to a gateway device deployed with the OpenResty platform. The method includes: receiving a question request from a client and forwarding the request to a large language model service, which returns generated text content via the SSE data stream protocol; intercepting and parsing the SSE data stream returned by the large language model service in real time to extract streaming text data; segmenting the streaming text data to obtain at least one text fragment; for each text fragment, calling the text-to-speech service in parallel to generate an audio fragment; and pushing the audio fragment returned by the text-to-speech service to the client in real time. This organically combines OpenResty gateway technology, SSE streaming communication, text segmentation, and parallel text-to-speech processing. By performing real-time parsing and segmentation of the SSE data stream returned by the large language model and parallel calling of the text-to-speech service at the gateway layer, a pipelined parallel processing mode for text generation and speech synthesis is achieved, alleviating the problems of high latency, low resource utilization, and poor user experience in long text scenarios. Attached Figure Description

[0017] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0018] Figure 1 This is an example of the delay in a traditional text-to-speech processing workflow; Figure 2 This is a schematic diagram of the system architecture of the streaming processing scheme provided in the embodiments of the present invention; Figure 3 A schematic diagram of pipelined parallel processing of text generation and speech synthesis provided in an embodiment of the present invention; Figure 4 This is a delay example of the streaming processing scheme provided in the embodiments of the present invention; Figure 5 A flowchart illustrating a real-time text-to-speech processing method for a large AI model provided in an embodiment of the present invention; Figure 6 A schematic diagram of the interactive process of a real-time text-to-speech processing method for a large AI model provided in an embodiment of the present invention; Figure 7 This invention provides a request execution order for the OpenResty gateway layer. Figure 8 This invention provides a fault-tolerant processing flow for text-to-speech service calls. Figure 9 This is a schematic diagram of the structure of a real-time AI large-scale model streaming text-to-speech processing device provided in an embodiment of the present invention; Figure 10 This is a schematic diagram of the structure of a gateway device provided in an embodiment of the present invention; Figure 11 This is a schematic diagram of the structure of a real-time AI large-scale model streaming text-to-speech processing system provided in an embodiment of the present invention. Detailed Implementation

[0019] The technical solution of the present invention will be clearly and completely described below with reference to the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] Traditional LLM (Limited Language Management) text-to-speech (TTS) systems typically employ a serial processing approach of "text first, then speech," leading to significant latency and resource consumption spikes in long text scenarios (such as generating 3000-character text). This invention proposes a streaming TTS architecture that real-time parses and processes LLM server-sent events (SSE) at a high-performance gateway layer (OpenResty). By intercepting and parsing the LLM's SSE text stream at the gateway layer, real-time text segmentation and parallel invocation of speech services are achieved, enabling real-time streaming speech synthesis that simultaneously generates text, converts it to speech, and plays the speech. This aims to significantly reduce user waiting time and dramatically improve the real-time performance and fluency of human-computer interaction.

[0021] The following section introduces text-to-speech technology, the SSE data stream protocol, and the OpenResty gateway technology.

[0022] Text-to-speech technology: Text-to-speech (TTS) technology refers to the technology of converting text information into natural and fluent speech output. Early TTS systems were mainly based on rule-based synthesis methods, generating speech through pre-recorded phoneme libraries and concatenation algorithms. Although this method is simple to implement, the generated speech is mechanical and stiff, lacking naturalness and expressiveness.

[0023] In the 21st century, with the development of machine learning technology, parametric synthesis methods began to emerge, the most representative of which is speech synthesis systems based on Hidden Markov Models (HMMs). These systems generate speech parameters through statistical modeling, which significantly improves the naturalness compared to rule-based synthesis, but still suffers from problems such as over-smoothing and lack of personalized expression.

[0024] In recent years, breakthroughs in deep learning technology have brought revolutionary changes to the field of TTS. In 2016, the WaveNet model was the first to use deep neural networks to directly generate audio waveforms, producing speech quality close to that of a real person. Subsequently, end-to-end neural network TTS models such as Tacotron and FastSpeech emerged, which can directly generate high-quality speech from text input end-to-end without the need for complex intermediate representations and manual feature engineering.

[0025] Current mainstream commercial TTS services all employ advanced neural network models. These services can not only generate speech in multiple languages ​​and dialects, but also support advanced features such as multiple voice options, speech rate adjustment, and emotional expression. Modern TTS systems have achieved a considerably high level in terms of naturalness, fluency, and expressiveness, and in some scenarios, they are even difficult to distinguish from human speech.

[0026] However, despite the maturity of TTS technology itself, in practical applications combined with large language models, efficiently converting LLM-generated text into speech output in real time still faces numerous technical challenges. Especially when processing long text content, the high latency caused by traditional batch processing severely impacts the user experience, which is precisely the core problem that this invention aims to solve.

[0027] SSE data stream protocol: Server-Sent Events (SSE) is a technical specification defined in the HTML5 standard for servers to push real-time data to clients. Unlike the request-response model of the traditional Hypertext Transfer Protocol (HTTP), SSE establishes a persistent one-way communication channel, allowing the server to continuously push data events to the client while the connection is maintained, without the client repeatedly initiating requests.

[0028] The SSE protocol is implemented based on the standard HTTP protocol, offering good compatibility and simplicity. When a client initiates an SSE request, the server returns a special Content-Type response header "text / event-stream," indicating that this is an event stream response. The server can then continuously send formatted text messages over the connection, each prefixed with "data:" and separated by two newline characters "\n\n." Upon receiving these messages, the client triggers the corresponding event handling function, thus achieving real-time data transmission.

[0029] SSE offers several unique advantages. First, based on the standard HTTP protocol, SSE can directly bypass enterprise firewalls and proxy servers without requiring special network configuration. Second, the SSE protocol is simple and lightweight, with low implementation costs; the server only needs to set the correct response headers and output data in the correct format. Third, SSE supports an automatic reconnection mechanism; when a connection is unexpectedly lost, the client will automatically attempt to re-establish the connection, ensuring reliable communication. Finally, SSE natively supports an event ID mechanism; the server can assign a unique identifier to each event, allowing the client to resume receiving data from where it left off after a disconnection and reconnection, preventing data loss or duplication.

[0030] In large language model (LLM) applications, the SSE protocol can be used to implement streaming text output. When the LLM starts generating text, it immediately pushes each generated token to the client via the SSE connection. Upon receiving these tokens, the client can render and display them in real time, giving the user a dynamic "typing" effect. This real-time feedback mechanism greatly improves the user experience, making human-computer interaction more natural and fluid.

[0031] A typical SSE data stream format is as follows: Whenever the LLM generates a new text fragment, the server sends a message such as "data: Artificial Intelligence\n\n". The client receives this message and gradually assembles these fragments to form the complete text content. This fragment-by-fragment transmission characteristic provides the technical possibility for real-time processing and conversion at the gateway layer, which is also an important technical basis for the embodiments of this invention.

[0032] OpenResty gateway technology: OpenResty is a high-performance web application platform based on the Nginx core and LuaJIT. The core concept of OpenResty is to combine the high-performance event-driven architecture of Nginx with the flexibility of the Lua scripting language, enabling developers to implement complex business logic at the web server level without writing traditional server-side programs.

[0033] Nginx, one of the world's most widely used high-performance web servers, is renowned for its exceptional concurrency capabilities, extremely low memory footprint, and highly modular design. Nginx employs an event-driven, non-blocking I / O (Input / Output) model, using asynchronous event processing mechanisms to handle tens of thousands of concurrent connections with minimal system resources. This architecture makes Nginx particularly suitable for use as a reverse proxy, load balancer, and API (Application Programming Interface) gateway.

[0034] OpenResty, built upon Nginx, deeply integrates the LuaJIT just-in-time compiler and a large number of high-quality Lua libraries. LuaJIT is a high-performance implementation of the Lua language, employing just-in-time compilation technology to compile Lua code into machine code for execution, achieving performance close to native C code. Through the ngx_lua module, OpenResty allows developers to embed Lua scripts at various stages of Nginx processing, including request rewriting, access control, content generation, and response filtering.

[0035] OpenResty provides a rich set of APIs, enabling Lua scripts to directly access Nginx's core functionalities, such as HTTP request parsing, response construction, upstream server communication, shared memory access, and timer management. These APIs are designed for non-blocking asynchronous operation, fully leveraging Nginx's event-driven model to ensure performance under high concurrency scenarios.

[0036] In practical applications, OpenResty can be used to build high-performance API gateways, Web Application Firewalls (WAFs), dynamic content caching, real-time data processing, and other systems. OpenResty's advantages lie not only in its superior performance but also in its flexibility and scalability. Developers can quickly implement various complex business logics without being limited by the fixed functional modules of traditional web servers.

[0037] For the embodiments of this invention, several key features of OpenResty make it an ideal technology choice. First, its high-performance streaming data processing capabilities enable real-time processing of SSE data streams returned by LLM without becoming a system bottleneck. Second, the flexibility of Lua scripts allows for the implementation of complex text fragmentation logic and TTS scheduling algorithms at the gateway layer. Third, OpenResty's non-blocking asynchronous I / O model naturally supports processing a large number of concurrent TTS requests, fully utilizing system resources. Finally, OpenResty's rich ecosystem and mature commercial application cases ensure the reliability and maintainability of the solution.

[0038] The shortcomings of existing LLM voice response solutions stem from the serial processing paradigm, leading to multi-dimensional problems. These are analyzed from the perspectives of performance, resources, user experience, and scalability.

[0039] Traditional text-to-speech processing workflows employ a batch processing model of "generation-conversion": User request → LLM generates complete text (waiting time T1) → TTS processes the entire text (waiting time T2) → User hears the speech for the first time, total delay = T1 + T2.

[0040] Taking the generation of a 3000-word text as an example, assuming the LLM generation speed is 30 words / second and the TTS processing speed is 100 words / second, then T1 = 3000 / 30 seconds = 100 seconds, T2 = 3000 / 100 seconds = 30 seconds; total latency = 130 seconds, resource utilization: 50% (LLM and TTS work alternately). Figure 1 As shown, users have to wait more than 2 minutes to hear the first word, which is unacceptable in real-time voice interaction scenarios, such as virtual humans or voice broadcasting systems.

[0041] The main shortcomings and problems of the existing solutions mentioned above are analyzed as follows: Low resource utilization: In batch processing mode, LLM generation and TTS conversion are two strictly sequential stages. During the LLM generation stage, the TTS service is idle; during the TTS conversion stage, the LLM is in a space state. This sequential processing method leads to underutilization of system resources, limited overall throughput, and a surge in average response time.

[0042] High peak memory usage: The service client needs to fully cache all text generated by the LLM (e.g., 3000 characters) and then submit it to the TTS service all at once. For long text scenarios, this can lead to excessive memory usage, increasing the system's memory pressure. Under high concurrency, this may trigger memory overflow or frequent garbage collection, affecting system stability.

[0043] Poor user experience: From a user's perspective, prolonged silent waiting can lead to anxiety and uncertainty. Even after the text has been generated, users can only see the text and still need to wait for TTS processing to complete before receiving audio feedback. This "text-only" delivery model fails to provide a progressive experience and contradicts the user's pursuit of instant response and a smooth experience.

[0044] Error recovery is difficult: In batch processing mode, if an error occurs during TTS processing (such as network interruption, generation interruption, or service timeout), all generated text needs to be regenerated and processed. This not only wastes previous computing resources but also further prolongs user waiting time.

[0045] Increased business complexity and maintenance costs: Adding common processing logic to multiple business service backends, such as load balancing, automatic retries, failover, rate limiting, identity authentication, security protection, semantic caching, token usage, quota budgets, and expenditure auditing, will increase business complexity and the probability of errors.

[0046] The improved scheme of this invention is designed as follows: like Figure 2 As shown, the streaming processing scheme proposed in this embodiment of the invention introduces an OpenResty LLM gateway as an intelligent proxy layer into the traditional architecture to achieve real-time parsing, splitting, and parallel processing of SSE data streams. The OpenResty LLM gateway can connect to one or more LLMs, such as cloud-based ChatGPT / Aws, cloud-based DeekSeek, and local On-Premises.

[0047] like Figure 3 As shown, the core idea of ​​the embodiment of the present invention is to transform "batch processing" into "stream processing". By parsing and splitting the SSE data stream returned by LLM in real time at the OpenResty gateway layer, the TTS service is called concurrently in units of sentences or semantic segments, thereby realizing the pipeline parallelism of text generation and speech synthesis.

[0048] like Figure 4 As shown, through pipelined parallel processing, the total time is 103 seconds (approximately the LLM generation time); the first word latency is 3 seconds (reduced by 97%); and the resource utilization rate is 95% (LLM and TTS work simultaneously).

[0049] The technical advantages of the streaming processing scheme proposed in this invention include: 1. Parallel processing: When LLM generates the Nth sentence, TTS can process N-1 sentences simultaneously.

[0050] 2. Low latency: Users can hear the speech immediately after the LLM generates the first complete sentence.

[0051] 3. Streaming: Audio data is pushed to the client in real time, without waiting for the full text processing to complete.

[0052] 4. Resource optimization: The gateway layer only caches sentence-level text fragments, adaptive text segmentation, and low memory resource consumption.

[0053] 5. Fault tolerance: The failure of a single segment in TTS does not affect other segments, enabling fine-grained retries.

[0054] The objectives of the embodiments of the present invention include: Reduce first-word pronunciation delay: Reduce the first-word pronunciation time (TTFA) from minutes to seconds. Taking a 3000-word text as an example, the traditional solution has a TTFA of 100 seconds (waiting for the full text to be generated), while the improved solution can complete the generation and conversion of the first sentence within 3-5 seconds, reducing TTFA by more than 95%.

[0055] Enhancing User Experience: By streaming voice output, users receive immediate feedback, creating a continuous "listen-and-generate" experience. This progressive delivery model aligns with natural human communication habits, significantly reducing waiting anxiety and increasing user satisfaction.

[0056] Optimized resource utilization: Through parallel processing, LLM and TTS services can work simultaneously, improving the overall system throughput. In high-concurrency scenarios, computing resources can be utilized more fully, reducing the processing cost per request.

[0057] Reduced memory usage: The gateway layer uses streaming processing, eliminating the need to cache the entire text. This reduces memory usage from O(n) (where n is the total length of the text) to O(m) (where m is the length of a single sentence, and typically m << (i.e., much smaller than) n). This is particularly important for handling long texts and high-concurrency scenarios.

[0058] Enhanced system resilience: Streaming processing supports more granular error handling and retry mechanisms. Failures or timeouts in a single sentence's TTS can be retried independently without affecting other completed segments. It also supports resume capability and error degradation, improving system robustness.

[0059] Reduce business complexity and maintenance costs: Common services can be decentralized at the gateway layer to reduce coupling and provide unified load balancing, automatic retries, failover, rate limiting, authentication, security protection, data anonymization, semantic routing, and semantic caching. It also has unified observability and cost control, such as token usage, quota budgets, and expenditure auditing.

[0060] To facilitate understanding of this embodiment, a detailed description of the real-time AI large-scale model streaming text-to-speech processing method disclosed in this embodiment of the invention is provided below.

[0061] This invention provides a real-time method for streaming text-to-speech processing of large AI models. This method is applied to gateway devices deployed with the OpenResty platform and can improve the user experience of AI interviewer products. See also... Figure 5 The diagram shows a real-time streaming text-to-speech processing method for large AI models. This method mainly includes the following steps S510 to S550: Step S510: Receive a problem request from the client and forward the problem request to the Large Language Model Service. The Large Language Model Service is used to return the generated text content by sending an event SSE data stream protocol through the server.

[0062] The gateway layer (proxy service) first receives requests from clients, carrying the user's question. It then forwards the request to the backend large language model service, either as is or after necessary processing. Once the large language model service begins processing the request, it doesn't wait to generate the entire text. Instead, it continuously pushes back the generated text fragments (e.g., word-by-word or sentence-by-sentence) in the form of a data stream immediately after generating one or more tokens via the SSE data stream protocol. This mechanism significantly reduces perceived latency for the user.

[0063] Step S520: Intercept and parse the SSE data stream returned by the large language model service in real time to extract streaming text data.

[0064] When the large language model service begins returning text via SSE data streams (typically in the format data: [text content]\n\n), the gateway layer does not passively wait for the entire response to complete. Instead, it actively intercepts these raw data streams in transit, chunk by chunk. The gateway layer then parses each chunk of data in real time: first identifying the SSE protocol format, removing protocol headers and footers such as data:, and then extracting the plain text content. These extracted text fragments can be temporarily stored in a buffer, preparing for the next step of intelligent segmentation and processing. This achieves real-time conversion and supply of text from SSE protocol format to plain text data, laying a crucial foundation for subsequent parallel speech synthesis.

[0065] In some possible embodiments, step S520 above may include: intercepting the SSE data stream returned by the large language model service in real time through the body_filter hook; parsing the SSE data stream using the SSE data stream protocol; and storing the parsed streaming text data in the session data buffer.

[0066] Step S530: The streaming text data is segmented to obtain at least one text fragment.

[0067] The extracted streaming text data is segmented to intelligently divide the continuous text stream into semantically coherent and appropriately sized text segments (such as complete sentences or phrases). In this embodiment, this process is not a simple cut to a fixed length, but rather based on an intelligent segmentation algorithm to identify the optimal segmentation points. For example, segmentation is first prioritized at punctuation marks such as periods and question marks that indicate the end of sentences to maximize semantic integrity. If the accumulated text length exceeds a preset threshold but no sentence end character is encountered, segmentation is performed at secondary punctuation marks such as commas and semicolons to balance latency and coherence. As a final safeguard, forced segmentation is performed when the text length is too long to prevent infinite waiting. Ultimately, through this segmentation process, the continuous text stream is transformed into a correctly ordered sequence of text segments suitable for independent transmission to a TTS service for speech synthesis, thereby achieving parallel pipelines for text generation and speech synthesis.

[0068] In some possible embodiments, step S530 above may include: splitting streaming text data into one or more text segments using an intelligent segmentation algorithm employing a multi-level segmentation strategy; wherein the multi-level segmentation strategy includes a first priority strategy, a second priority strategy, and a third priority strategy; the first priority strategy is to segment when a sentence end character is detected; the second priority strategy is to segment at secondary punctuation marks when the accumulated text length exceeds a preset first threshold and no sentence end character is detected; the third priority strategy is to perform forced segmentation when the accumulated text length exceeds a preset second threshold; the second threshold is greater than the first threshold.

[0069] Considering that very short text fragments, such as sentences with only a few words, would lead to excessive API calls and network overhead if submitted separately to TTS processing, thus reducing efficiency, in this embodiment, the aforementioned intelligent fragmentation algorithm is further configured as follows: for the initial text fragment obtained through the first priority strategy, it is determined whether the length of the initial text fragment is less than a preset minimum length threshold; if it is less than the minimum length threshold, the initial text fragment is buffered and merged with subsequent text data, and the merged text is fragmented while ignoring the sentence termination character at the merging point, until the length of the fragmented text reaches the minimum length threshold. The minimum length threshold is less than the aforementioned first threshold.

[0070] The first threshold, second threshold, and minimum length threshold mentioned above can all be set according to actual needs, and are not limited here. To ensure performance, this embodiment also provides an adaptive optimization strategy for the intelligent segmentation algorithm: by monitoring the average response time and current load of the text-to-speech service, the segmentation threshold in the intelligent segmentation algorithm is dynamically adjusted. The segmentation threshold includes one or more of the first threshold, second threshold, and minimum length threshold.

[0071] This embodiment establishes a dynamic feedback loop, enabling the text segmentation strategy to intelligently adapt to the real-time processing capabilities of the backend TTS service. Specifically, it continuously monitors the health metrics of the TTS service (such as average response time, error rate, and current concurrent requests) and uses these metrics as input parameters to dynamically adjust the segmentation threshold in the intelligent segmentation algorithm. When the TTS service response slows down or becomes overloaded, the segmentation threshold is automatically increased, tending to generate larger text segments, thereby reducing the frequency of TTS service calls and avoiding further pressure on the already heavily loaded TTS service. Conversely, when the TTS service responds quickly and the load is light, the segmentation threshold is decreased, generating smaller text segments to pursue lower end-to-end latency and more immediate voice feedback. In this way, a dynamic optimal balance can be achieved between the two goals of "low latency" and "high throughput / stability".

[0072] In one possible implementation, after each TTS service call, regardless of success or failure, the call's duration and status (success / failure) are recorded. This data is stored in OpenResty's shared memory dictionary to avoid the overhead of synchronizing data between worker processes. An adaptive threshold manager (a scheduled task) reads TTS call metrics from the shared memory for the most recent preset time period (e.g., the past 30 seconds) and calculates the Average Response Time (ART) and error rate; then, the sharding threshold is adjusted according to preset rules. These rules can be simple if-else logic or more complex formulas. For example: Normal state (ART < 500ms): Fragmentation thresholds are set as follows: first threshold = 50 words, second threshold = 150 words, minimum length threshold = 10 words; aiming for low latency. Load warning (500ms≤ART<1000ms): Fragmentation threshold adjusted to first threshold=100 words, second threshold=250 words, minimum length threshold=15 words; start increasing fragmentation to reduce the number of calls; High load (ART ≥ 1000ms or error rate > 5%): adjust the fragmentation threshold to first threshold = 200 words, second threshold = 400 words, minimum length threshold = 20 words; prioritize system stability and significantly reduce call frequency; Service recovery: When the indicators fall back, the sharding threshold also gradually returns to normal levels to avoid lag effects.

[0073] Before performing sharding, the intelligent sharding algorithm first reads the current dynamic threshold calculated by the adaptive threshold manager from shared memory, instead of using a fixed configuration value.

[0074] Step S540: For each text segment, call the text-to-speech service in parallel to generate an audio segment.

[0075] In this embodiment, for each text segment obtained through intelligent segmentation, an independent asynchronous task is immediately initiated to call the text-to-speech service. This mechanism is the core of achieving parallelization of text generation and speech synthesis. Specifically, when a complete text segment (such as a sentence) is ready, the gateway layer does not wait for the previous TTS call to complete, but instead initiates a new conversion request to the TTS service in a non-blocking, parallel manner. This means that while the large language model is still generating the next paragraph, the speech synthesis task of the previous paragraph has already begun processing simultaneously. This parallel "pipeline" working mode completely changes the serial waiting of "generating the full text and then converting" in traditional solutions, allowing speech segments to be continuously and with low latency pushed to the client, closely following the generation rhythm of text segments, thus achieving a smooth experience of "generating, converting, and playing simultaneously."

[0076] Step S550: Push the audio segment returned by the text-to-speech service to the client in real time.

[0077] In this embodiment, once the text-to-speech service completes processing a text segment and returns the corresponding binary audio segment (such as PCM (Pulse Code Modulation) or OPUS format), the gateway layer immediately pushes the audio segment to the client via a previously established WebSocket or similar bidirectional streaming connection. The client does not need to wait for the entire audio file to be generated; it can start playing it immediately upon receiving the first audio segment. This mechanism ensures that users can hear the speech output almost in real-time, synchronized with the text generation progress of the large language model, ultimately achieving a seamless interactive experience of "thinking, writing, and reading simultaneously," completely eliminating the long silent waiting period in traditional solutions.

[0078] Optionally, the aforementioned AI large-scale model-based real-time text-to-speech processing method further includes: during the segmentation of streaming text data, performing at least one of the following processes on the streaming text data: data anonymization, compliance checking, and semantic caching. Integrating data anonymization and compliance checking during the segmentation process proactively filters out sensitive information (such as phone numbers) or illegal content at the source of data processing, ensuring that the output text complies with laws and regulations. Simultaneously, semantic caching can directly return the voice results of previously processed requests for the same purpose, greatly improving efficiency and reducing costs.

[0079] Further optionally, the above-mentioned AI large-scale model streaming text-to-speech real-time processing method further includes: performing fault tolerance processing during the process of calling the text-to-speech service, the fault tolerance processing including at least one of the following: implementing an exponential backoff retry mechanism for the failure of the text-to-speech service call for a single text segment; setting a timeout for the text-to-speech service call and performing degradation processing after the timeout; and pushing preset background audio to the client to maintain playback continuity when the text-to-speech service is unavailable or the call fails.

[0080] The fault tolerance implemented in the TTS call process can ensure the resilience of the service: it intelligently responds to temporary failures through exponential backoff retries, uses timeout and degradation mechanisms to prevent the failure of a single segment from blocking the entire pipeline, and masks technical failures by inserting preset background audio when the service is unavailable, thus maintaining a seamless and continuous auditory experience for users, thereby ensuring high availability while pursuing low latency.

[0081] The AI ​​large-scale model streaming text-to-speech real-time processing method provided in this invention can receive question requests from clients and forward them to a large language model service. The large language model service returns generated text content via the SSE data stream protocol. The method intercepts and parses the SSE data stream returned by the large language model service in real time to extract streaming text data. It then segments the streaming text data to obtain at least one text fragment. For each text fragment, it calls the text-to-speech service in parallel to generate an audio fragment. Finally, it pushes the audio fragment returned by the text-to-speech service to the client in real time. This method organically combines OpenResty gateway technology, SSE streaming communication, text segmentation, and parallel text-to-speech processing. By performing real-time parsing and splitting of the SSE data stream returned by the large language model at the gateway layer and by calling the text-to-speech service in parallel, it achieves a pipelined parallel processing mode for text generation and speech synthesis, alleviating the problems of high latency, low resource utilization, and poor user experience in long text scenarios.

[0082] To facilitate understanding, the above-mentioned AI large-scale model streaming text-to-speech real-time processing method will be described in detail below.

[0083] This invention addresses the problems of high latency, low resource utilization, and poor user experience in traditional text-to-speech processing by proposing an innovative streaming real-time conversion solution based on the OpenResty gateway. This solution achieves a pipelined parallel processing mode for text generation and speech synthesis by performing real-time parsing, splitting, and intelligent processing of the SSE data stream returned by the large language model at the gateway layer, significantly reducing the first-word pronunciation time and overall response latency. Experimental results show that this solution can reduce user-perceived latency by more than 70% in long text processing scenarios, reducing the first-word pronunciation latency from minutes to seconds, providing a better user experience for voice interaction applications. This invention has significant theoretical and practical value in promoting the application of artificial intelligence technology in real-time interactive scenarios.

[0084] The innovation of this invention lies in its organic integration of multiple technologies, including OpenResty gateway technology, SSE streaming communication, intelligent text segmentation algorithm, and parallel TTS processing, to construct a complete end-to-end solution. This cross-layer, multi-technology integrated system innovation provides new ideas and methods for solving latency issues in LLM and TTS combination scenarios.

[0085] A complete streaming process includes the following stages: 1. The client initiates a request; 2. The OpenResty gateway receives and forwards LLM (SSE streaming return) requests; 3. The OpenResty gateway receives SSE data streams in real time and performs fragmentation processing; 4. Parallel call to the TTS service to transform each segment; 5. Audio streams are pushed to the client for playback in real time.

[0086] See Figure 6The diagram illustrates the interactive flow of a real-time streaming text-to-speech processing method for large AI models. The client sends a request to the LLM gateway; the LLM gateway forwards the request to the LLM service; the LLM service returns the generated event stream to the LLM gateway, such as returning {data: sentence fragment 1, type: message} and {data: sentence fragment 2, type: message} sequentially; the LLM gateway pushes the received event streams ({data: sentence fragment 1, type: message}, {data: sentence fragment 2, type: message}, etc.) to the client sequentially, allowing the client to display the text sentence; simultaneously, the LLM gateway parses, filters, and buffers the event stream, segmenting it into complete and valid sentences, and then requests TTS speech synthesis {text: complete sentence} from the TTS service; the TTS service returns an audio data stream to the LLM gateway, which then pushes the complete sentence's audio data stream to the client via WebSocket, allowing the client to decode and play it.

[0087] The detailed implementation steps are as follows: Step 1: The client initiates a request.

[0088] The client sends an HTTP request to the OpenResty gateway, carrying the user's question, such as: "Please write an article about artificial intelligence in 1000 words," and requires the applicant to summarize the article in no more than 100 words, quickly extracting the core information.

[0089] The request header can be: POST https: / / gateway.example.com / stream Accept: text / event-stream Step 2: The OpenResty gateway forwards the request to the LLM.

[0090] The OpenResty gateway's service call client connects to an LLM (such as GPT-4, DeepSeek, or a locally deployed LLM) and enables streaming output. It also allows setting request parameters such as temperature, maximum token count, Top K, and Top P. The temperature parameter adjusts the randomness or determinism of the generated text; the maximum token count determines the maximum length of the generated text; Top K sampling randomly selects one of the K words with the highest probability in the prediction distribution as the next word; and kernel sampling (Top P sampling) selects a minimum set of words whose cumulative probability reaches a specified P value.

[0091] Step 3: The OpenResty gateway receives and processes the SSE data stream from the LLM in real time and performs fragmentation.

[0092] The returned header can be: Content-Type: text / event-stream Transfer-Encoding: chunked Cache-Control: no-cache Connection: keep-alive The LLM service starts generating text. Every time a certain number of tokens are generated, a data stream is sent via SSE. After the gateway receives the response data stream, it intercepts and parses the data in real time, extracts the data from the returned data stream (if the message is in a custom format, it needs to be parsed), and uses an intelligent text fragmentation strategy to detect and buffer complete sentences. It can also filter and anonymize key data and perform compliance checks, etc.

[0093] LLM begins streaming SSE data (including event message type, data, and sequence ID). The structure and format of the data can be defined; for demonstration purposes, text content is used here, as shown in the example below: event: Session, data: "{ Guid} "\n\n event: Delta, data: Artificial Intelligence event: Delta, data: as Event: Delta; Data: 21st Century event: Delta, data: most event: Delta, data: revolutionary event: Delta, data: technology event: Delta, data: one of them event: Delta, data: , currently... event: Delta, data: profound event: Delta, data: change event: Delta, data: ... event: Finish, data: "none" The OpenResty gateway layer request execution order is as follows: Figure 7 As shown, the response data stream is mainly intercepted in real time through the body_filter hook.

[0094] The key processing logic is in the `body_filter_by_lua` stage. This is a special processing stage where the script is called multiple times when the upstream service (LLM service) returns the response body, processing a piece of data each time. Leveraging this feature, true streaming processing can be achieved, meaning data is processed immediately upon arrival without waiting for a complete response.

[0095] In body_filter, the current data block is first obtained (via ngx.arg[1]), and then the SSE protocol is parsed. If the data block content type is Gzip, it needs to be decompressed. The SSE data format is relatively simple. Each message starts with "data:" and ends with two newline characters. If there is a specific format, it needs to be parsed to extract the required data. In addition, a session data buffer needs to be maintained because a data block may contain incomplete SSE messages, which need to be accumulated to the next data block to form a complete message. Then the text content is parsed and appended to the buffer. At the same time, the sentence boundary detection algorithm is run to scan the buffer and find sentence end markers, such as Chinese period, question mark, exclamation mark, English period, question mark, exclamation mark, etc. In order to avoid misjudgment, the intelligent fragmentation algorithm also needs to consider some special cases, such as ellipsis, punctuation in quotation marks, decimal point in numbers, etc.

[0096] When a complete sentence is detected, the intelligent segmentation algorithm evaluates its length. If the sentence length is within a reasonable range (e.g., 10-300 characters), it is treated as a TTS processing unit. If the sentence is too short (e.g., less than 10 characters), it may wait to accumulate more content or merge with the next sentence to improve TTS efficiency. If the sentence is too long (e.g., more than 300 characters), it may need to be segmented at secondary boundaries (e.g., commas, semicolons) to avoid excessive processing time for a single TTS task. Through this segmentation strategy, the algorithm can autonomously and finely control sentence breaks and pauses.

[0097] To balance latency and audio coherence, and to avoid buffering excessively large text data, the intelligent segmentation algorithm employs the following three-layer segmentation strategy: Priority 1, sentence boundaries: Segmentation is performed at sentence-ending marks such as periods, exclamation marks, and question marks (e.g., "。", "!", "?", ";", "\n"). This is the ideal segmentation point, ensuring the best semantic integrity and audio naturalness.

[0098] Priority 2, secondary boundaries: When the accumulated text exceeds the threshold but has not encountered a sentence end character, it is segmented at secondary punctuation marks such as commas, semicolons, and colons. These punctuation marks indicate pauses within a sentence, which, while not as ideal as sentence boundaries, can still maintain relative semantic coherence.

[0099] Priority 3, Forced Segmentation: When the text length exceeds the forced threshold (i.e., the second threshold), it must be segmented even if there are no suitable punctuation marks to avoid indefinite waiting.

[0100] The algorithm strategy also considers the minimum length limit of text segments. Submitting excessively short text fragments, such as sentences of only a few words, to TTS processing individually would lead to excessive API calls and network overhead, ultimately reducing efficiency. Therefore, when an excessively short sentence is identified, the algorithm can choose to temporarily hold it in a buffer, waiting for the next sentence. If the combined length is suitable, it is submitted together. This strategy is very useful when processing conversational text, as conversations often contain short responses such as "yes" or "okay."

[0101] The algorithm can also perform adaptive optimization. By monitoring the average response time and current load of the TTS service, it dynamically adjusts the sharding threshold. When the TTS service responds quickly and the load is low, smaller shards can be used to pursue lower latency. When the TTS service responds slowly and the load is high, larger shards can be used to reduce the number of calls and avoid increasing the load. This adaptive mechanism allows the system to maintain good performance under different conditions.

[0102] The following is an example of text segmentation: Raw stream: Artificial intelligence, as one of the most revolutionary technologies of the 21st century, is profoundly changing all aspects of human society. From medical diagnosis to autonomous driving, from intelligent assistants to content creation, the application scenarios of AI technology are becoming increasingly widespread.

[0103] Fragmentation results: Excerpt 1: "Artificial intelligence, as one of the most revolutionary technologies of the 21st century, is profoundly changing all aspects of human society." Excerpt 2: "From medical diagnosis to autonomous driving, from smart assistants to content creation, the application scenarios of AI technology are becoming increasingly widespread." Step 4: Invoke the TTS speech synthesis service in parallel.

[0104] For each complete segment obtained, the TTS speech synthesis service is immediately invoked in parallel to obtain the speech data stream result. Parameters such as timbre, speech rate, volume, bit rate, voice cloning, output format, and the model used are attached during the request process.

[0105] The request header can be as follows: GET wss: / / yun.example.com / tts Connection: upgrade Upgrade: websocket After that, it will first return a 101 status code, indicating that the connection has been upgraded to the WebSocket protocol, and then send the complete sentence data.

[0106] The following fault tolerance and degradation strategies can also be adopted during the call: Single-segment retry: When TTS fails, only the current sentence segment is retried, without affecting other sentence segments; Timeout control: Set the timeout period for TTS requests to avoid prolonged blocking; Overload control: When the TTS service is overloaded, calls are suspended and degraded processing is performed; Background fill: Insert short background music when TTS fails or becomes unavailable to maintain playback continuity.

[0107] TTS speech synthesis services can be deployed to edge nodes to further reduce latency.

[0108] Specifically, the entire process handles various exceptions. If the LLM service times out or returns an error, appropriate error information needs to be returned to the client. If the TTS service call fails, a decision on whether to retry needs to be made based on the reason for the failure. If a TTS task fails after multiple retries or the TTS service is overloaded, a degradation strategy is adopted, such as inserting a background audio clip or skipping the clip, to avoid affecting the overall process.

[0109] In addition, an error handling retry mechanism is designed. For network errors or temporary failures, exponential backoff retries are implemented. That is, after the first failure, a retry is performed immediately; after the second failure, a retry is performed after a wait of, for example, 100 milliseconds; and after the third failure, a retry is performed after a wait of, for example, 300 milliseconds (this can be configured). The number of retries can be limited to 3 to avoid excessive retries leading to latency accumulation. Detailed logs are recorded for each retry to facilitate subsequent problem analysis.

[0110] See Figure 8The diagram illustrates a fault-tolerant processing flow for a text-to-speech (TTS) service call. After initiating a TTS request, the LLM data stream is first submitted to the TTS service, and then a response is awaited to determine if the call was successful. If the call is successful, the audio stream data is obtained, pushed to the audio queue, and then pushed to the client for audio playback. If the call fails, error type identification is performed; for network errors and call timeouts, a retry is initiated; if the retry succeeds, the audio stream data is obtained, pushed to the audio queue, and then pushed to the client for audio playback; if the retry fails more than three times, a degradation process is implemented, inserting a background music stream, logging the event, and notifying monitoring, before pushing the stream to the audio queue and then to the client for audio playback.

[0111] Step 5: The audio stream is pushed to the client and played.

[0112] Use WebSocket to push audio data streams to the client in real time and play the audio using a player that supports streaming audio.

[0113] The overall timeline sequence is shown in Table 1 below.

[0114] Table 1

[0115] In Table 1 above, — indicates no data or no operation.

[0116] In summary, the key points of the embodiments of the present invention include: OpenResty Gateway intercepts large model SSE data streams: When a large model returns streaming data, OpenResty Gateway intercepts and parses each data stream segment in real time.

[0117] Real-time triggered streaming TTS speech synthesis: Each received text segment is buffered and segmented, and the corresponding speech segment is generated by calling local or cloud-based TTS.

[0118] Streaming speech synthesis and splicing: Each speech segment is pushed to the client immediately after it is generated, realizing "generating while listening".

[0119] Dual-stream parallel transmission: Text stream (SSE) and voice stream (WebSocket) are transmitted simultaneously, reducing latency and improving user experience.

[0120] Multi-level segmentation strategy: semantic integrity is prioritized, segmenting at complete sentences; delay control, segmenting at secondary boundaries (commas, semicolons) when accumulated text exceeds a threshold; forced truncation, avoiding infinite waiting in extremely long texts; and dynamic adaptive segmentation threshold adjustment.

[0121] Reduce complexity and coupling: Unify various general capabilities and push them down to the gateway layer to reduce business complexity and coupling, and facilitate unified management and observation.

[0122] This invention addresses the high latency and poor user experience issues inherent in traditional text-to-speech (TTS) processing by proposing an innovative real-time streaming processing solution based on the OpenResty gateway. By implementing technologies such as real-time parsing and splitting of SSE data streams, intelligent text segmentation, concurrent TTS scheduling, and error degradation at the gateway layer, it achieves pipelined parallel processing of LLM text generation and TTS speech synthesis, significantly reducing first-word pronunciation latency and improving system performance and user experience.

[0123] Experimental results show that the improved scheme achieves significant improvements in key indicators such as first-word pronunciation delay, total processing time, resource utilization, and system throughput. User experience evaluation and practical application data further verify the effectiveness and practical value of the scheme. With the continuous development of large language model technology and the continuous expansion of application scenarios, the demand for low-latency, high-quality voice interaction will continue to grow. The streaming processing concept and technical solution proposed in this invention provides an important reference for building the next generation of intelligent voice interaction systems.

[0124] As described above, these technical features together constitute the uniqueness of this invention, provide a solid foundation for intellectual property protection, and have high technical barriers and practical application value.

[0125] Corresponding to the above-described real-time text-to-speech processing method for large AI models, this embodiment of the invention also provides a real-time text-to-speech processing device for large AI models, which is applied to a gateway device deployed with the OpenResty platform. See also Figure 9 The diagram shown illustrates the structure of a real-time AI large-scale model streaming text-to-speech processing device. This device includes: Forwarding module 901 is used to receive problem requests from clients and forward the problem requests to the large language model service. The large language model service is used to return the generated text content by sending an event SSE data stream protocol through the server. Parsing module 902 is used to intercept and parse the SSE data stream returned by the large language model service in real time to extract streaming text data; The fragmentation module 903 is used to fragment the streaming text data to obtain at least one text segment. Module 904 is invoked to call the text-to-speech service in parallel for each text segment to generate an audio segment; The push module 905 is used to push audio segments returned by the text-to-speech service to the client in real time.

[0126] The AI ​​large-scale model streaming text-to-speech real-time processing device provided in this invention can receive question requests from clients and forward them to a large language model service. The large language model service returns generated text content via the SSE data stream protocol. The device intercepts and parses the SSE data stream returned by the large language model service in real time to extract streaming text data. It then segments the streaming text data to obtain at least one text fragment. For each text fragment, it calls the text-to-speech service in parallel to generate an audio fragment. Finally, it pushes the audio fragment returned by the text-to-speech service to the client in real time. This organically combines OpenResty gateway technology, SSE streaming communication, text segmentation, and parallel text-to-speech processing. By performing real-time parsing and splitting of the SSE data stream returned by the large language model and parallel calling of the text-to-speech service at the gateway layer, it achieves a pipelined parallel processing mode for text generation and speech synthesis, alleviating the problems of high latency, low resource utilization, and poor user experience in long text scenarios.

[0127] Furthermore, the parsing module 902 is specifically used to: intercept the SSE data stream returned by the large language model service in real time through the body_filter hook; parse the SSE data stream according to the SSE data stream protocol; and store the parsed streaming text data in the session data buffer.

[0128] Furthermore, the aforementioned segmentation module 903 is specifically used to: split streaming text data into one or more text segments using an intelligent segmentation algorithm employing a multi-level segmentation strategy; wherein, the multi-level segmentation strategy includes a first priority strategy, a second priority strategy, and a third priority strategy; the first priority strategy is to segment when a sentence end character is detected; the second priority strategy is to segment at secondary punctuation marks when the accumulated text length exceeds a preset first threshold and no sentence end character is detected; the third priority strategy is to force segmentation when the accumulated text length exceeds a preset second threshold; the second threshold is greater than the first threshold.

[0129] Furthermore, the aforementioned intelligent text segmentation algorithm is configured as follows: for the initial text segment obtained through the first priority strategy, determine whether the length of the initial text segment is less than a preset minimum length threshold; if it is less than the minimum length threshold, buffer and merge the initial text segment with the subsequent text data, and segment the merged text without ignoring the sentence termination character at the merging point, until the length of the segmented text reaches the minimum length threshold.

[0130] Furthermore, the aforementioned AI large-scale model streaming text-to-speech real-time processing device also includes: The optimization module is used to dynamically adjust the sharding threshold in the intelligent sharding algorithm by monitoring the average response time and current load of the text-to-speech service. The sharding threshold includes one or more of a first threshold, a second threshold, and a minimum length threshold.

[0131] Furthermore, the aforementioned sharding module 903 is also used to: perform at least one of the following processes on the streaming text data during the sharding process: data desensitization, compliance check, and semantic caching; The aforementioned calling module 904 is also used to: perform fault tolerance processing during the process of calling the text-to-speech service, the fault tolerance processing including at least one of the following: implementing an exponential backoff retry mechanism for the failure of the text-to-speech service call for a single text segment; setting a timeout for the text-to-speech service call and performing degradation processing after the timeout; and pushing preset background audio to the client to maintain playback continuity when the text-to-speech service is unavailable or the call fails.

[0132] The AI ​​large-scale model streaming text-to-speech real-time processing device provided in this embodiment has the same implementation principle and technical effects as the aforementioned AI large-scale model streaming text-to-speech real-time processing method embodiment. For the sake of brevity, any parts not mentioned in the AI ​​large-scale model streaming text-to-speech real-time processing device embodiment can be referred to the corresponding content in the aforementioned AI large-scale model streaming text-to-speech real-time processing method embodiment.

[0133] like Figure 10 As shown in the figure, an embodiment of the present invention provides a gateway device, including: a processor 1001, a memory 1002 and a bus. The memory 1002 stores a computer program that can run on the processor 1001. The gateway device is deployed with the OpenResty platform. When the gateway device is running, the processor 1001 and the memory 1002 communicate through the bus. The processor 1001 executes the computer program to realize the above-mentioned AI large model streaming text-to-speech real-time processing method.

[0134] Specifically, the memory 1002 and processor 1001 mentioned above can be general-purpose memory and processor, without any specific limitations here.

[0135] This invention also provides a real-time AI large-scale model streaming text-to-speech processing system, such as... Figure 11 As shown, the AI ​​large model streaming text-to-speech real-time processing system includes the aforementioned gateway device 1101, as well as a large language model service 1102 and a text-to-speech service 1103, which are respectively connected to the gateway device 1101.

[0136] Specifically, gateway device 1101 receives requests from clients and forwards them to large language model service 1102; large language model service 1102 generates text content corresponding to the request in a streaming manner and returns it to gateway device 1101 via SSE data stream protocol; gateway device 1101 also intercepts and parses the SSE data stream returned by large language model service 1102 in real time to extract streaming text data; segments the streaming text data to obtain at least one text segment; for each text segment, calls text-to-speech service 1103 in parallel; text-to-speech service 1103 synthesizes the text segments into corresponding audio segments and returns them to gateway device 1101; gateway device 1101 also pushes the audio segments returned by text-to-speech service 1103 to the client in real time.

[0137] This invention also provides a computer-readable storage medium storing a computer program. When a processor runs this computer program, it executes the AI ​​large-model streaming text-to-speech real-time processing method described in the preceding method embodiments. The computer-readable storage medium includes various media capable of storing program code, such as USB flash drives, external hard drives, read-only memory (ROM), RAM, magnetic disks, or optical disks.

[0138] In this document, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. Furthermore, the term "at least one" in this document means any combination of at least two of any one or more elements. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0139] In all examples shown and described herein, any specific values ​​should be interpreted as merely exemplary and not as limitations; therefore, other examples of exemplary embodiments may have different values.

[0140] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of methods, systems, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0141] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of modules is only a logical functional division, and there may be other division methods in actual implementation. Furthermore, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the coupling or direct coupling or communication connection shown or discussed may be through some communication interface; the indirect coupling or communication connection between apparatuses or modules may be electrical, mechanical, or other forms.

[0142] The modules described as separate components may or may not be physically separate. Similarly, the components shown as modules may or may not be physical modules; they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0143] In addition, the functional modules in the various embodiments of the present invention can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module.

[0144] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A real-time method for streaming text-to-speech processing using a large AI model, characterized in that, The AI ​​large-model streaming text-to-speech real-time processing method, applied to gateway devices deployed with the OpenResty platform, includes: The system receives a question request from a client and forwards the question request to the Large Language Model Service, which is used to return the generated text content by sending an event SSE data stream protocol via the server. The SSE data stream returned by the large language model service is intercepted and parsed in real time to extract streaming text data; The streaming text data is segmented to obtain at least one text fragment; For each of the text segments, a text-to-speech service is invoked in parallel to generate an audio segment; The audio segments returned by the text-to-speech service are pushed to the client in real time.

2. The AI ​​large-scale model streaming text-to-speech real-time processing method according to claim 1, characterized in that, The real-time interception and parsing of the server-sent event SSE data stream returned by the large language model service to extract streaming text data includes: The SSE data stream returned by the large language model service is intercepted in real time via the body_filter hook; The SSE data stream is parsed using the SSE data stream protocol; The parsed streaming text data is stored in the session data buffer.

3. The AI ​​large-scale model streaming text-to-speech real-time processing method according to claim 1, characterized in that, The process of segmenting the streaming text data to obtain at least one text fragment includes: The streaming text data is split into one or more text segments by an intelligent segmentation algorithm that employs a multi-level segmentation strategy. The multi-level fragmentation strategy includes a first priority strategy, a second priority strategy, and a third priority strategy. The first priority strategy is to fragment when a sentence terminator is detected. The second priority strategy is to fragment at secondary punctuation marks when the cumulative text length exceeds a preset first threshold and no sentence terminator is detected. The third priority strategy is to force fragmentation when the cumulative text length exceeds a preset second threshold. The second threshold is greater than the first threshold.

4. The AI ​​large-scale model streaming text-to-speech real-time processing method according to claim 3, characterized in that, The intelligent sharding algorithm is also configured as follows: For the initial text segment obtained through the first priority strategy, it is determined whether the length of the initial text segment is less than a preset minimum length threshold; if it is less than the minimum length threshold, the initial text segment is buffered and merged with subsequent text data, and the merged text is segmented while ignoring the sentence termination character at the merging point, until the length of the segmented text reaches the minimum length threshold.

5. The AI ​​large-scale model streaming text-to-speech real-time processing method according to claim 4, characterized in that, The AI ​​large-model streaming text-to-speech real-time processing method also includes: By monitoring the average response time and current load of the text-to-speech service, the sharding threshold in the intelligent sharding algorithm is dynamically adjusted. The sharding threshold includes one or more of the first threshold, the second threshold, and the minimum length threshold.

6. The real-time AI large-scale model streaming text-to-speech processing method according to claim 1, characterized in that, The AI ​​large-model streaming text-to-speech real-time processing method also includes: During the process of segmenting the streaming text data, at least one of the following processes is performed on the streaming text data: data desensitization, compliance check, and semantic caching; During the invocation of the text-to-speech service, fault tolerance processing is performed, which includes at least one of the following: implementing an exponential backoff retry mechanism for failed invocations of the text-to-speech service for a single text segment; setting a timeout for invocations of the text-to-speech service and performing degradation processing after the timeout; and pushing preset background audio to the client to maintain playback continuity when the text-to-speech service is unavailable or the invocation fails.

7. A real-time AI large-scale model streaming text-to-speech processing device, characterized in that, The AI ​​large-model streaming text-to-speech real-time processing device, applicable to gateway devices deployed with the OpenResty platform, includes: The forwarding module is used to receive a question request from the client and forward the question request to the large language model service. The large language model service is used to return the generated text content by sending an event SSE data stream protocol through the server. The parsing module is used to intercept and parse the SSE data stream returned by the large language model service in real time to extract streaming text data; The segmentation module is used to segment the streaming text data to obtain at least one text fragment; The calling module is used to call the text-to-speech service in parallel for each of the text segments to generate audio segments; The push module is used to push the audio segments returned by the text-to-speech service to the client in real time.

8. A gateway device, comprising a memory and a processor, wherein the memory stores a computer program executable on the processor, characterized in that, The gateway device is deployed with the OpenResty platform, and when the processor executes the computer program, it implements the AI ​​large model streaming text-to-speech real-time processing method as described in any one of claims 1-6.

9. A real-time AI large-scale model streaming text-to-speech processing system, characterized in that, The system includes the gateway device as described in claim 8, and further includes a large language model service and a text-to-speech service respectively connected to the gateway device.

10. A computer-readable storage medium storing a computer program thereon, characterized in that, The computer program, when executed by the processor, performs the AI ​​large-scale model streaming text-to-speech real-time processing method as described in any one of claims 1-6.

Images