Construction method of real-time streaming voice intelligent question and answer service system

Abstract

The invention discloses a construction method of a real-time streaming voice intelligent question and answer service system. The construction method comprises the following steps: receiving input voice data; calling an independent voice recognition service to convert the voice data into an input text; the method comprises the following steps: constructing cue words of a large language model based on an input text, initiating an LLM streaming request to an independent large language model service, and obtaining streaming text answers generated by LLM in real time; segmenting the flow type text answer in real time through a dynamic sentence segmentation device to generate a plurality of clauses; calling an independent speech synthesis service in parallel for each clause, and converting the text into an audio data block; and combining the audio data blocks into streaming audio data according to the generation sequence, and returning the streaming audio data to a client for playing in real time. According to the invention, by decoupling ASR, LLM and TTS services and combining the asynchronous streaming framework and the strategy of "wanting while speaking", the real-time performance and flexibility of the voice question-answering system are significantly improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method for constructing a real-time streaming voice intelligent question-answering service system. Background Art

[0002] With the rapid development of artificial intelligence technology, voice question-answering systems have become one of the core applications of human-computer interaction and are widely used in smart homes, customer service, mobile terminals and other fields. Traditional voice question-answering systems are usually based on the following technologies: 1. Automatic Speech Recognition (ASR): converts user voice input into text data.

[0003] 2. Natural Language Processing (NLP): Understand and process the converted text data to generate corresponding answers. NLP technology includes Natural Language Understanding (NLU) and Natural Language Generation (NLG).

[0004] 3. Speech synthesis technology: Convert the generated text answer into speech output and return it to the user.

[0005] The typical process of traditional voice question answering system is as follows: Figure 1 As shown in the figure, the speech input is first converted into text by the ASR model, then the question-answering result is obtained by the NLU model, and finally the speech output is generated by the Text-to-Speech (TTS) model. This type of solution has high real-time performance due to its small model size and serialized process, but its disadvantage is that the generalization ability of the NLU model is limited and it is only applicable to a single scenario. Different scenarios require the training of different NLU models, so this type of solution is difficult to adapt to the complex and changing open-ended question-answering needs.

[0006] In order to achieve more intelligent and generalized voice Q&A, the industry uses large language models (LLMs) to replace NLU models to understand and answer questions. The use of LLMs can make the Q&A system more intelligent and personalized, and can achieve universal Q&A across scenarios, significantly improving the accuracy and diversity of answers. However, the autoregressive inference mechanism of LLMs results in a long time delay in obtaining complete answers, and the TTS conversion can only be started after the complete text is generated, resulting in a significant increase in the overall delay of audio output, which seriously affects the smoothness of the user experience.

[0007] Some solutions try to use multimodal large models, such as Figure 2As shown in the figure, the audio is directly used as the input of the large model to generate the answer audio in a streaming manner. However, the uniformity of the generated timbre, the strong coupling between the model and the audio codec module, and the high fine-tuning cost limit the practical application flexibility of this technology.

[0008] In addition, the existing technologies of ASR, NLU / LLM, and TTS services mostly adopt a tightly coupled architecture. Model replacement or upgrade requires reconstruction of the entire system, which makes it difficult to adapt to the differentiated requirements of different business scenarios for model performance and resource usage.

[0009] Therefore, how to achieve low-latency, high-concurrency streaming voice interaction while ensuring intelligent question-answering, and improve the modularity and scalability of the system architecture has become a technical problem that needs to be solved urgently in this field. Summary of the invention

[0010] The purpose of the present invention is to provide a method for constructing a real-time streaming voice intelligent question-answering service system to solve the problems raised in the above technical background.

[0011] To achieve the above object, the present invention adopts the following technical solutions: The present application provides a method for constructing a real-time streaming voice intelligent question-answering service system, comprising the following steps: Step S1: receiving voice data input by a user; Step S2: calling an independent speech recognition service (ASR Server) to convert the speech data into input text; Step S3: construct a prompt of a large language model (LLM) based on the input text, initiate an LLM streaming request to an independent large language model service (LLM Server), and obtain the streaming text answer generated by the LLM in real time; Step S4: segmenting the streaming text answer in real time by a dynamic sentence segmenter to generate multiple clauses; Step S5: Call an independent speech synthesis service (TTS Server) for each clause in parallel to convert the text into an audio data chunk; Step S6: combining the audio data blocks into streaming audio data in the order of generation, and returning the streaming audio data to the client for playback in real time; The ASR Server, LLM Server and TTS Server are independent services that are decoupled from each other, and the LLM Server and TTS Server support streaming output.

[0012] In a preferred embodiment, the working rules of the dynamic sentence segmenter include: Segment the streaming text generated by LLM according to punctuation or semantic boundaries; Limit the maximum length of the first clause. If it exceeds the threshold, it will be truncated. If the length of a clause is less than the minimum length threshold, it is merged with the subsequent clauses and output.

[0013] In a more preferred embodiment, the dynamic sentence segmenter performs the following operations after each LLM generated text iteration: Record the end index position of the current text traversal to avoid repeated processing of the segmented text; Merge clauses that do not meet the minimum length threshold into the subsequent text.

[0014] In a preferred embodiment, step S5 comprises: Create an independent TTS request thread for each clause, and control the number of concurrent requests through the thread pool; Each TTS request thread stores the generated audio data blocks in the asynchronous audio queue (audio_chunks-Queue) in sequence; The asynchronous audio queue is detected in real time, and the audio data blocks are extracted in the order in which they are queued and returned to the client.

[0015] In a more preferred embodiment, step S5 further includes a termination control step: When a termination request is received from the client, the termination event stop-event is triggered to terminate the current LLM streaming request, dynamic sentence segmentation and all TTS request threads.

[0016] In a more preferred embodiment, the TTS request thread execution process of each clause in step S5 includes: Create an independent asynchronous queue audio-chunk-j for clause sj, and add audio-chunk-j to the asynchronous audio queue (audio_chunks-Queue); Send a streaming conversion request for clause sj to the TTS Server, and store the returned audio data blocks in order into audio-chunk-j; When the audio stream conversion of clause sj is completed, insert the clause end identifier None into audio-chunk-j; Each time an audio data block is stored in audio-chunk-j, a stop-event is detected. If the stop-event is triggered, the current TTS request thread is terminated.

[0017] In a preferred embodiment, in step S3, the request process of the LLM Server includes: Build prompts containing Retrieval Enhanced Generation (RAG) technology based on business scenarios; Receive LLM-generated text snippets in real time through an async llm stream generator.

[0018] In a preferred embodiment, in step S6, the implementation of returning the streaming audio data includes: (a) During the text generation process of LLM, it is detected in real time whether there is an audio data block in the asynchronous audio queue. If so, it is returned in the order in which it was queued; (b) After the LLM generation is completed, the task end identifier end is inserted into the asynchronous audio queue, and the remaining audio data blocks in the asynchronous audio queue are continued to be processed and returned in the order in which they were put into the queue until the task end identifier end is detected or the asynchronous audio queue is cleared.

[0019] In a more preferred embodiment, the step (a) specifically comprises the following steps: Step a.1: Before each iteration of LLM streaming text generation, create an empty temporary storage variable audio_chunks to cache the audio data of the currently processed subqueue; Step a.2: After each iteration of LLM generates text, check whether the asynchronous audio queue is non-empty: If the asynchronous audio queue is not empty and audio_chunks is empty, dequeue the first subqueue audio-chunk-j from the asynchronous audio queue and assign it to audio_chunks; Step a.3: Determine whether audio_chunks is not empty and contains unreturned audio data chunks: If the conditions are met, all audio data blocks in audio_chunks are extracted in order and returned to the client in real time; If the extracted audio data block is empty, reset audio_chunks to empty and get the next subqueue from the asynchronous audio queue again; If audio_chunks is empty or there is no extractable data, continue to the next LLM iteration.

[0020] Furthermore, the step (b) specifically comprises the following steps: Step b.1: After LLM generation is completed, loop detection of stop-event and asynchronous audio queue status: If the termination event is triggered, the processing flow will be exited directly; If audio_chunks is empty, dequeue a new subqueue audio-chunk-j from the asynchronous audio queue and assign it to audio_chunks; If the asynchronous audio queue is empty, wait for the preset time and then re-detect; Step b.2: When audio_chunks is the end identifier end, the processing flow is terminated; Step b.3: Extract audio data blocks from audio_chunks in order and return them to the client, while looping: If the audio data block is empty, reset audio_chunks to empty and recheck the queue; If audio_chunks is empty, continue extracting after asynchronous waiting; The extraction operation is interrupted immediately after the termination event is detected.

[0021] In a preferred embodiment, the method is implemented by an asynchronous framework, comprising: Use FastAPI to build asynchronous service interfaces; ASR requests, LLM streaming generation, and TTS requests are designed as asynchronous non-blocking operations.

[0022] Compared with the prior art, the technical solution of the present invention has the following beneficial effects: The present invention discloses a method for constructing a real-time streaming voice intelligent question-answering service system. The method decouples automatic speech recognition (ASR), large language model (LLM) and text-to-speech (TTS) services into independent modular services, constructs an asynchronous streaming processing framework, and adopts a "think while speak" strategy to achieve parallel advancement of text generation and speech synthesis, thereby significantly improving the real-time performance and flexibility of the system: the text output by LLM streaming is segmented in real time by a dynamic sentence segmenter, clauses are generated based on punctuation marks, semantic boundaries and first sentence length restriction rules, and the clauses are submitted in parallel to the TTS service to generate an audio stream by using an asynchronous thread pool, thereby breaking the traditional serial processing mode.

[0023] The present invention also combines the hierarchical management of asynchronous audio queues (audio_chunks-Queue) with the design of the FastAPI asynchronous framework to realize non-blocking operations of ASR requests, LLM generation, and TTS conversion, thereby improving system throughput and resource utilization efficiency. At the same time, the retrieval enhanced generation (RAG) technology is integrated to optimize LLM input prompts, enhance the accuracy and scenario adaptability of open domain question and answer, and ensure the continuity of voice output through the fault-tolerant mechanism of the dynamic segmenter (minimum length merging, termination index record), ultimately forming a low-latency, high-accuracy, and scalable intelligent voice interaction system suitable for high-concurrency and strong real-time scenarios such as smart homes and real-time customer service. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0025] Figure 1 It is a schematic diagram of the technical solution of the traditional voice question-answering system; Figure 2 This is an architecture diagram of the existing technology using a multimodal large model for voice question answering; Figure 3 This is the architecture diagram of the intelligent question-answering service used in this application; Figure 4 This is the framework diagram of the streaming intelligent voice question-answering service of this application; Figure 5 It is a framework diagram of a streaming voice intelligent question-answering service system based on a large model provided by this application; Figure 6 is a flow chart of step S4-3 in an embodiment of the present application; Figure 7 It is a flow chart of step S5 in the embodiment of the present application. DETAILED DESCRIPTION

[0026] In order to make the above and other features and advantages of the present invention more clear, the present invention is further described below in conjunction with the accompanying drawings. It should be understood that the specific embodiments given herein are for the purpose of explaining to those skilled in the art and are only exemplary and not restrictive.

[0027] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0028] The present invention adopts a modular service architecture to design an intelligent question-answering system, decoupling speech recognition (ASR), large language model (LLM) and speech synthesis (TTS) services into independent service modules, supporting flexible selection of different model combinations for each module, while retaining high concurrency characteristics and being compatible with existing model services. The service architecture is as follows Figure 3As shown. To improve the real-time performance of audio response, the present invention adopts the "think while speaking" strategy based on the streaming generation characteristics of LLM and TTS. Through the parallel design of LLM service and TTS service, while LLM generates text, some content is dynamically segmented and submitted to TTS service to generate audio stream in parallel, which significantly reduces end-to-end delay. Furthermore, the retrieval-augmented generation (RAG) technology can be integrated into the LLM service, and the prompt construction can be enhanced by domain knowledge to improve the accuracy of question and answer and the adaptability to the scene.

[0029] Example: The Streaming Intelligent Voice Question and Answer (SIVQA) service system constructed by the present invention is based on three decoupled independent service modules: ASR, LLM, and TTS. LLM and TTS support streaming output, and the services do not interfere with each other and have high throughput. The SIVQA service can be constructed using any service that meets the above conditions.

[0030] The streaming voice intelligent question answering service system (SIVQA) provided by the present invention is as follows Figure 4As shown, its input is voice data, and its output is streaming audio data and returned to the client. The client can directly play while receiving the voice data. After receiving the voice input, the SIVQA service calls an independent speech recognition service (ASR Server) to convert the voice data into input text. Based on the input text, the prompt of the large language model (LLM) can be constructed according to different business scenarios, and a streaming request can be initiated to the independent large language model service (LLM Server). The LLM service can use a general large model or an enhanced model integrating RAG technology to generate text answers in a streaming manner. In order to obtain the voice answer to the question faster, the present invention designs a dynamic sentence segmenter, which divides the entire answer text of the LLM into multiple clauses in real time according to the result returned by the LLM streaming. The dynamic sentence segmenter segments the LLM answer text based on rules (such as punctuation marks, semantic boundaries), and at the same time, in order to shorten the delay of the first audio return, the maximum length of the first clause is limited (for example: 10 Chinese characters or 10 English words), thereby reducing the delay of the language streaming output. According to the strategy of "thinking while speaking", the answer text is dynamically segmented when the LLM streams the answer. For each sentence sj obtained by segmentation, a request is sent to an independent speech synthesis service (TTS Server) in parallel to convert the corresponding text into speech data. At the same time, the SIVQA service will detect the return status of the TTS request in parallel, and combine the audio results returned by each request into audio data chunks in sequence according to the order of the TTS request, and stream each audio data segment back to the client in real time.

[0031] Specifically, the implementation process of the method for constructing the real-time streaming voice intelligent question-answering service system provided by the present invention is as follows.

[0032] The present invention uses FastAPI to build an asynchronous service interface. Based on its asynchronous characteristics, the blocking parts in the system, such as automatic speech recognition (ASR) requests, large language model (LLM) streaming generation, and text-to-speech (TTS) requests, are designed as asynchronous non-blocking operations through multi-threading and asynchronous functions, thereby improving the concurrency of the service system and building an asynchronous streaming processing framework.

[0033] The design of the entire asynchronous service is as follows Figure 5 As shown, the specific implementation steps are as follows: 1. Service initialization phase: S1: Create a TTS request thread pool (TTS post ThreadPool) to control the number of running child threads and the concurrency of TTS concurrent requests.

[0034] S2: Create service routing and voice API interfaces, and define asynchronous endpoints for voice input and audio stream output.

[0035] 2. Service operation stage: Step 1: Receive audio data input by the user.

[0036] Step 2: Call an independent speech recognition service (ASR Server) through an asynchronous request to convert the received voice data into input text.

[0037] Step 3: Build the prompt of the large language model (LLM) based on the input text, initiate an LLM streaming request to the independent large language model service (LLM Server), and obtain the streaming text answer generated by the LLM in real time through the asynchronous streaming generator (async llm streamgenerator).

[0038] Step 4: Use the dynamic sentence segmenter to segment the streaming text generated by LLM in real time to generate multiple clauses. The segmentation rules include: Segmentation based on punctuation or semantic boundaries; Limit the maximum length of the first clause (L1), and force truncation if it exceeds the limit; If the clause length is less than the minimum threshold (L2), it is merged with the subsequent clause and output.

[0039] Step 5: Call an independent speech synthesis service (TTS Server) in parallel for each clause to convert the text into an audio data chunk. Specifically: Create an independent TTS request thread for each clause, and control the number of concurrent requests through the thread pool; Each TTS request thread stores the generated audio data blocks in the asynchronous audio queue (audio_chunks-Queue) in sequence; Detect the asynchronous audio queue in real time, extract the audio data blocks in the order they were queued, and return them to the client.

[0040] Step 6: Combine the generated audio data blocks into streaming audio data in the order of generation, and return them to the client in real time for playback.

[0041] The ASR Server, LLM Server and TTS Server are independent services that are decoupled from each other, and the LLM Server and TTS Server support streaming output.

[0042] The following is a detailed introduction: S1: Receive voice input data audio-data.

[0043] S2: Send the audio - data to the ASR service in an asynchronous request manner to avoid blocking the service interface, and wait for it to be converted into the text text.

[0044] S3: Concatenate the converted text text into an LLM prompt (prompt), create a termination event (stop - event) to interrupt the child thread, initialize an asynchronous audio queue (audio_chunks - Queue) to store the audio data chunks returned by TTS, and create a dynamic sentence splitter for sentence splitting.

[0045] S4: Create a streaming asynchronous LLM request, and the specific steps are as follows: S4 - 1: Send an LLM service request using the prompt to obtain an asynchronous streaming generator (async llmstream generator).

[0046] S4 - 2: Iterate over the asynchronous streaming generator to continuously obtain the streaming text output by the LLM. At the same time, dynamically perform sentence splitting according to the streaming text output by the LLM, and convert the segmented clauses into audio streams. The specific sub - steps are as follows: S4 - 2 - 1: Each time the asynchronous streaming generator is iterated, the text generated by the LLM gradually becomes longer. At this time, perform a dynamic sentence split to split out the newly generated clause sj (0 < j <= n); if no complete clause is split out, return an empty string.

[0047] S4 - 2 - 2: After each iteration, detect whether the termination event (stop - event) is triggered. If it is triggered, terminate the iteration and exit.

[0048] S4 - 2 - 3: For each complete clause sj generated, immediately create a child - thread task TTS - post - j and submit it to the thread pool (TTS ThreadPool) for execution to convert the text of the clause sj into audio data.

[0049] S4 - 2 - 3 - 1: In the TTS - post - j task, first create an independent asynchronous queue audio - chunk - j for the clause sj to store the audio data returned by the request, and add it to the asynchronous audio queue (audio_chunks - Queue).

[0050] S4 - 2 - 3 - 2: In the TTS - post - j task, send a streaming conversion request to the TTS Server using the clause sj, and sequentially store the returned audio data chunks into audio - chunk - j; S4-2-3-3: When the conversion of the audio streaming data returned in step S4-2-3-2 is completed, insert the end identifier None into audio-chunk-j to mark the completion of the generation of this audio segment.

[0051] S4-2-3-4: Each time an audio data block is stored in audio-chunk-j, it is detected whether the stop event is triggered. If it is triggered, all operations of the current TTS request thread are terminated.

[0052] S4-3: At each iteration of the asynchronous streaming generator, it is simultaneously detected whether there are audio data blocks in the asynchronous audio queue (audio_chunks-Queue). If so, the audio data blocks are returned to the service request end in an orderly manner according to the order in which they were enqueued.

[0053] S4-4: After iterating the asynchronous streaming generator, insert the end identifier end into the asynchronous audio queue (audio_chunks-Queue) to mark the completion of the LLM text generation task.

[0054] S5: After the iteration of step S4 is completed, if there are still unreturned audio data chunks in the asynchronous audio queue (audio_chunks-Queue), each audio data chunk (chunk) in the queue is returned in order according to the order in which it was queued.

[0055] S6: Build an asynchronous termination background function. When the client request triggers termination, the background function is activated. At this time, the stop event is triggered, terminating all sub-thread tasks in the current request.

[0056] In the above content, step S4-3 and step S5 together complete the orderly return of all audio data chunks, but due to the different stages, the two work in different ways.

[0057] (1) Step S4-3: During the iterative asynchronous streaming generator phase, each iteration of the text will take a certain amount of time. During this period, the audio_chunks-Queue may receive some audio data chunks, so some audio data chunks will be returned in order without interrupting the LLM iteration. The detailed steps are as follows (see Figure 6 ): S4-3-0: Create an empty variable audio_chunks before iteration to take out the audio-chunk-j queue in order.

[0058] S4-3-1: After each iteration, determine whether the length of audio_chunks-Queue is greater than 0. If it is greater than 0, there is a text-to-audio task. If audio_chunks is empty, then audio_chunks-Queue dequeues an audio-chunk and assigns it to audio_chunks.

[0059] S4-3-2: Determine whether audio_chunks is empty and whether the length of the audio_chunks queue is greater than 0.

[0060] S4-3-2-1: If audio_chunks is not empty and the length of the audio_chunks queue is greater than 0, pop out all the audio data blocks in the current audio_chunks queue in order and return them to the client, and then proceed to the next iteration.

[0061] If the value of the popped-up audio data block is None, it means that the text-to-audio conversion task is completed, and audio_chunks is assigned to None. In the next iteration, step S4-3-1 pops up audio-chunk from the audio_chunks-Queue queue and assigns it to audio_chunks.

[0062] S4-3-2-2: If audio_chunks is empty or the audio_chunks queue length is 0, continue to the next iteration.

[0063] (2) Step S5: At this stage, LLM has completed the generation of all texts. At this time, all clauses have completed the creation of TTS-post-j tasks. S4-3 may have returned some audio data blocks, and there may still be some audio data in audio_chunks that has not been arranged and returned. Therefore, at this stage, the audio data that has not been completely returned will continue to be arranged in order according to the queue entry order and returned to the client. The detailed steps are as follows (see Figure 7 ): S5-1: Check whether the stop-event is triggered. If it is triggered, exit the S5 step directly, otherwise continue.

[0064] S5-2: Determine whether audio_chunks is None: S5-2-1: If audio_chunks is None, dequeue an audio-chunk from the audio_chunks-Queue and assign it to audio_chunks, otherwise start executing from step S5-2-4.

[0065] S5-2-2: If the audio_chunks-Queue queue in the previous step is empty, then asynchronously sleep or wait for 10 milliseconds and repeat the execution from step S5-1 again, otherwise continue to execute downward.

[0066] S5-2-3: If audio_chunks is the character end, exit step S5 directly, otherwise continue.

[0067] S5-2-4: Take out all audio data blocks from audio_chunks in order and return them to the client. Details are as follows: S5-2-4-1: Check whether the stop-event is triggered. If it is triggered, exit the S5-2-4 step directly, otherwise continue.

[0068] S5-2-4-2: Determine whether the audio_chunks queue is empty: if it is empty, sleep asynchronously or wait for 10 milliseconds, and repeat the steps from S5-2-4-1; otherwise, continue to execute downward; S5-2-4-3: Pop an audio data block from audio_chunks; S5-2-4-4: If the pop-up audio data block is None, it means that the audio data in audio_chunks has been taken out, audio_chunks is set to None, and then the step S5-1 is executed; otherwise, the audio data block is returned to the client; S5-2-4-5: Continue to execute from S5-2-4-1 until the audio data block is None, and then jump out of the loop of S5-2-4.

[0069] The text generated by LLM contains punctuation marks, so the dynamic sentence segmenter uses common Chinese and English punctuation marks as the basis for sentence segmentation.

[0070] When segmenting sentences, the text generated by LLM is first preprocessed, and each Chinese character and English word is segmented as a separate character. After each traversal of the characters, the subscript index position idx of the last character is recorded. The next time the text generated by LLM is traversed, it starts from the idx character index position to avoid repeated traversal of the previously generated text content; at the same time, in order to avoid the high latency of the first audio data return, the maximum length of the first clause is limited to L1, and it is directly truncated when it exceeds L1. In addition, the minimum length of each clause is also limited to L2. If the clause length is less than L2, it is merged with the next sentence and returned.

[0071] In summary, the present invention significantly improves the real-time performance and flexibility of the voice question-answering system by decoupling ASR, LLM and TTS services, combining the asynchronous streaming framework with the "think while speaking" strategy. The dynamic sentence segmenter and parallel TTS processing effectively reduce the audio stream return delay, and the asynchronous queue management ensures the orderliness of the data and the system throughput.

[0072] The specific embodiments of the present invention are described in detail above, but they are only examples, and the present invention is not limited to the specific embodiments described above. For those skilled in the art, any equivalent modifications and substitutions made to the present invention are also within the scope of the present invention. Therefore, the equalization changes and modifications made without departing from the spirit and scope of the present invention should be included in the scope of the present invention.

Claims

1. A method for constructing a real-time streaming voice intelligent question-answering service system, characterized in that: The steps include: Step S1: receiving voice data input by a user; Step S2: calling an independent speech recognition service ASR Server to convert the speech data into input text; Step S3: construct a prompt of the large language model LLM based on the input text, initiate an LLM streaming request to the independent large language model service LLM Server, and obtain the streaming text answer generated by the LLM in real time; Step S4: segmenting the streaming text answer in real time by a dynamic sentence segmenter to generate multiple clauses; Step S5: calling an independent speech synthesis service TTS Server for each clause in parallel to convert the text into an audio data block; Step S6: combining the audio data blocks into streaming audio data in the order of generation, and returning the streaming audio data to the client for playback in real time; The ASR Server, LLM Server and TTS Server are independent services that are decoupled from each other, and the LLM Server and TTS Server support streaming output.

2. The method according to claim 1, characterized in that The working rules of the dynamic sentence segmenter include: Segment the streaming text generated by LLM according to punctuation or semantic boundaries; Limit the maximum length of the first clause. If it exceeds the threshold, it will be truncated. If the length of a clause is less than the minimum length threshold, it is merged with the subsequent clauses and output.

3. The method according to claim 1, characterized in that: The step S5 comprises: Create an independent TTS request thread for each clause, and control the number of concurrent requests through the thread pool; Each TTS request thread stores the generated audio data blocks in the asynchronous audio queue in sequence; The asynchronous audio queue is detected in real time, and the audio data blocks are extracted in the order in which they are queued and returned to the client.

4. The method according to claim 3, characterized in that The step S5 also includes a termination control step: When a termination request is received from the client, the termination event stop-event is triggered to terminate the current LLM streaming request, dynamic sentence segmentation and all TTS request threads.

5. The method according to claim 3, characterized in that: The TTS request thread execution process of each clause in step S5 includes: Create an independent asynchronous queue audio-chunk-j for clause sj, and add audio-chunk-j to the asynchronous audio queue; Send a streaming conversion request for clause sj to the TTS Server, and store the returned audio data blocks in order into audio-chunk-j; When the audio stream conversion of clause sj is completed, insert the clause end identifier None into audio-chunk-j; Each time an audio data block is stored in audio-chunk-j, a stop-event is detected. If the stop-event is triggered, the current TTS request thread is terminated.

6. The method according to claim 1, characterized in that In step S3, the request process of the LLM Server includes: Build prompts containing search enhancement generation technology based on business scenarios; Receive LLM-generated text snippets in real time via an asynchronous streaming generator.

7. The method according to claim 3, characterized in that In step S6, the implementation method of returning the streaming audio data includes: (a) During the text generation process of LLM, it is detected in real time whether there is an audio data block in the asynchronous audio queue. If so, it is returned in the order in which it was queued; (b) After the LLM generation is completed, the task end identifier end is inserted into the asynchronous audio queue, and the remaining audio data blocks in the asynchronous audio queue are continued to be processed and returned in the order in which they were put into the queue until the task end identifier end is detected or the asynchronous audio queue is cleared.

8. The method according to claim 7, characterized in that The step (a) specifically comprises the following steps: Step a.1: Before each iteration of LLM streaming text generation, create an empty temporary storage variable audio_chunks to cache the audio data of the currently processed subqueue; Step a.2: After each iteration of LLM generates text, check whether the asynchronous audio queue is non-empty: If the asynchronous audio queue is not empty and audio_chunks is empty, dequeue the first subqueue audio-chunk-j from the asynchronous audio queue and assign it to audio_chunks; Step a.3: Determine whether audio_chunks is not empty and contains unreturned audio data chunks: If the conditions are met, all audio data blocks in audio_chunks are extracted in order and returned to the client in real time; If the extracted audio data block is empty, reset audio_chunks to empty and get the next subqueue from the asynchronous audio queue again; If audio_chunks is empty or there is no extractable data, continue to the next LLM iteration.

9. The method according to claim 8, characterized in that The step (b) specifically comprises the following steps: Step b.1: After LLM generation is completed, the stop-event and asynchronous audio queue status are detected cyclically: If the stop-event is triggered, the processing flow will be exited directly; If audio_chunks is empty, dequeue a new subqueue audio-chunk-j from the asynchronous audio queue and assign it to audio_chunks; If the asynchronous audio queue is empty, wait for the preset time and then re-detect; Step b.2: When audio_chunks is the end identifier end, the processing flow is terminated; Step b.3: Extract audio data blocks from audio_chunks in order and return them to the client, while looping: If the audio data block is empty, reset audio_chunks to empty and recheck the queue; If audio_chunks is empty, continue extracting after asynchronous waiting; The extraction operation is interrupted immediately after the stop-event is detected.

10. The method according to claim 1, characterized in that The method is implemented through an asynchronous framework and includes: Use FastAPI to build asynchronous service interfaces; ASR requests, LLM streaming generation, and TTS requests are designed as asynchronous non-blocking operations.

Images