Adaptive buffer management method, system, device, medium and program product for streaming response

Abstract

The application discloses a kind of self-adapting buffer management methods, systems, equipment, medium and program product of streaming response, method includes: according to user request, determine minimum buffer granularity parameter;Incremental content segment generated based on user request is obtained;Incremental content segment is classified as inference segment and text segment, inference segment is written into inference buffer, and text segment is written into text buffer;For inference buffer, if the length of inference segment accumulated in inference buffer is greater than or equal to minimum buffer granularity parameter, then the inference segment in inference buffer is output and emptied;For text buffer, when the text segment in text buffer is first greater than or equal to minimum buffer granularity parameter, residual inference segment set with complete semantics in inference buffer is output, and the final answer text is formed using the text segment in text buffer and output.The application can effectively improve the perceived continuity of user on inference chain, thereby reducing the fragmentation output of inference content.

Description

Technical Field

[0001] This invention relates to the field of large model technology, and in particular to an adaptive buffer management method, system, device, medium and program product for streaming response. Background Technology

[0002] With the widespread application of multimodal big data models in scenarios such as dialogue, decision support, and content generation, user expectations for interaction have shifted from "waiting for the final answer" to "real-time access to the thought process and phased content".

[0003] However, traditional streaming output schemes have the following drawbacks:

[0004] 1. Most existing technologies distinguish between "reasoning" and "text response" as different chunk types in their implementation, but still use incremental refresh driven by pure token / character length, without recognizing the semantic boundaries of the reasoning chain (hypothesis formation → verification → convergence). This results in truncation at logical transitions, syntactic incompleteness, or temporary self-correcting statements. Users facing fragmented thinking need to repeatedly backtrack and piece together the information, and may mistake transitional hypotheses that have not yet converged for stable conclusions, reducing readability and decision accuracy.

[0005] 2. Although the existing "reasoning chunk" and "response chunk" are separated in terms of type, the output often overlaps in time and lacks an explicit stage event (start / completion) and a centralized flushing mechanism for the remaining reasoning chunks. The reasoning residue is delayed or appears intertwined with the first batch of answer texts. Users cannot determine whether the system is still reasoning or has entered the final draft stage. It is easy to prematurely adopt intermediate conclusions that have not been fully verified or to ignore the final integration section, which affects the task execution rhythm and risk control, and weakens the integrity of subsequent tracking / auditing.

[0006] 3. The reasoning process and the final executable / admissible response lack layered buffers and explicit stage markers. Exploratory thinking and definitive conclusions appear interspersed, making it difficult for users to distinguish between "still building hypotheses" and "finalized". This may lead to premature execution of unverified hypotheses or neglect of key conclusions, affecting the decision-making pace and risk control.

[0007] 4. Fixed buffering strategies cannot simultaneously address the real-time requirements and information integrity of different tasks. For example, the semantic block granularity differs significantly between steps such as image parsing, long text generation, and video recommendation.

[0008] 5. When a user terminates the flow (the flow ends), residual thoughts often remain and are not processed or output, resulting in loss of context or difficulty in tracing the source.

[0009] In addition, existing technical solutions often only provide a Boolean-level switch for "whether to display thinking", lacking comprehensive capabilities such as "intelligent completion and refresh of residual content during the thinking stage", "layered buffering of text and reasoning logic", and "dynamic minimum granular control".

[0010] Therefore, there is an urgent need to invent a buffer management algorithm that can adaptively adjust and support selective output of inference content in order to solve the problems existing in the current technology. Summary of the Invention

[0011] In view of this, embodiments of the present invention provide an adaptive buffer management method, system, device, medium, and program product for streaming response, which at least partially solves the problems existing in the prior art.

[0012] Other features and advantages of the invention will become apparent from the following detailed description, or may be learned in part by practice of the invention.

[0013] To achieve the above objectives, the embodiments of the present invention provide the following technical solutions:

[0014] According to a first aspect of the present invention, an adaptive buffer management method for streaming responses is provided, the method comprising:

[0015] Determine the minimum buffer granularity parameter based on the user's request;

[0016] Retrieve incremental content fragments generated based on user requests;

[0017] The incremental content fragments are classified into inference fragments and text fragments. The inference fragments are written to the inference buffer, and the text fragments are written to the text buffer.

[0018] For the inference buffer, if the cumulative length of the inference fragments in the inference buffer is greater than or equal to the minimum buffer granularity parameter, then the inference fragments in the inference buffer are output and the inference buffer is cleared.

[0019] For the text buffer, when a text fragment in the text buffer is greater than or equal to the minimum buffer granularity parameter for the first time, the residual inference fragments with complete semantics in the inference buffer are output in a concentrated manner, and the final answer text is formed and output using the text fragments in the text buffer.

[0020] Furthermore, based on user requests, the minimum buffer granularity parameters are determined, including:

[0021] Obtain user intent based on user request;

[0022] The task type is determined based on the user's intent;

[0023] The minimum buffer granularity parameter is dynamically adjusted based on the task type.

[0024] Furthermore, the inference fragment in the inference buffer is output, including:

[0025] The reasoning fragments in the reasoning buffer are combined into readable reasoning semantic blocks for output;

[0026] If the reasoning fragment is a critical exception statement, then the critical exception statement is output immediately;

[0027] If the reasoning fragment is an unclosed statement, then the unclosed statement is not output until it continues to accumulate and form a complete closed statement;

[0028] When a predetermined number of consecutive summarizing reasoning segments and / or predictive reasoning segments exist, the reasoning process terminates and proceeds to the text stage.

[0029] Furthermore, the residual reasoning fragments with complete semantics in the reasoning buffer are collected and output, and the text fragments in the text buffer are used to form the final answer text and output it, including:

[0030] Query the remaining inference fragments in the inference buffer;

[0031] If the residual inference fragment exists in the inference buffer, then the residual inference fragment is used to form a complete inference tail block and output.

[0032] Trigger and output the thought completion event;

[0033] Read the remaining text fragments in the text buffer;

[0034] The remaining text fragments are used to form a text tail block;

[0035] Based on the text tail block, the final text block is generated and output.

[0036] Furthermore, the method also includes:

[0037] If the user actively terminates the request or the system terminates abnormally, the contents of the inference buffer and text buffer are sorted out and then cleared.

[0038] Furthermore, the method also includes:

[0039] Link tracing is performed on the outputs of the inference buffer and the text buffer, and the output inference blocks and text blocks are aggregated in chronological order and archived.

[0040] According to a second aspect of the present invention, an adaptive buffer management system for streaming responses is provided, the system comprising:

[0041] Configuration management module, LLM text stream generator, buffer manager, inference buffer, text buffer, event handler, and tracing module;

[0042] The configuration management module is used to configure the minimum buffer granularity parameter;

[0043] The LLM text stream generator is used to generate incremental content fragments based on user requests;

[0044] The buffer manager is used to classify the incremental content fragments into inference fragments and text fragments, write the inference fragments into the inference buffer, and write the text fragments into the text buffer;

[0045] The inference buffer is used to store the inference fragments. If the length of the accumulated inference fragments in the inference buffer is greater than or equal to the minimum buffer granularity parameter, the inference fragments in the inference buffer are output and the inference buffer is cleared.

[0046] The text buffer is used to store the text fragments. When the text fragments in the text buffer are greater than or equal to the minimum buffer granularity parameter for the first time, the residual inference fragments with complete semantics in the inference buffer are output in a concentrated manner, and the text fragments in the text buffer are used to form the final answer text and output it.

[0047] The event processor is used to trigger and output a thinking completion event after a complete reasoning tail block is formed and output.

[0048] The tracking module is used to perform link tracing on the output of the inference buffer and the text cache, aggregate the output inference blocks and text blocks in chronological order, and archive and save them.

[0049] According to a third aspect of the present invention, an adaptive buffer management device for streaming response is provided, the device comprising: a processor and a memory;

[0050] The memory is used to store one or more program instructions;

[0051] The processor is configured to run one or more program instructions to perform the steps of an adaptive buffer management method for streaming responses as described in any of the preceding claims.

[0052] According to a fourth aspect of the present invention, a computer-readable storage medium is provided, on which a computer program is stored, wherein when executed by a processor, the computer program implements the steps of an adaptive buffer management method for streaming response as described in any of the preceding claims.

[0053] According to a fifth aspect of the present invention, a computer program product is provided, the computer program product comprising a computing program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions that, when executed by a computer, cause the computer to implement the steps of an adaptive buffer management method for streaming response as described in any of the preceding claims.

[0054] This invention discloses an adaptive buffer management method, system, device, medium, and program product for streaming responses. The method includes: determining a minimum buffer granularity parameter based on a user request; acquiring incremental content fragments generated based on the user request; classifying the incremental content fragments into inference fragments and text fragments, writing the inference fragments into an inference buffer, and writing the text fragments into a text buffer; for the inference buffer, if the cumulative length of the inference fragments in the inference buffer is greater than or equal to the minimum buffer granularity parameter, then outputting the inference fragments in the inference buffer and clearing the inference buffer; for the text buffer, when the length of the text fragments in the text buffer is greater than or equal to the minimum buffer granularity parameter for the first time, then outputting the remaining inference fragments with complete semantics in the inference buffer, using the text fragments in the text buffer to form the final answer text and outputting it. This invention can effectively improve the user's perception of the continuity of the inference chain, thereby reducing the fragmented output of inference content. Attached Figure Description

[0055] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely exemplary, and those skilled in the art can derive other embodiments based on the provided drawings without creative effort.

[0056] Figure 1 A flowchart illustrating an adaptive buffer management method for streaming response provided in an embodiment of the present invention;

[0057] Figure 2 A schematic diagram of the architecture of a streaming response adaptive buffer management system provided in an embodiment of the present invention;

[0058] Figure 3 This is a schematic diagram of the interaction process of each module in a streaming response adaptive buffer management system provided in an embodiment of the present invention. Detailed Implementation

[0059] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0060] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0061] Figure 1 A flowchart of an adaptive buffer management method for streaming responses according to an embodiment of the present invention is shown.

[0062] like Figure 1 As shown, the adaptive buffer management method for streaming response according to an embodiment of the present invention may include steps S100, S200, S300, S401 and S402.

[0063] Specifically, taking the scenario of a user requesting "Please help me analyze the recent order sales situation" as an example, the system is expected to first present data understanding and diagnostic reasoning (original structure → outliers → trend hypothesis), and then output summary conclusions and action suggestions. This scenario has the typical characteristics of data table structuring, indicator trends and multi-stage reasoning.

[0064] In step S100, the minimum buffer granularity parameter is determined according to the user request.

[0065] Specifically, the above steps include:

[0066] Based on the user's request "Please help me analyze the recent order sales situation", the user intent is parsed and the task type is determined to be "Sales Analysis" based on the user intent. The minimum buffer granularity parameter min_buffer_size is dynamically adjusted according to the task type to speed up the initial feedback of key indicators.

[0067] The minimum buffer granularity parameter min_buffer_size is used to determine whether the current accumulated content has reached the standard of a "semantic block" that can be output at one time. It can be approximated by character length or extended to token / sentence level, supporting different model contexts and providing tuning space for different scenarios (long text generation vs. fast question answering).

[0068] Next, prepare the data assets for the response, internally retrieve order records from the last 30 days (such as date, quantity, actual payment amount, etc.) as conceptual reasoning material, rather than directly using them for streaming output.

[0069] Optionally, a thinking input switch can be set to selectively control thinking input. The reasoning buffer can be dynamically turned off through external configuration to ensure privacy, compliance or simplicity mode requirements. The text buffer logic can still run independently after the reasoning buffer is turned off.

[0070] The embodiments of the present invention employ a dual-buffering separation strategy, which accumulates reasoning fragments (thinking chains, structured intermediate conclusions) in the model output separately from the final text content that can be presented, avoiding semantic interference and irrelevant frequent refreshes. The reasoning content can be stripped off and turned off without affecting the stability of text refresh.

[0071] In step S200, incremental content fragments generated based on user requests are obtained.

[0072] Then, in step S300, the incremental content fragments are classified into inference fragments and text fragments, the inference fragments are written to the inference buffer, and the text fragments are written to the text buffer.

[0073] Optionally, the above-mentioned buffer can be integrated with multimodal content (intermediate tags for image parsing, recommended video candidate sets) as an independent buffer category, which is easy to expand.

[0074] Next, in step S401, for the inference buffer, if the cumulative length of the inference fragments in the inference buffer is greater than or equal to the minimum buffer granularity parameter, the inference fragments in the inference buffer are output and the inference buffer is cleared.

[0075] Specifically, the above steps include:

[0076] The inference fragments written to the inference buffer are interpreted in a structured manner. After several short sentences reach the minimum buffer granularity parameter, they are combined and output in a group (e.g., 3-4 sentences) to form a readable inference block, thereby reducing fragmentation. For example, the short sentences "Top 5 SKUs account for 62% of total sales", "Channel A order number increased by 15% month-on-month", and "Average order value of promotion-marked orders is 8% lower" are combined and output.

[0077] The inference fragments written to the inference buffer are processed for abnormal statements. For example, when abnormal fragments such as "an abnormal peak occurred in the middle of week 3, suspected to be a large-scale event" and "SKUX's conversion rate did not drop after the event, indicating a potential increase in structural demand" are identified, the threshold judgment module immediately refreshes the buffer when the cumulative length reaches the standard, ensuring that key anomalies are quickly seen by users.

[0078] The inference fragments written to the inference buffer are used to construct trend assumptions to avoid outputting inference content that is not yet closed (such as "If the current growth rate of channel A is maintained next week, the upper limit of weekly sales is expected to increase by 10%~12%").

[0079] When multiple consecutive reasoning segments are detected to end with summarizing or predictive words (such as "expected", "may lead to", "to form stability", etc.), the stage coordinator is about to enter the response stage.

[0080] In step S401, for the text buffer, when the text fragment in the text buffer is greater than or equal to the minimum buffer granularity parameter for the first time, the residual inference fragments with complete semantics in the inference buffer are output in a concentrated manner, and the final answer text is formed and output using the text fragments in the text buffer.

[0081] Specifically, the above steps include:

[0082] When a text fragment in the text buffer (such as "Based on the sales performance of the past 30 days, it can be summarized into three main characteristics:...") is greater than or equal to the minimum buffer granularity parameter for the first time, query the residual inference fragment in the inference buffer.

[0083] If there are residual inference fragments in the inference buffer, the flushing event lush_remaining_reasoning is triggered, which forms a complete inference tail block and outputs the residual inference fragments that have not yet reached the minimum buffer granularity parameter but have complete semantics.

[0084] Then, the "Thinking Complete" event THINKING_COMPLETE is output, indicating to the user that the subsequent content will move into the final summary, reducing the cost of judging whether the subsequent blocks still belong to "reasoning".

[0085] Read the remaining text fragments in the text buffer, use the remaining text fragments to form a text tail block, and generate and output the final text block based on the text tail block.

[0086] The final text block mentioned above includes a summary block and a suggestion block.

[0087] The summary block above is a text block formed by the description of each feature and supporting data sentences, such as "Feature 1: Increased sales concentration; Feature 2: Channel structure shifts towards A; Feature 3: Promotional activities temporarily put downward pressure on average order value".

[0088] The aforementioned suggestion blocks are segmented and buffered during the generation process to ensure that each suggestion block is presented completely. Examples include "Suggestion 1: Increase the regular inventory allocation for SKU X", "Suggestion 2: Conduct targeted promotions for retention in Channel B", and "Suggestion 3: Review the ROI of the third week's campaign". These action semantics are segmented and buffered during the generation process to ensure that each suggestion block is presented completely.

[0089] In addition, statements related to risks and assumptions are output as separate supplementary blocks, such as "The forecast growth rate is subject to high uncertainty due to the holiday effect".

[0090] This invention, through a thinking stage transition, triggers a concentrated flush of the remaining reasoning buffer when the first text content block is about to be output or has completed the output stage, ensuring the integrity of the thinking ending and preventing the loss of fragments. Subsequently, a "thinking completed" event is issued to clarify the semantic boundaries of the stage and avoid fragmented segments affecting the continuity of reading.

[0091] Preferably, if the user terminates midway through the inference phase, a double-buffered residual flush is immediately performed to ensure that key diagnostics that have been formed but not yet output are not lost. If the user terminates during the text phase, the tail segments of the unrefreshed summary and suggestion blocks will still be output and marked "User Aborted" to avoid content residue caused by interruption.

[0092] Preferably, link tracing is performed on the outputs of the inference buffer and text buffer, aggregating the output inference blocks and text blocks in chronological order to construct a complete analysis chain from data observation to anomaly detection, trend hypothesis, and conclusion suggestions. This supports subsequent evaluation metrics (such as "average length of inference blocks," "semantic closure rate," and "number of residual flushing events and word count") and provides feedback for model optimization (e.g., if the fragmentation ratio is high at a certain stage, the min_buffer_size can be adjusted or the semantic boundary recognition strategy can be enhanced). Throughout the stream's lifecycle, inference and text blocks are collected and concatenated into an ordered semantic sequence, facilitating subsequent auditing, retrieval, or link tracing.

[0093] Corresponding to the above-described adaptive buffer management method for streaming responses, the present invention also provides an adaptive buffer management system for streaming responses, the system comprising a configuration management module, an LLM text stream generator, a buffer manager, an inference buffer, a text buffer, an event handler, and a tracing module.

[0094] The configuration management module described above is used to configure the minimum buffer granularity parameter and the think input switch.

[0095] The LLM text stream generator described above is used to generate incremental content fragments based on user requests.

[0096] The aforementioned buffer manager is used to classify incremental content fragments into inference fragments and text fragments, write inference fragments to the inference buffer, and write text fragments to the text buffer.

[0097] The aforementioned inference buffer is used to store inference fragments. If the length of the accumulated inference fragments in the inference buffer is greater than or equal to the minimum buffer granularity parameter, the inference fragments in the inference buffer are output and the inference buffer is cleared.

[0098] The aforementioned text buffer is used to store text fragments. When the text fragment in the text buffer is greater than or equal to the minimum buffer granularity parameter for the first time, the residual inference fragments with complete semantics in the inference buffer are collected and output. The text fragments in the text buffer are used to form the final answer text and output it.

[0099] The aforementioned event handler is used to trigger and output the "thinking complete" event after a complete reasoning tail block is formed and output.

[0100] The aforementioned tracing module is used to perform link tracing on the output of the inference buffer and text buffer, aggregate the output inference blocks and text blocks in chronological order, and archive and save them.

[0101] Figure 2 A schematic diagram of the architecture of the adaptive buffer management system for streaming response provided by the present invention is shown.

[0102] Figure 3 This diagram illustrates the interaction flow of each module in a streaming response adaptive buffer management system provided by the present invention.

[0103] In addition, embodiments of the present invention also provide an adaptive buffer management device for streaming responses, the device comprising: a processor and a memory; the memory for storing one or more program instructions; the processor for executing one or more program instructions to perform the steps of an adaptive buffer management method for streaming responses as described above.

[0104] In addition, embodiments of the present invention also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of an adaptive buffer management method for streaming response as described above.

[0105] In addition, embodiments of the present invention also provide a computer program product, which includes computer program instructions that, when executed by a processor, implement the steps of an adaptive buffer management method for streaming response as described above.

[0106] The adaptive buffer management method, system, device, medium, and program product for streaming response provided in this invention have the following advantages:

[0107] 1. Improve user perception of continuity: The reasoning chain appears as a more complete semantic block, thereby reducing fragmentation.

[0108] 2. Supports privacy and mode switching: When inference output is turned off, only the text stream is displayed, which is suitable for government and enterprise confidentiality or terminal lightweight mode.

[0109] 3. Reduce cognitive load: Clearly define the "start of thinking" and "completion of thinking" events to help users grasp the stage of the response.

[0110] 4. Reduce loss and residue: Ensure content integrity through residual flushing during termination or phase switching.

[0111] 5. Balance between adjustable performance and real-time performance: Adaptive optimization is achieved by dynamically adjusting the minimum buffer granularity disk wiping number min_buffer_size.

[0112] 6. Easy to trace and audit: Utilizes a link tracing system to obtain double-buffered aggregated output, enhancing post-review capabilities.

[0113] 7. Compatibility and Expansion: Different buffer types such as "structured reasoning tags" and "multimodal intermediate descriptions" can be added in the future.

[0114] In this embodiment of the invention, the processor can be an integrated circuit chip with signal processing capabilities. The processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in this embodiment of the invention. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in this embodiment of the invention can be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The processor reads information from the storage medium and, in conjunction with its hardware, completes the steps of the above methods. The storage medium can be memory, for example, volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDRSDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchlink DRAM (SLDRAM), and Direct Rambus RAM (DRRAM).The storage media described in the embodiments of this invention are intended to include, but are not limited to, these and any other suitable types of memory. Those skilled in the art will recognize that the functions described in the above examples can be implemented using a combination of hardware and software. When applied software, the corresponding functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein communication media include any medium that facilitates the transmission of computer programs from one place to another. Storage media can be any available medium accessible to general-purpose or special-purpose computers. Although the invention has been described in detail above with general description and specific embodiments, modifications or improvements can be made to it, which will be apparent to those skilled in the art. Therefore, such modifications or improvements made without departing from the spirit of the invention are all within the scope of protection claimed by the invention.

[0115] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Any simple modifications, equivalent changes, or alterations made by those skilled in the art using the disclosed technical content shall fall within the protection scope of the present invention.

Claims

1. An adaptive buffer management method for streaming response, characterized in that, The method includes: Determine the minimum buffer granularity parameter based on the user's request; Retrieve incremental content fragments generated based on user requests; The incremental content fragments are classified into inference fragments and text fragments. The inference fragments are written to the inference buffer, and the text fragments are written to the text buffer. For the inference buffer, if the cumulative length of the inference fragments in the inference buffer is greater than or equal to the minimum buffer granularity parameter, then the inference fragments in the inference buffer are output and the inference buffer is cleared. For the text buffer, when the text fragment in the text buffer is greater than or equal to the minimum buffer granularity parameter for the first time, the residual inference fragments with complete semantics in the inference buffer are output in a concentrated manner, and the final answer text is formed and output using the text fragments in the text buffer; The remaining inference fragments with complete semantics in the inference buffer are collected and output. The text fragments in the text buffer are used to form the final answer text and output it, including: Query the remaining inference fragments in the inference buffer; If the residual inference fragment exists in the inference buffer, then the residual inference fragment is used to form a complete inference tail block and output. Trigger and output the thought completion event; Read the remaining text fragments in the text buffer; The remaining text fragments are used to form a text tail block; Based on the text tail block, the final text block is generated and output.

2. The adaptive buffer management method for streaming response according to claim 1, characterized in that, Based on the user's request, determine the minimum buffer granularity parameters, including: Obtain user intent based on user request; The task type is determined based on the user's intent; The minimum buffer granularity parameter is dynamically adjusted based on the task type.

3. The adaptive buffer management method for streaming response according to claim 1, characterized in that, Output the inference fragments in the inference buffer, including: The reasoning fragments in the reasoning buffer are combined into readable reasoning semantic blocks for output; If the reasoning fragment is a critical exception statement, then the critical exception statement is output immediately; If the reasoning fragment is an unclosed statement, then the unclosed statement is not output until it continues to accumulate and form a complete closed statement; When a predetermined number of consecutive summarizing reasoning segments and / or predictive reasoning segments exist, the reasoning process terminates and proceeds to the text stage.

4. The adaptive buffer management method for streaming response according to claim 1, characterized in that, The method further includes: If the user actively terminates the request or the system terminates abnormally, the contents of the inference buffer and text buffer are sorted out and then cleared.

5. The adaptive buffer management method for streaming response according to claim 1, characterized in that, The method further includes: Link tracing is performed on the outputs of the inference buffer and the text buffer, and the output inference blocks and text blocks are aggregated in chronological order and archived.

6. A streaming response adaptive buffer management system, characterized in that, The system includes: a configuration management module, an LLM text stream generator, a buffer manager, an inference buffer, a text buffer, an event handler, and a tracing module; The configuration management module is used to configure the minimum buffer granularity parameter; The LLM text stream generator is used to generate incremental content fragments based on user requests; The buffer manager is used to classify the incremental content fragments into inference fragments and text fragments, write the inference fragments into the inference buffer, and write the text fragments into the text buffer; The inference buffer is used to store the inference fragments. If the length of the accumulated inference fragments in the inference buffer is greater than or equal to the minimum buffer granularity parameter, the inference fragments in the inference buffer are output and the inference buffer is cleared. The text buffer is used to store the text fragments. When the text fragments in the text buffer are greater than or equal to the minimum buffer granularity parameter for the first time, the residual inference fragments with complete semantics in the inference buffer are output in a concentrated manner, and the text fragments in the text buffer are used to form the final answer text and output it. The remaining inference fragments with complete semantics in the inference buffer are collected and output. The text fragments in the text buffer are used to form the final answer text and output it, including: Query the remaining inference fragments in the inference buffer; If the residual inference fragment exists in the inference buffer, then the residual inference fragment is used to form a complete inference tail block and output. Trigger and output the thought completion event; Read the remaining text fragments in the text buffer; The remaining text fragments are used to form a text tail block; Based on the aforementioned text end block, generate and output the final text block; The event processor is used to trigger and output a thinking completion event after a complete reasoning tail block is formed and output. The tracking module is used to perform link tracing on the output of the inference buffer and the text cache, aggregate the output inference blocks and text blocks in chronological order, and archive and save them.

7. A streaming response adaptive buffer management device, characterized in that, The device includes: a processor and a memory; The memory is used to store one or more program instructions; The processor is configured to run one or more program instructions to perform the steps of the adaptive buffer management method for streaming responses as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of an adaptive buffer management method for streaming responses as described in any one of claims 1 to 5.

9. A computer program product, characterized in that, The computer program product includes computer program instructions that, when executed by a processor, implement the steps of an adaptive buffer management method for streaming responses as described in any one of claims 1 to 5.

Images