A large model-based streaming response method and device

Abstract

Embodiments of the present specification provide a large model-based streaming response method and device. The method comprises: obtaining updated content output by a large model in a current stage; inputting the updated content into a buffer, and determining whether there is a target semantic boundary marker in the content in the buffer, the buffer being used to cache updated content in the current stage and historical incomplete recognition content, the historical incomplete recognition content being content in updated content output by a large model in a previous stage that is not determined to belong to a semantic boundary marker; determining target content with a target semantic type in the buffer according to the presence of the target semantic boundary marker in the buffer; and pushing the target content to a front end to enable the front end to render the target content according to the target semantic type. The method can present the output content of the large model in a low-latency, semantically clear and stable rendering manner.

Description

Technical Field

[0001] The embodiments in this specification relate to the field of artificial intelligence technology, and more specifically, to a streaming response method and apparatus based on a large model. Background Technology

[0002] Large Language Models (LLMs) typically generate text progressively on a token-by-token basis, thus inherently possessing a chunk-by-chunk output characteristic. Various application systems built upon LLMs, such as question-answering systems, dialogue systems, content generation systems, and agent systems, often require all or part of the output to be presented to the end user in a streaming manner during the model's generation process to improve the real-time nature of interaction and the user experience.

[0003] In practical applications, the content generated by large language models may include not only the target response displayed to the user, but also intermediate generated content, internally processed content, or other content that is not suitable for direct display. For example, in some agent systems, especially in implementations represented by the ReAct (Reasoning + Acting) paradigm, large language models can continuously generate task decomposition results, staged reasoning content (such as thought chains), tool invocation instructions, tool return results, and final answer content through a loop of alternating reasoning and action. In ordinary question-and-answer or dialogue scenarios, thought chains and answer content may also appear. Since the above content is usually output step by step according to word units, how to identify, classify, and selectively output the generated content in real time becomes an important technical problem that needs to be solved when large language model application systems achieve streaming responses.

[0004] In related technologies, one approach involves waiting for the large language model to generate a complete response before cleaning, filtering, or reorganizing the results on the backend. This could involve removing thought processes through rule matching or regular expression replacement before returning the processed response to the client. While this approach avoids directly exposing inappropriate content to the user, the long wait for complete generation results in delayed feedback and high latency, making it unsuitable for real-time interaction in time-consuming tasks. Another approach is to directly push the raw content generated by the large language model to the client, where it is parsed, filtered, and rendered. While this reduces backend processing time, the raw output may contain internal processing content, control tags, structural fragments, or incompletely closed units. Direct transmission of these components could leak internal reasoning steps (such as trial and error or sensitive logic) and lead to client-side parsing errors, display inconsistencies, rendering jitter, or unstable presentation. A third approach involves collecting model output within fixed time windows and sending it in batches after each window expires. This method can shorten the first packet waiting time compared to the scheme of generating the entire packet before returning it, but since the output is still pushed in an intermittent batch manner, it is difficult to achieve a continuous and smooth real-time streaming response.

[0005] Furthermore, existing technologies for filtering and separating content generated by large language models typically employ methods such as pattern matching, rule truncation, or marker recognition. However, in streaming scenarios, model output is often received continuously as incomplete fragments. Related markers, structural boundaries, or semantic units may be split into multiple fragments, leading to problems such as inaccurate boundary recognition, content truncation, residual markers, or fragment misalignment in simple processing methods. This, in turn, affects the continuity and accuracy of streaming output, as well as the terminal rendering effect.

[0006] Therefore, it is necessary to provide a new streaming response technology solution to achieve low-latency, continuous, and terminal-friendly real-time streaming output. Summary of the Invention

[0007] This specification provides a streaming response scheme based on large models, which can present the output content of large models in a low-latency, semantically clear, and stable rendering manner.

[0008] Firstly, embodiments of this specification provide a streaming response method based on a large model, comprising: acquiring updated content output by the large model in the current stage; inputting the updated content into a buffer and determining whether the content in the buffer contains a target semantic boundary marker, the buffer being used to cache the updated content of the current stage and historical unidentified content, the historical unidentified content being content in the updated content output by the large model in the previous stage that was not determined to belong to a semantic boundary marker; determining target content with a target semantic type in the buffer based on the presence of the target semantic boundary marker in the buffer; and pushing the target content to the front end so that the front end renders the target content according to the target semantic type.

[0009] In some embodiments, obtaining the updated content based on the output of the large model at the current stage includes: obtaining the context update event of the large model at the current stage, wherein the context update event carries the cumulative output content of the large model; comparing the cumulative output content with the historical cumulative output content, and extracting the difference information in the cumulative output content relative to the historical cumulative output content as the updated content, wherein the historical cumulative output content is the content carried in the context update event of the large model at the previous stage.

[0010] In some embodiments, determining the target content with a target semantic type in the buffer based on the presence of the target semantic boundary marker in the buffer includes: if the target semantic boundary marker exists in the buffer, determining the position of the target semantic boundary marker in the buffer, and determining the content before the target semantic boundary marker as the target content with a semantic type corresponding to the current data state, wherein the current data state is determined based on the previously identified semantic boundary marker.

[0011] In some embodiments, the method further includes: updating the current data state based on the target semantic boundary marker; and determining the content after the target semantic boundary marker as target content with a semantic type corresponding to the current data state.

[0012] In some embodiments, determining the target content with a target semantic type in the buffer based on the presence of the target semantic boundary marker in the buffer includes: if the target semantic boundary marker is not present in the buffer, determining whether the end part of the content in the buffer matches the prefix part of the target semantic boundary marker; if the end part matches the prefix part of the target semantic boundary marker, retaining the end part as historical incomplete identification content, and determining the content in the buffer other than the historical incomplete identification content as target content with a semantic type corresponding to the current data state, wherein the current data state is determined based on the semantic boundary marker identified last time.

[0013] In some embodiments, the method further includes: if the end portion does not match the prefix portion of the target semantic boundary marker, then determining all content in the buffer as target content with the semantic type corresponding to the current data state based on the current data state, wherein the current data state is determined based on the semantic boundary marker identified last time.

[0014] In some embodiments, the current data state includes a thinking state and a response state, the semantic type corresponding to the thinking state is a thinking type, the semantic type corresponding to the response state is a response type, and the semantic boundary markers include a thinking start marker and a thinking end marker.

[0015] In some embodiments, the method further includes: after the streaming response of the large model ends, obtaining the dialogue logs, tool call statistics and generated diary summaries during the session; and storing the dialogue logs, tool call statistics and diary summaries in association.

[0016] Secondly, embodiments of this specification provide a streaming response device based on a large model, comprising: an acquisition module for acquiring updated content output by the large model in the current stage; a buffer module for inputting the updated content into a buffer and determining whether the content in the buffer contains a target semantic boundary marker, the buffer being used to cache the updated content of the current stage and historical unidentified content, the historical unidentified content being content in the updated content output by the large model in the previous stage that was not determined to belong to a semantic boundary marker; a determination module for determining target content with a target semantic type in the buffer based on the presence of the target semantic boundary marker in the buffer; and a push module for pushing the target content to the front end so that the front end renders the target content according to the target semantic type.

[0017] Thirdly, embodiments of this specification provide a computing device including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, it implements the method described in any of the implementations in the first aspect.

[0018] In the solution provided in this specification, a buffer is introduced to cache historically incompletely identified content (i.e., content that was not determined to belong to semantic boundary markers in the previous stage). The currently updated content is then concatenated with the historically incompletely identified content. By determining whether target semantic boundary markers exist in the buffer, cross-segment semantic boundary markers are accurately identified, thereby precisely dividing the buffer content into target content with specific semantic types. This mechanism effectively avoids parsing anomalies caused by marker truncation, enabling the backend to complete semantic separation of content at the source and push target content with semantic type identifiers to the frontend. The frontend can then perform differentiated rendering (e.g., folding, hiding, or displaying thought content in a special style, while displaying answer content normally), thus eliminating the risk of frontend rendering jitter, errors, or displaying illegal characters. Furthermore, by acquiring the updated content of the current stage and processing it in real time by inputting it into the buffer, there is no need to wait for the complete response to be generated before transmission, nor is it limited by a fixed time window. Once the target content with the target semantic type is identified in the buffer, it can be pushed to the front end immediately. This incremental real-time processing mechanism reduces the response latency perceived by the user, achieves smooth and continuous streaming output, and improves the fluency of human-computer interaction. Attached Figure Description

[0019] To more clearly illustrate the technical solutions of the embodiments in this specification, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart of a streaming response method based on a large model, as described in the embodiments of this specification.

[0021] Figure 2 This is a schematic diagram of the streaming response system architecture based on a large model in the embodiments of this specification;

[0022] Figure 3 This is a flowchart of another streaming response method based on a large model in the embodiments of this specification;

[0023] Figure 4 This is a schematic diagram of the structure of the streaming response device based on a large model in the embodiments of this specification. Detailed Implementation

[0024] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this specification.

[0025] To facilitate understanding of the solutions in this specification, some terms involved in this solution will be explained below.

[0026] In this specification, the large language model may also be referred to simply as the large model. A large language model is a natural language processing model based on deep learning techniques, typically with billions to hundreds of billions or even more parameters, possessing powerful language understanding and generation capabilities. Large language models can employ the Transformer architecture or its variants (such as GPT, BERT, etc.), which utilizes an attention mechanism to globally model sequential data, efficiently handling long-distance dependencies and thus performing exceptionally well in natural language tasks. Large language models learn the statistical features and semantic relationships of language through pre-training on large-scale corpora, enabling them to generalize. The core capabilities of large language models include, but are not limited to: understanding contextual semantics, generating coherent and grammatically correct text, performing logical reasoning, and handling multi-task scenarios. Their usage typically includes two modes: direct inference and fine-tuning. In direct inference mode, the user guides the large language model to generate specific outputs by designing prompts. Prompts can be task descriptions or instructions in text form, used to stimulate the large language model's semantic understanding and generation capabilities. In fine-tuning mode, large language models are further trained on small-scale datasets within a specific domain to optimize their performance on specific tasks. The powerful generalization capabilities and flexibility of large language models make them an important tool in the field of artificial intelligence, providing efficient and accurate solutions for automated text generation and understanding.

[0027] In some embodiments, large language models can also understand and generate data from other modalities (such as visual and audio data). In this case, large language models can also be called multimodal large language models (MLLMs). MLLMs provide a richer and more natural interactive experience by integrating multiple types of input and output, such as text, images, and sound. The core advantage of MLLMs lies in their ability to process and understand information from different modalities and fuse this information to complete complex tasks. For example, MLLMs can analyze an image and generate descriptive text, or generate a corresponding image based on a text description. This cross-modal understanding and generation capability makes MLLMs widely applicable across multiple fields.

[0028] It should be noted that the key technologies of large language models can be found in the detailed description in the paper "A Survey of Large Language Models" (paper number: arXiv:2303.18223v16, published on March 11, 2025), and will not be repeated here.

[0029] An intelligent agent is an intelligent system capable of perceiving external input, understanding task objectives, making autonomous decisions, and completing target tasks by executing corresponding operations. In application scenarios based on large language models, intelligent agents typically use large language models as the core cognitive and reasoning units. This enables the system not only to understand the user's needs expressed in natural language but also to analyze, break down, plan, and execute tasks based on contextual information, thus exhibiting a certain degree of autonomy, interactivity, and goal orientation.

[0030] Compared to traditional large language model applications that only perform single-turn question answering or text generation, intelligent agents place greater emphasis on closed-loop capabilities from understanding to action. That is, intelligent agents not only need to generate text results, but also need to decide whether to invoke external tools, access knowledge bases, query interfaces, execute code, operate business systems, or continuously adjust strategies throughout the multi-step process, depending on task requirements. Especially in complex task scenarios, intelligent agents typically do not complete all processing at once, but rather gradually approach the target result through multiple rounds of reasoning, state updates, and action execution. Therefore, intelligent agents are better suited to task scenarios requiring planning, feedback, and dynamic decision-making, such as information retrieval, data analysis, process automation, business collaboration, and complex question answering.

[0031] To ensure that intelligent agents can complete tasks stably and efficiently, a supporting execution engine is typically required to uniformly schedule, control, and manage their operation. The execution engine is the core component supporting the agent's task execution, responsible for transforming the understanding, reasoning, and planning capabilities of the large language model into a feasible execution flow. Typically, after receiving a task request from the user, the execution engine, combined with preset system prompts, contextual information, and descriptions of available tools, drives the large language model to analyze, break down, and make decisions about the task. During execution, it coordinates multiple stages, including model generation, tool invocation, result observation, state maintenance, and final result output.

[0032] In actual operation, the execution engine typically needs to manage multiple rounds of interaction. For example, in implementations represented by paradigms such as ReAct, the execution engine organizes the large language model to alternately perform reasoning and action: on the one hand, the large language model generates the analysis results, execution intent, or invocation decisions for the current step; on the other hand, the execution engine triggers corresponding external tools, service interfaces, or internal modules based on the generated results, and re-injects the results returned by the tools into the context, allowing the large language model to carry out the next round of reasoning until the task objective is achieved. Thus, the execution engine not only undertakes model invocation control functions, but also assumes responsibilities such as process orchestration, context management, tool scheduling, and execution status tracking.

[0033] Furthermore, intelligent agent execution engines designed for real-world applications typically possess streaming output capabilities. For instance, in streaming interaction scenarios, the execution engine can push real-time updates generated by a large language model during execution to the client, enhancing the user's perception of task progress.

[0034] As mentioned earlier, during the execution of large models or agents, the model typically generates a large amount of output information, including inference content and the final answer, step by step. The front-end's requirements for displaying the model's output content differ across application scenarios. For example, in large model question-answering scenarios, users usually only want to see the final answer, so the inference part needs to be removed from the model's output, and the cleaned answer content returned to the front-end. In contrast, in deep inference scenarios, users typically want to see both the inference process and the final answer simultaneously, so the front-end needs to use different rendering methods for the inference content and the answer content.

[0035] However, in existing streaming solutions, most large model interfaces return a single text stream. Each chunk (fragment or data block) typically contains only a newly added string of characters and does not carry semantic tags for the content type. Furthermore, boundary markers used to distinguish different semantic content may be split into multiple chunks during streaming, making it difficult to accurately identify the semantic type of the current content based on a single chunk.

[0036] For example, the original output of the large model is: <think> Internal reasoning content< / think> The final answer.

[0037] During streaming, it can be arbitrarily split into:

[0038] Chunk 1: <thi

[0039] Chunk 2: nk> Internal Reasoning

[0040] Chunk 3: Content

[0041] Chunk 4: ink> Final Answer

[0042] In the above circumstances, <think>The start and end markers span multiple chunks, and there is no complete fragment within a single segment. <think> ...< / think> If regular expressions are used only for a single chunk <think> .*?< / think> Identifying the start and end points of the inference content will not allow for the identification of split tags. In such cases, the front-end typically needs to wait for the complete output content to arrive before rendering; otherwise, during streaming, the front-end may temporarily receive incomplete Markdown, JSON, or tag content, leading to issues such as unclosed code blocks, subsequent content being misinterpreted as code, incorrect tag hierarchy, and JSON parsing failures. Although the rendering result may return to normal after subsequent chunks arrive, users may still experience noticeable page flickering, jittering, or brief formatting errors.

[0043] Based on this, this specification proposes a streaming response method based on a large model. This method introduces a buffer to cache historically incompletely identified content (i.e., content in the previous chunk that was not determined to belong to semantic boundary markers). The current chunk is then concatenated with the historically incompletely identified content. By determining whether target semantic boundary markers exist in the buffer, the method accurately identifies semantic boundary markers across chunks, thereby precisely dividing the buffer content into target content with specific semantic types. This mechanism effectively avoids parsing anomalies caused by marker truncation, enabling the backend to perform semantic separation of content at the source and push target content with semantic type identifiers to the frontend. The frontend can then perform differentiated rendering (e.g., folding, hiding, or displaying thought content in a special style, while displaying answer content normally), thus eliminating the risk of frontend rendering jitter, errors, or displaying illegal characters.

[0044] Figure 1 This is a flowchart of a streaming response method based on a large model in an embodiment of this specification. This method can be applied to the execution engine of an agent or the application server of a large model, or to any other device, platform or device cluster with computing and processing capabilities, including steps 101-104 as shown below.

[0045] In step 101, the updated content based on the output of the large model at the current stage is obtained.

[0046] In this embodiment, during task execution, the large model typically generates output content incrementally using tokens as the basic unit, and returns the newly added updated content incrementally in a streaming manner through an interface. The updated content can correspond to text fragments formed by decoding one or more tokens. These text fragments include, but are not limited to, character fragments, sub-word fragments, word fragments, phrase fragments, or sentence segments. Specifically, the updated content can include the model-generated response, content related to the reasoning process (hereinafter also referred to as thinking content), and information related to tool calls, etc.

[0047] The current stage refers to the current incremental generation interval of the large model in the streaming output generation process. It usually corresponds to the current interface push, pull, or receive event. This interval includes the generation process of one or more tokens and directly corresponds to the newly added output content (i.e., updated content) returned in the current one-time event. In other words, the current stage is the generation state that the large model is in at any time when it is executing the task. The data generated and transmitted in this state is the updated content of the current stage.

[0048] For example, during streaming output, the text sequence generated by a large model can be divided into multiple chunks and transmitted sequentially, with each chunk carrying all or part of the updated content for the current stage.

[0049] There are multiple ways to obtain updated content, and this embodiment does not limit this. In one implementation, the server can listen to the output stream of the large model, and capture that part of the content as the updated content whenever the large model generates character fragments corresponding to several new tokens. For example, when the complete output content of the large model is "ABC", the updated content output at different stages is "A", "B" and "C" respectively.

[0050] In another implementation, the upstream large model interface or agent framework might return the cumulative output of the large model up to the current point in time. The cumulative output at the current stage refers to the complete content generated by the large model during its current execution phase up to the current time point. For example, if the complete output of the large model is "ABC", the cumulative output at different stages would be "A", "AB", and "ABC" respectively. In this case, the specific operation to obtain the updated content could include: obtaining the context update event of the large model at the current stage, which carries the cumulative output of the large model; comparing the cumulative output with the historical cumulative output, extracting the difference information between the cumulative output and the historical cumulative output, and determining this difference information as the updated content. The historical cumulative output is the content carried in the context update event of the large model at the previous stage. This implementation method will be explained in detail below.

[0051] Specifically, the server can maintain a variable `current_cumulative_content` to record historical states, representing the cumulative output content. Initially, this variable is an empty string or null. The server then listens for context update events from the large model interface or the Agent execution framework. During the large model generation process, the upstream system periodically or in an event-driven manner publishes context update events. Unlike directly pushing incremental tokens, these events carry fields containing the complete text sequence generated by the large model up to the current moment.

[0052] For example, suppose the large model is generating a final, complete answer that is "Hello, world." At a certain moment, the upstream sends the first context update event, which carries the cumulative output content as "you". At this time, the value of the historical cumulative output content is empty; At time 1, the upstream sends a second context update event, which carries the cumulative output content "Hello," while the historical cumulative output content stored in the previous time step is "You"; At time 1, the upstream sends the third context update event, which carries the cumulative output content "Hello, World". At this time, the historical cumulative output content stored in the previous time is "Hello,". This continues until the generation ends.

[0053] Next, at any given time, the currently received cumulative output is compared with the historical cumulative output stored at the previous time. Since the generation of the large model is append-only, the cumulative output is prefixed with the historical cumulative output. Therefore, the difference can be extracted through string truncation or length calculation.

[0054] The specific algorithm is as follows:

[0055] 1. Obtain the length of the historical cumulative output content, denoted as... .

[0056] 2. Obtain the length of the cumulative output content, denoted as . .

[0057] 3. If Then, extract the index from the cumulative output content. The substring from the beginning to the end. This substring is the update content (Delta Content) for the current stage.

[0058] 4. If If the current event does not involve the generation of new text, then the updated content will be empty.

[0059] Finally, the historical cumulative output content is updated. The current cumulative output content is assigned to history_cumulative_content, which serves as the comparison reference when the next context update event arrives.

[0060] For example, during the second round of processing (i.e., at the moment), in the received context update event, the cumulative output content is "Hello", while the historical cumulative output content is "Hi". By truncating "Hello" starting from index 1, the new incremental information "ello" for this time can be obtained, and then the historical cumulative output content is updated to "Hello".

[0061] By extracting the incremental information, repeated transmission and processing of already processed historical content can be avoided, improving processing efficiency. Subsequently, only this updated content is pushed to the front end to reduce network load and front-end rendering pressure.

[0062] In step 102, the updated content is input into the buffer, and it is determined whether the content in the buffer contains a target semantic boundary marker.

[0063] In this embodiment, a buffer mechanism is introduced to process streaming data. The buffer is used to cache the updated content of the current stage and the historical unrecognized content. The historical unrecognized content refers to the content in the updated content received in the previous stage that may be part of a certain semantic boundary marker (such as a thinking start marker and a thinking end marker), which may be a character sequence truncated by a segment and cannot be confirmed as a semantic boundary marker, or may only be a special character similar to the semantic boundary marker.

[0064] The target semantic boundary marker is a pre-agreed special string or tag used to distinguish content of different semantic types. For example, different types of semantic content may be answer content, thinking content, and tool call related information respectively, and the target semantic boundary marker may be a start marker and / or an end marker corresponding to different semantic types of content, for example, may be a start marker and / or a thinking end marker corresponding to the thinking content, or may be an answer start marker and / or an answer end marker corresponding to the answer content.

[0065] This embodiment does not limit the specific form of the adopted semantic boundary marker, which can be set according to the system implementation manner, the model output format or the access protocol. For example, <![CDATA[ can be used as the thinking start marker, and ]]> can be used as the thinking end marker; or use <think> and< / think> , <reasoning> and< / reasoning> and other tags as the thinking start marker and the thinking end marker respectively, and use <answer> and< / answer> as the answer start marker and the answer end marker respectively. These markers divide the output stream of the large model into different semantic regions, such as a thinking region and an answer region.

[0066] In this step, after the update content is appended to the buffer, for example, by splicing the update content after the historical unrecognized content, the system may scan the complete string in the buffer to determine whether a target semantic boundary marker exists, and if the target semantic boundary marker exists, the position of the marker may also be determined. Due to the characteristics of streaming transmission, a complete semantic boundary marker may be split by Chunks generated by network transmission or model generation. For example, when the target semantic boundary marker is <![CDATA[, the end of the previous Chunk is <![CD, and the start of the next Chunk is ATA[. If only a single Chunk is parsed independently, the target semantic boundary marker cannot be identified. In this embodiment, the buffer retains the characters <![CD (i.e., the historical unrecognized content) that were not completely matched in the previous round, and splices them with the update content of the current round to obtain <![CDATA[. At this time, after detecting the content in the buffer, it is determined that the target semantic boundary marker exists.

[0067] In addition, there is also a case where the historical unrecognized content is only a special character similar to the semantic boundary marker, for example, the special character is the same as the first half of the target semantic boundary marker. Exemplarily, the complete form of the special character is <![CDDD], which may be split during transmission, for example, the end of the previous Chunk is <![CD, and the start of the next Chunk is DD]. In this case, when the previous Chunk is received, it is difficult to determine whether the characters <![CD in the Chunk belong to the semantic boundary marker. In this embodiment, the <![CD characters can be retained through the buffer, and after the update content of the next round is obtained, they are spliced to obtain <![CDDD], so as to determine that there is no target semantic boundary marker in the content in the buffer.

[0068] In step 103, the target content with the target semantic type in the buffer is determined according to the existence of the target semantic boundary marker in the buffer.

[0069] In this step, the target content with the target semantic type in the buffer may be determined according to the identified existence of the target semantic boundary marker, so as to convert the original character stream in the buffer into data with a clear semantic type (such as "thinking content" or "answer content").

[0070] This embodiment does not limit the specific method for determining the existence of the target semantic boundary marker. For example, the string of the preset target semantic boundary marker can be matched with the content in the buffer. If the match is successful, the target semantic boundary marker is determined to exist; if the match fails, the target semantic boundary marker is determined to not exist. Alternatively, regular expressions can be used to perform pattern matching on the content in the buffer to identify whether the target semantic boundary marker exists. Furthermore, when the target semantic boundary marker conforms to a preset structured syntax, its existence can be identified through structure parsing. Finally, a preset state machine can be maintained to parse the streaming content in the buffer step by step in real time and identify whether the target semantic boundary marker exists.

[0071] This embodiment does not limit the specific method of determining target content with a target semantic type based on the presence of target semantic boundary markers. For example, it can be determined by the position and type of the target semantic boundary markers. For instance, when the output of the large model only contains thinking content and answer content, and the output order of the answer content is after the thinking content, if a thinking end marker is determined to exist in the buffer, the content before the thinking end marker in the buffer can be determined as thinking content, and the content after the thinking end marker can be determined as answer content. Alternatively, it can be determined based on historical recognition records of semantic boundary markers. For instance, when the historical recognition records show that the updated content output by the large model has already contained a start boundary marker of the target semantic type (such as a thinking start marker) but not an end marker of the end type (such as a thinking end marker), and there are no target semantic boundary markers in the buffer, the content in the buffer can all be determined as thinking content.

[0072] In one implementation, the target content with the target semantic type in the buffer can be determined by combining the current data state and the presence of the target semantic boundary marker. The current data state refers to the current output state of the large model, which corresponds to the semantic type of the content output by the large model and is dynamically determined based on the previously identified semantic boundary marker. For example, if the previously identified semantic boundary marker was a "thinking start" marker, it means that the updated content output by the large model after the "thinking start" marker is "thinking content," so the current data state is determined to be the "thinking state." Similarly, if the previously identified semantic boundary marker was an "answer start" marker, it means that the updated content output by the large model after the "answer start" marker is "answer content," so the current data state is determined to be the "answer state." Likewise, if the previously identified semantic boundary marker was a "tool call start" marker, it means that the updated content output by the large model after this marker is "tool call information," so the current data state is determined to be the "tool call state." Furthermore, the target semantic boundary marker to be detected can be different in different current data states. For example, in the "thinking" state, the target semantic boundary marker to be detected could be a "thinking end" marker, and in the "tool call" state, the target semantic boundary marker to be detected could be a "tool call end" marker.

[0073] In addition, for cases where the output of a large model only includes thinking content and reasoning content, only the thinking start marker and thinking end marker can be specified as target semantic boundary markers. All content outside the thinking boundary markers is treated as answer content, without the need to additionally identify the answer start marker and / or answer end marker (i.e., only the answer start marker and / or answer end marker are treated as ordinary semantic boundary markers, not target semantic boundary markers), in order to reduce the computational burden of the buffer. When the last identified semantic boundary marker is the thinking end marker, it means that the updated content output by the large model after the thinking end marker is the answer content, and the current data state is determined to be the answer state.

[0074] The specific processing logic can be divided into the following scenarios based on the presence of target semantic boundary markers:

[0075] Scenario 1: The target semantic boundary marker exists in the buffer.

[0076] If a complete target semantic boundary marker (e.g., an end-of-thinking marker) is detected in the buffer, its position within the buffer is determined. Then, the content preceding this marker is identified as target content with the semantic type corresponding to the current data state. For example, after a start-of-thinking marker was previously detected, the current data state switches to the thinking state. If a complete target semantic boundary marker (e.g., an end-of-thinking marker) is detected in the buffer, the content before the marker is determined to be of the thinking type. Similarly, if an answer start-of-response marker was previously detected, the current data state switches to the answer state. If a complete target semantic boundary marker (e.g., an end-of-response marker) is detected in the buffer, the content before the marker is determined to be of the answer type.

[0077] Subsequently, the current data state is updated based on the detected marker type. For example, if a "thinking start" marker is detected, the state is switched from "nothing" to "thinking"; if a "thinking end" marker is detected, the state is switched from "thinking" to "responding". Furthermore, the content following the target semantic boundary marker is processed according to the updated current data state. Specifically, the content following the target semantic boundary marker is determined to be target content with the semantic type corresponding to the current data state. For example, after switching the current data state from "thinking" to "responding", the content following the target semantic boundary marker in the buffer can be determined as the "responding" type.

[0078] Scenario 2: The buffer does not contain a complete target semantic boundary marker, but the end part matches the marker prefix.

[0079] If the target semantic boundary marker is not present in the buffer, the system further determines whether the end of the content in the buffer matches the prefix of the target semantic boundary marker. For example, if the string match fails, the system will check whether the end of the buffer matches the prefix of a target semantic boundary marker.

[0080] If a match is found, it means the currently received fragment may contain the beginning of a truncated marker. In this case, the system retains the end of the match in the buffer as historical incomplete recognition content, to be used for further matching when the next updated content arrives. For example, when the target semantic boundary is marked as... <think>when the end portion of the buffer is <th, then with <think>The prefix part is matched.

[0081] The remaining content in the buffer, excluding this part of the historical incomplete identification content, can be identified as target content with the semantic type corresponding to the current data state, since it has been determined that it does not belong to the incomplete marker, and can be pushed immediately.

[0082] Scenario 3: There is no complete target semantic boundary marker in the buffer, and the end part does not match any marker prefix.

[0083] If neither scenario one nor scenario two is satisfied, it means that all content in the current buffer does not belong to any part of the target semantic boundary marker. In this case, all content in the buffer is directly identified as target content with the semantic type corresponding to the current data state, and can be pushed to the front end, while the buffer is cleared.

[0084] For example, when the current data state is in the thinking state, if it is determined that there is no thinking end marker in the buffer, it means that the current large model is still outputting thinking content, and the content in the buffer can be determined as thinking content.

[0085] Through the above logic, the system can accurately decompose the continuous character stream in the buffer into independent semantic content. Even if the marker is split into multiple data packets, the correctness of the parsing can be guaranteed, avoiding the misinterpretation of the marker characters as the main text output, or the failure of state switching due to the omission of markers.

[0086] In step 104, the target content is pushed to the front end so that the front end can render the target content according to the target semantic type.

[0087] After determining the target content with a target semantic type, the server pushes the target content and its corresponding semantic type to the front end, so that the front end can render the target content accordingly based on its semantic type and display it to the user. Furthermore, the target content pushed to the front end can also be removed from the buffer.

[0088] In this step, the target content can be encapsulated into a corresponding event data packet according to its semantic type and pushed to the front end. The event data packet can use JSON, XML, or other agreed-upon data formats, and must at least include a type field (e.g., a "type" field) to identify the semantic type and a content field (e.g., a "content" field) to carry the specific target content. The value of the type field corresponds to the previously determined target semantic type, such as "thinking" or "answer".

[0089] After receiving the push event data packet, the frontend parses the type field and performs differentiated processing on the target content in the content field according to the preset rendering strategy. Specifically, when the type field is identified as the thinking type, the frontend can execute the first rendering strategy, such as displaying the content in a non-conspicuous way (e.g., low opacity, collapsed state, or status indicator), or filtering it out under a specific security policy, thereby avoiding direct exposure of the internal reasoning logic and maintaining the cleanliness of the interactive interface; when the type field is identified as the answer type, the frontend executes the second rendering strategy, appending this part of the content to the main dialogue display area and rendering it in real time according to a rich text format (e.g., Markdown).

[0090] Furthermore, if the front-end does not need to display content with certain semantic types, then when the target content contains content with those specific semantic types, it is not necessary to push such content to the front-end. For example, if the front-end does not need to display tool call information, then when the target content contains tool call information, this type of content can be omitted from the front-end. As another example, for internal thought content that is not allowed to be shown to the user, the server can either not send such content, convert it into a general status notification, or only send the start and end status of the thought process.

[0091] In one implementation, this embodiment may also include an exception handling mechanism. During the streaming response process, if an exception is detected, such as an external tool call failure, a large model generation timeout, a network connection interruption, or an internal model execution error, the server will immediately terminate the current streaming data push and send a standardized exception event (e.g., an ERROR event or a TOOL_ERROR event) to the frontend. The exception event carries an error code and error description. Upon receiving the exception event, the frontend will interrupt the current incremental rendering process, reset or maintain the current session state, and display a user-friendly error message. This mechanism effectively prevents frontend parsing logic blocking, page freezes, or inconsistent state issues caused by backend exceptions.

[0092] Through the above method, this embodiment transforms the original unstructured character stream into a structured event stream with clear semantic tags. This eliminates the need for the front-end to handle complex regular expression parsing and state maintenance tasks, effectively avoiding rendering errors caused by differences in front-end parsing capabilities or tag truncation, and achieving decoupling between content and display. Furthermore, by acquiring the updated content of the current large model and processing it in real-time into the buffer, there is no need to wait for a complete response to be generated before transmission, nor is it limited by a fixed time window. Once target content with the target semantic type is identified in the buffer, it can be immediately pushed to the front-end. This incremental real-time processing mechanism reduces the user-perceived response latency, achieves smooth and continuous streaming output, and improves the fluency of human-computer interaction.

[0093] Currently, after a large model or intelligent agent completes its streaming response, the resulting execution data is typically stored in a scattered manner. This includes detailed statistics on tool calls, execution time, token consumption, and other key monitoring data, making subsequent auditing, analysis, and optimization difficult. Therefore, this embodiment can further automate the collection, generation, and associated storage of multi-dimensional data generated during the session to address the problems of data dispersion, difficulty in auditing, and optimization in current technologies.

[0094] Specifically, after the streaming response process ends, or upon detecting a session termination event (such as a DONE event, connection closure, or timeout), the first step is to retrieve the dialogue logs from the session. During the streaming response process, the server records the complete interaction history between the user and the large model in real time as a dialogue log. The dialogue log includes, but is not limited to, user input information, large model output information, and metadata information. User input information may include the user's original query text, reference identifiers of uploaded multimodal data (such as images and files), etc. Large model output information may include the final answer after semantic separation, optional thought process content, and instructions and results returned by tool calls. Metadata information may include session ID, user ID, timestamp, request source, model version, etc. The dialogue log can be a structured sequence of records. For example, it can be recorded as a series of event objects arranged chronologically, each object containing information such as event type, occurrence time, and content payload. This log data is typically temporarily stored in memory or a cache for quick retrieval at the end of the session.

[0095] Then, tool call statistics and generated log summaries can be obtained. During the execution of the ReAct paradigm agent, the large model may call external tools (such as search engines, calculators, database query interfaces, code interpreters, etc.) multiple times. The server intercepts and records detailed information for each tool call at the execution engine level. Tool call statistics can be structured data obtained by aggregating and analyzing the original call records, including but not limited to call frequency, the total number of times each tool is called; call duration, i.e., the average response time, maximum response time, and total duration of each tool call; success rate / failure rate, i.e., the ratio of successful to failed tool calls, and the specific error code distribution for failures; parameter distribution, i.e., the parameter values ​​or parameter patterns of frequently used tools; and token consumption, i.e., the number of additional input / output tokens generated with each tool call. By extracting this statistical information, the unstructured execution process can be transformed into quantifiable performance indicators, facilitating subsequent evaluation of the agent's execution efficiency and the rationality of tool use.

[0096] A diary summary is a high-level overview of the core content of the current conversation, aiming to form a semi-structured memory fragment for subsequent user profile building, context recovery, or long-term memory retrieval. In this embodiment, the diary summary can be generated from a rule template or generated by asynchronously calling a large model (which can be the same model as the main task or a dedicated lightweight summary model) to perform secondary reasoning on the conversation content. The specific process can be as follows:

[0097] 1. Construct a summary prompt. Using the obtained dialogue logs (which can be anonymized to remove sensitive personal information) and key tool call information as input, construct a dedicated summary prompt. This prompt instructs the large model to act as a "recorder," extracting key tasks, main conclusions, user preferences, and important decisions from the conversation.

[0098] 2. Generate summary content. The large model generates a natural language summary of the diary based on the prompt words. For example: "The user queried the sales data for Q4 2023 and compared the growth rates in East China and South China. Finally, a visualization chart was generated, and it was suggested to pay attention to the abnormal fluctuations in East China."

[0099] 3. Extract tags and keywords. Key tags, such as "data analysis," "sales report," and "regional comparison," can be extracted from the generated summary or raw logs for categorization and indexing.

[0100] Next, the conversation logs, tool call statistics, and diary summaries are linked and stored. To achieve closed-loop data management and efficient retrieval, this embodiment links these three types of data using a unique session identifier and stores them in a database. The specific storage structure can use a relational database or a non-relational database (such as MongoDB, Elasticsearch, etc.).

[0101] By linking the original dialogue text, tool statistics during execution, and high-level semantic summaries, operations personnel or developers can fully trace the agent's execution trajectory, quickly pinpointing performance bottlenecks or logical errors. Simultaneously, tool call statistics provide data support for optimizing agent tool selection strategies. For example, if a tool call is found to be excessively time-consuming or has a high failure rate, the tool's interface can be optimized or the tool description in the prompt can be adjusted. Log summaries, as a high-level semantic representation of the conversation, occupy little storage space and are easily retrieved. The system can analyze log summaries of users' historical conversations to build dynamic user profiles (such as "this user frequently focuses on sales data" or "prefers visual displays"), thereby providing more personalized proactive services or preloading relevant context in subsequent conversations. Complete dialogue logs and operation records meet the data security and compliance auditing requirements of enterprise-level applications, ensuring that every AI decision is traceable.

[0102] Through the above steps, this embodiment realizes an automated, multi-dimensional session data archiving mechanism, which overcomes the shortcomings of traditional solutions such as scattered logs, lack of semantic summarization, and difficulty in quantitative evaluation, and provides a solid data foundation for the continuous iteration and operation of large model applications.

[0103] The following example further illustrates the solution presented in this specification. Figure 2 and Figure 3 This embodiment will explain how to use a lightweight state machine to identify cross-chunk semantic boundary markers in real time and accurately during streaming, and dynamically decompose the original character stream into independent structured event streams for front-end rendering.

[0104] Figure 2 A schematic diagram of a system architecture based on a ReAct Agent execution engine according to an embodiment of this specification is shown. This execution engine can reside on the server side. Figure 2 As shown, the execution engine mainly includes a central control component (ReActAgentExecutor), an incremental calculator (Incremental Delta Calculator), a state machine processor (ThinkContent Processor), an archiving service (SessionArchiveService), and auxiliary modules.

[0105] The overall control component receives user requests, initializes the context environment (including loading the tool list and build prompts), and starts the streaming execution loop of the large model.

[0106] Incremental calculator, configured to compare the length of the historical cumulative output content and the newly arrived cumulative output content, and extract the newly added string segment (i.e., the update content) of this time. Since the upstream large model interface usually returns the cumulative output content up to the current moment, this component records the processed cumulative length or content by maintaining a local string builder. Whenever a new context update event is received, it compares the difference between the current cumulative content and the historical cumulative output content, and intercepts the newly added update content (Delta) of this time.

[0107] State machine processor, with a built-in state machine, configured to receive the update content from the incremental calculator in real time. According to the current state (such as "answer state" or "thinking state"), the processor scans the input stream to identify preset target semantic boundary markers (for example, using <![CDATA[ as the thinking start marker and ]]> as the thinking end marker). It has the capability to process cross-chunk markers, can use a buffer to retain the character prefix that has not been completely matched (historical unrecognized content), and complete splicing and recognition when subsequent segments arrive, so as to disassemble the original character stream into independent semantic target content.

[0108] Archiving service, which is asynchronously triggered after the streaming response is completed, integrates the conversation log, tool call statistical information and the generated diary summary, and writes them into the persistent storage in association to form a complete observable data chain.

[0109] Auxiliary module, including a tool library (providing external API calling capability) and a log module, to support the actual execution and recording of the agent.

[0110] Figure 3 shows a flowchart of a large model-based streaming response method provided according to an embodiment of the present specification. As Figure 3 shown, the specific execution steps of the method are as follows:

[0111] Stage 1: Initialization and Startup (Steps S2.1 - S3.1)

[0112] S2.1: The server receives a user request, and loads necessary configuration information, including the available tool list and historical memory context.

[0113] S2.2: Send initialization events (such as INIT and MEMORY_INIT) to the frontend, so that the frontend interface is ready for reception.

[0114] S3.1: Initialize the streaming execution context by calling AgentExecutor.executeStream(), and subscribe to Flux <agentexecutecontext>data stream, which enables the execution engine to capture the incremental context state snapshots (i.e., context update events) generated by the large model in real time in a non-blocking manner. At this point, the initial data state inside the state machine processor is set to ANSWER_STATE (answer state) or another default state, and the prefix buffer to be matched is cleared.

[0115] Phase 2: Real-time parsing and pushing of streaming data (Steps S3.2 - S3.7)

[0116] This phase proceeds in a loop until the streaming response ends.

[0117] S3.2: Listen for updates of new Context fragments (content output by the large model). If there is an update, enter the processing flow; if not (i.e., the streaming ends), jump to the archiving phase.

[0118] S3.3 & S3.4 (incremental calculation): Obtain the complete output string stageOutput (cumulative output content) of the current phase. By comparing the processed length recorded in the locally cached contentBuilder, the difference delta = stageOutput.substring(lastLength) is intercepted. This delta is the new updated content generated by the large model in the current phase.

[0119] S4.0 (abnormality detection): During the execution process, if an abnormality such as a tool call error or LLM timeout is detected, the normal parsing is immediately interrupted, S4.1 is executed, and a standardized ERROR or TOOL_ERROR event is sent to the front end to prevent the front end from freezing or state inconsistency.

[0120] S3.5 (state machine parsing): If there is no abnormality, the calculated delta is input into the ThinkContentProcessor. The processor scans the delta according to the current data state (Current State) to detect the presence of the target semantic boundary marker, which is divided into the following situations:

[0121] Situation 1 (a complete target semantic boundary marker exists): If <![CDATA[ is detected under ANSWER_STATE, the content before this marker is identified as the answer type (MARKDOWN_DELTA), and the state is switched to THINKING_STATE; if ]]> is detected under THINKING_STATE, the content before this marker is identified as the thinking type (THINK_DELTA), and the state is switched back to ANSWER_STATE.

[0122] Case 2 (Cross-segment Processing): If a complete target semantic boundary marker is not found, but the end of delta matches the prefix of the target semantic boundary marker (e.g., <![CD), the prefix is stored in the Prefix Buffer, and the remaining content is processed and pushed according to the current data state. The splicing and continued matching are performed after the next delta arrives.

[0123] Case 3 (No Match): If there is neither a complete marker nor a prefix match, the entire delta is directly determined as the target content of the corresponding semantic type and pushed, and the buffer is cleared.

[0124] S3.6 (Event Encapsulation and Pushing): The parsed target content is encapsulated into a standardized event object (including fields such as eventId, timestamp, type, and payload).

[0125] If the semantic type is THINK_DELTA, the front end appends it to a hidden debug window or an accordion panel to display the thinking process in real time.

[0126] If the semantic type is MARKDOWN_DELTA, the front end appends it to the main chat interface and performs Markdown rendering to display the final answer in real time.

[0127] S3.7 (Status Update): Update the recorded length or content of the local contentBuilder to prepare for the next incremental calculation, and then return to S3.2 to continue listening.

[0128] Phase 3: Session Archiving and Data Storage (Steps S5.1 - S5.4)

[0129] When the streaming response ends (the judgment result in S3.2 is No), the archiving process is triggered:

[0130] S5.1: Trigger the archiveSession process, that is, start the archiving and persistent storage program for session data.

[0131] S5.2: Save the complete conversation log, including user queries (Query), model answers (Answer), and intermediate tool call records (ToolCalls).

[0132] S5.3: Update the frequency statistics information of tool calls for subsequent performance analysis and optimization.

[0133] S5.4: Asynchronously invoke a large model (which may be a dedicated lightweight model) to generate a daily session summary. This summary is a high-level overview of the core content of the session, facilitating subsequent retrieval and user profile construction.

[0134] The aforementioned dialogue logs, tool statistics, and diary summaries are associated and written to persistent storage (such as a database) to complete the data loop.

[0135] Through the above architecture and process, this embodiment not only achieves accurate semantic separation of large model output content and smooth front-end rendering, but also establishes a comprehensive back-end data auditing and analysis mechanism, effectively improving the maintainability and user experience of intelligent agent applications.

[0136] Figure 4 This is a schematic diagram of the streaming response device based on a large model, as described in the embodiments of this specification. This device can be applied to any device, platform, or cluster of devices with computing and processing capabilities. The device includes:

[0137] The acquisition module 41 is used to acquire the updated content based on the output of the large model at the current stage;

[0138] The buffer module 42 is used to input the updated content into the buffer and determine whether the content in the buffer contains the target semantic boundary marker. The buffer is used to cache the updated content of the current stage and the historical unidentified content. The historical unidentified content is the content in the updated content output by the large model in the previous stage that has not been determined to belong to the semantic boundary marker.

[0139] The determination module 43 is used to determine the target content with the target semantic type in the buffer based on the presence of the target semantic boundary marker in the buffer;

[0140] The push module 44 is used to push the target content to the front end so that the front end can render the target content according to the target semantic type.

[0141] In some embodiments, the acquisition module 41 is specifically used to acquire the context update event of the current stage large model, the context update event carrying the cumulative output content of the large model; compare the cumulative output content with the historical cumulative output content, and extract the difference information between the cumulative output content and the historical cumulative output content as the update content, the historical cumulative output content being the content carried in the context update event of the previous stage large model.

[0142] In some embodiments, the determining module 43 is specifically used to determine the position of the target semantic boundary marker in the buffer if a target semantic boundary marker exists in the buffer, and to determine the content before the target semantic boundary marker as the target content with the semantic type corresponding to the current data state, wherein the current data state is determined based on the semantic boundary marker identified last time.

[0143] In some embodiments, the determining module 43 is further configured to update the current data state based on the target semantic boundary marker; and to determine the content after the target semantic boundary marker as the target content with the semantic type corresponding to the current data state.

[0144] In some embodiments, the determining module 43 is specifically used to determine whether the end part of the content in the buffer matches the prefix part of the target semantic boundary mark if there is no target semantic boundary mark in the buffer; if the end part matches the prefix part of the target semantic boundary mark, the end part is retained as historical unidentified content, and the content in the buffer other than the historical unidentified content is determined as target content with the semantic type corresponding to the current data state, and the current data state is determined based on the semantic boundary mark identified last time.

[0145] In some embodiments, the determining module 43 is further configured to determine all contents in the buffer as target content with the semantic type corresponding to the current data state if the end portion does not match the prefix portion of the target semantic boundary marker, and the current data state is determined based on the semantic boundary marker identified last time.

[0146] In some embodiments, the current data state includes a thinking state and a response state. The semantic type corresponding to the thinking state is the thinking type, and the semantic type corresponding to the response state is the response type. The semantic boundary markers include a thinking start marker and a thinking end marker.

[0147] In some embodiments, the device further includes a storage module (not shown) for acquiring the dialogue logs, tool call statistics and generated diary summaries during the session after the streaming response of the large model ends; and storing the dialogue logs, tool call statistics and diary summaries together.

[0148] This specification also provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed in a computer, it causes the computer to perform the method described in any of the above embodiments.

[0149] This specification also provides a computing device, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, it implements the method described in any of the above embodiments.

[0150] This specification also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the method described in any of the above embodiments.

[0151] In some cases, the actions or steps described in the claims can be performed in a different order than that shown in the embodiments and still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0152] Those skilled in the art will understand that one or more embodiments of this specification can be provided as a method, system, or computer program product. Therefore, one or more embodiments of this specification may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, one or more embodiments of this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0153] One or more embodiments of this specification can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a particular task or implement a particular abstract data type. One or more embodiments of this specification can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0154] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, system embodiments are basically similar to method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. In the description of this specification, the terms "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this specification. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described can be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0155] The above are merely embodiments of one or more embodiments of this specification and are not intended to limit the scope of this specification. Various modifications and variations can be made to the one or more embodiments of this specification by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of the claims.< / agentexecutecontext> < / think> < / think> < / think>

Claims

1. A streaming response method based on a large model, comprising: Obtain the updated content based on the output of the large model at the current stage; The updated content is input into the buffer, and it is determined whether the content in the buffer contains a target semantic boundary marker. The buffer is used to cache the updated content of the current stage and the historical unidentified content. The historical unidentified content is the content in the updated content output by the large model in the previous stage that was not determined to belong to the semantic boundary marker. Based on the presence of the target semantic boundary marker in the buffer, the target content with the target semantic type in the buffer is determined; The target content is pushed to the front end so that the front end can render the target content according to the target semantic type.

2. The method according to claim 1, wherein, The process of obtaining the updated content based on the output of the large model at the current stage includes: Obtain the context update event of the large model at the current stage, wherein the context update event carries the cumulative output content of the large model; The cumulative output content is compared with the historical cumulative output content, and the difference information between the cumulative output content and the historical cumulative output content is extracted as the update content. The historical cumulative output content is the content carried in the context update event of the previous stage of the large model.

3. The method according to claim 1, wherein, Based on the presence of the target semantic boundary marker in the buffer, the target content with the target semantic type in the buffer is determined, including: If the target semantic boundary marker exists in the buffer, the position of the target semantic boundary marker in the buffer is determined, and the content before the target semantic boundary marker is determined as the target content with the semantic type corresponding to the current data state. The current data state is determined based on the semantic boundary marker identified last time.

4. The method according to claim 3, wherein, The method further includes: The current data state is updated based on the target semantic boundary marker; The content following the target semantic boundary marker is determined as the target content with the semantic type corresponding to the current data state.

5. The method according to claim 1, wherein, The step of determining the target content with the target semantic type in the buffer based on the presence of the target semantic boundary marker in the buffer includes: If the target semantic boundary marker is not present in the buffer, then it is determined whether the end of the content in the buffer matches the prefix of the target semantic boundary marker. If the last part matches the prefix part of the target semantic boundary marker, the last part is retained as historical incomplete identification content, and the content in the buffer other than the historical incomplete identification content is determined as target content with the semantic type corresponding to the current data state. The current data state is determined based on the semantic boundary marker identified last time.

6. The method according to claim 5, wherein, The method further includes: If the last part does not match the prefix part of the target semantic boundary marker, then all the contents in the buffer are determined as target content with the semantic type corresponding to the current data state, based on the current data state. The current data state is determined based on the semantic boundary marker identified last time.

7. The method according to any one of claims 3-6, wherein, The current data state includes a thinking state and a response state. The semantic type corresponding to the thinking state is the thinking type, and the semantic type corresponding to the response state is the response type. The semantic boundary markers include a thinking start marker and a thinking end marker.

8. The method according to claim 1, wherein, The method further includes: After the streaming response of the large model ends, obtain the dialogue logs, tool call statistics and generated diary summaries during the session; The dialogue logs, tool call statistics, and diary summaries are stored together.

9. A streaming response device based on a large model, comprising: The acquisition module is used to acquire the updated content based on the output of the large model at the current stage; A buffer module is used to input the updated content into a buffer and determine whether the content in the buffer contains a target semantic boundary marker. The buffer is used to cache the updated content of the current stage and the historical unidentified content. The historical unidentified content refers to the content in the updated content output by the large model in the previous stage that was not determined to belong to the semantic boundary marker. The determination module is used to determine the target content with the target semantic type in the buffer based on the presence of the target semantic boundary marker in the buffer; The push module is used to push the target content to the front end, so that the front end can render the target content according to the target semantic type.

10. A computing device comprising a memory and a processor, wherein the memory stores executable code, and the processor, when executing the executable code, implements the method of any one of claims 1-8.

Images