Method for stream output of semi-structured response by intelligent agent
By using the general Json format and adaptive json analysis algorithm in the agent, dynamically analyzing the semi-structured response body output from the large model, solving the limited scope of application, high cost and delay of the semi-structured response of the agent stream output, the application range of the agent is limited, the cost and the delay of the semi-structured response of the agent flow output is solved. This problem has achieved richer data processing and low latency response effects.
Patent Information
- Application Number
- CN202410960936.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-17
- Publication Date
- 2024-11-12
AI Technical Summary
The existing methods of agent flow output semi-structured responses have problems such as limited scope of application, high implementation costs, large response delays, and possible inconsistent results.
The general Json format is used as the semi-structured data mode, and the non-complete json structured response body output by predefined streaming response body and adaptive json analysis algorithm are dynamically parsed to realize the semi-structured response of streaming output.
It significantly increases the richness of output information, supports a better personalized experience, reduces complexity, and maintains low latency response, and is suitable for a wide range of intelligent application scenarios.
Smart Images

Figure CN118939837A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a method for an intelligent agent to stream-output semi-structured responses. Background Art
[0002] At present, the methods of streaming text response output by agents are divided into: (1) outputting unstructured responses, which usually requires the large language model to output unformatted text. The agent does not need to perform additional parsing and can directly output it in a streaming manner; (2) outputting semi-structured responses. A common method is to require the large model to output all semi-structured information at once. The output adopts the more common Markdown format. The agent usually does not perform additional parsing and directly outputs it in a streaming manner. It is usually used in simple text display scenarios; another common method is that for the same prompt word, the agent requires the large model to output a part of the target result in an unstructured manner multiple times, parses each output in a streaming manner, dynamically assembles and constructs the target result, and then performs programmatic processing. Finally, it outputs a response that meets the requirements of a specific semi-structured mode in a streaming manner; (3) outputting structured responses, which usually requires the large model to output a response with a custom structure. The agent needs to perform additional parsing and finally outputs a response that meets the requirements of a specific structured mode in a streaming manner.
[0003] For outputting unstructured streaming responses, the intelligent agent requires that the large model output is usually unformatted text, which is usually only applicable to single text response scenarios and cannot meet the requirements of programmatic processing scenarios for (semi-)structured data.
[0004] For outputting structured streaming responses, the intelligent agent usually requires the large model to output a custom format, which lacks universality. Since the proportion of similar custom format data in the training process of the large model is much smaller than that of common unstructured data such as Markdown, JSON, and HTML, the large model is prone to hallucinations when outputting custom structure responses, resulting in the output structure or content cannot be correctly parsed. It is not suitable for widespread use and is only suitable for special customized situations.
[0005] For outputting semi-structured streaming responses, (1) one method is for the agent to require the large model to output the common Markdown format, which is more versatile and easy to parse in a streaming manner. (2) Another common method is for the agent to require the large model to output a part of the target result in an unstructured manner for the same prompt word multiple times, and dynamically assemble and splice to construct the target result that meets the requirements of the semi-structured mode.
[0006] As for the first method, since it only performs simple formatting on the text, its applicable scenarios are limited and its shortcomings are obvious. (a) The syntax is simple and cannot support relatively more complex and personalized text display requirements. It is usually only applicable to scenarios where simple style streaming display is used in browsers, etc.; (b) Since it defines its syntax format for text editing scenarios, it does not support the expression of multi-layer nested relationships, etc., and its ability to express data structure relationships is relatively limited. The process of parsing and mapping to suit programmatic processing is complicated, and it is difficult to meet the requirements of programmatic processing scenarios for (semi-)structured data;
[0007] The second method has obvious disadvantages: (a) due to the illusion of the large model itself, requiring the large model to output part of the target result multiple times for the same prompt word may cause the semantic inconsistency of the contents of the final spliced result, which is not as expected and difficult to optimize and control; (b) requiring the large model to output part of the target result multiple times for the same prompt word, the number of tokens consumed and the delay in outputting the final result are proportional to the number of times the large model is called. The overall processing cost and delay cost are huge, and it lacks practicality. Summary of the invention
[0008] The technical problem to be solved by the present invention is to overcome the defects of the prior art, such as limited scope of application, high implementation cost, large response delay and possible inconsistency in result content, and to provide a method for an intelligent agent to stream output semi-structured responses.
[0009] The present invention solves the above technical problems through the following technical solutions:
[0010] The present invention provides a method for an agent to stream output a semi-structured response, the method comprising the following steps:
[0011] Step 1: predefine the streaming response body, predefine the calling method of the intelligent agent, and obtain the streaming output of the corresponding JSON structure field;
[0012] Step 2: Dynamically adapt to the large model streaming output fragments. The intelligent agent adaptively parses the incomplete JSON structure response body output during the large model streaming output process, dynamically parses the correct content, and fills it back into the predefined JSON response body to achieve streaming output semi-structured response;
[0013] Step 3: The agent dynamically streams the semi-structured response, and the agent performs streaming output.
[0014] The specific operation steps of step one are:
[0015] A1. Use the common lightweight Json format as the semi-structured data mode, combine the needs of specific scenarios, define the semi-structured response body of the agent's streaming output, and define the complete json output structure;
[0016] A2. Define the output requirements of the large model prompt words. In the response output part of the prompt words, require the large model to output the response content in the predefined json format.
[0017] The process of step 2 is to initialize the streaming response body variable semi_structured_response according to the predefined semi-structured JSON streaming response body, construct a complete JSON response body, and set the predefined JSON field values in the response body structure to corresponding null values. Initialize the accumulation variable llm_response_content of the large model streaming output content and set its value to an empty string.
[0018] The specific operation steps of step 2 are:
[0019] B1. The agent iteratively obtains the incremental content of a streaming output from the large model;
[0020] B2. The agent adds the acquired incremental content to llm_response_content;
[0021] B3. The agent calls the adaptive JSON parsing algorithm to parse all the content that has been output by the large model. If it can be parsed correctly dynamically, it will be stored in cur_semi_structured_response and enter step B4, otherwise it will enter step B1;
[0022] B4. Push the parsing results to the agent to output a semi-structured response module in a streaming format.
[0023] The operating steps of the adaptive JSON parsing algorithm are:
[0024] C1. Parse the currently input unparsed string according to the standard JSON syntax requirements and try to parse out a JSON structure element. If the parsing is successful, continue to iterate and execute step C1. If the parsing is unsuccessful, proceed to step C2.
[0025] C2. Stop parsing and return a json structure object containing all successfully parsed elements.
[0026] The specific operation steps of step three are:
[0027] D1. Compare the JSON result parsed this time with the result parsed last time;
[0028] D2. Extract the fields with different JSON field values in the response structure and the content of the difference (increment);
[0029] D3. Copy the predefined semi_structured_response variable and update the different fields and contents.
[0030] The json in step 1 is the abbreviation of "JavaScript Object Notation", which is a text-based format and is structured data containing key-value mapping, nested objects and array information.
[0031] The intelligent agent in step 1 is an intelligent entity, which is based on the cloud and takes AI as the core to build a three-dimensional perception, global coordination, accurate judgment, continuous evolution, and open intelligent system;
[0032] In the context of LLM, an agent is an agent that can autonomously understand, plan and make decisions, and perform complex tasks.
[0033] The LLM refers to a large model, which is a language model used for natural language processing. It is a pre-trained deep learning model that learns language representation and semantic understanding through large-scale text data.
[0034] The LLM model adopts the Transformer architecture, which consists of multiple encoder layers and decoder layers;
[0035] In the pre-training stage, the LLM model learns the statistical features and semantic information of the language from large-scale text data through unsupervised learning;
[0036] In the application stage, LLM models are used for various natural language processing tasks.
[0037] On the basis of being in accordance with the common sense in the art, the above-mentioned preferred conditions can be arbitrarily combined to obtain the preferred embodiments of the present invention.
[0038] The positive and progressive effects of the present invention are:
[0039] The present invention adopts the universal Json format as the structure definition of the semi-structured response of the agent streaming output, which is more convenient for programmatic processing. Compared with conventional streaming output, it can significantly increase the richness of output information and support the provision of a better personalized experience. It is a more universal method and can be widely used.
[0040] The overall implementation complexity is low, and there are no special requirements for prompt words and large models. The response effect remains the same as the output of the large model. The response delay does not increase significantly compared to the streaming output of the large model, and low latency can still be maintained. BRIEF DESCRIPTION OF THE DRAWINGS
[0041] Figure 1 The present invention is a schematic diagram of the overall process of the method for an intelligent agent to stream output a semi-structured response according to an embodiment of the present invention.
[0042] Figure 2 for Figure 1 The figure shows a flow chart of a predefined streaming response body of a method for an agent to stream output a semi-structured response.
[0043] Figure 3 for Figure 1 The fragmentary flow chart of the dynamic adaptive large model streaming output of the method for streaming output of semi-structured responses by an intelligent agent is shown.
[0044] Figure 4 for Figure 1 The figure shows a flow chart of an adaptive JSON parsing algorithm for a method in which an intelligent agent streams out semi-structured responses.
[0045] Figure 5 for Figure 1 The shown diagram is a flow chart of the method for the agent to stream output semi-structured responses.
[0046] Figure 6 for Figure 1 The figure shows a schematic diagram of the overall operation steps of the method for an intelligent agent to stream output semi-structured responses. DETAILED DESCRIPTION
[0047] The present invention is further described below by way of examples, but the present invention is not limited to the scope of the examples.
[0048] Figures 1 to 6 The structure diagram of an embodiment of the method for streaming output of semi-structured responses by an agent of the present invention is shown. The method for streaming output of semi-structured responses by an agent comprises the following steps:
[0049] Step 1: predefine the streaming response body, predefine the calling method of the intelligent agent, and obtain the streaming output of the corresponding JSON structure field;
[0050] Step 2: Dynamically adapt to the large model streaming output fragments. The intelligent agent adaptively parses the incomplete JSON structure response body output during the large model streaming output process, dynamically parses the correct content, and fills it back into the predefined JSON response body to achieve streaming output semi-structured response;
[0051] Step 3: The agent dynamically streams the semi-structured response, and the agent performs streaming output.
[0052] The present invention constructs an intelligent agent that supports streaming output of semi-structured responses based on a basic large model. The intelligent agent adopts the universal Json format as a semi-structured mode, dynamically adapts to the incomplete semi-structured fragment content output during the streaming output of the large model, and realizes streaming output of semi-structured responses.
[0053] In intelligent agent application scenarios for end users, the response speed of the intelligent agent has a very important impact on user experience satisfaction and user stickiness. The present invention mainly solves the obvious defects of the above existing methods in realizing intelligent agent streaming output semi-structured responses in intelligent agent application scenarios for end users, mainly including limited scope of application, high implementation cost, large response delay and inconsistency problems that may exist in the result content. The present invention can be widely used in various intelligent agent application scenarios requiring low latency, supports intelligent agent streaming output of semi-structured information, can support more personalized data processing needs, better meet user experience, and at the same time, the implementation cost and effect are the same as those of large models when output in a non-streaming manner.
[0054] The specific operation steps of step one are:
[0055] A1. Use the common lightweight Json format as the semi-structured data mode, combine the needs of specific scenarios, define the semi-structured response body of the agent's streaming output, and define the complete json output structure;
[0056] A2. Define the output requirements of the large model prompt words. In the response output part of the prompt words, require the large model to output the response content in the predefined json format.
[0057] The agent ensures that each response is output in a streaming manner according to the complete response body structure. Each output is the incremental content of the corresponding JSON field. The caller of the agent parses the response according to the predefined response body structure, obtains the streaming output of the corresponding JSON structure field, and then proceeds to the next step of programmatic data processing.
[0058] The process of step 2 is to initialize the streaming response body variable semi_structured_response according to the predefined semi-structured JSON streaming response body, construct a complete JSON response body, and set the predefined JSON field values in the response body structure to corresponding null values. Initialize the accumulation variable llm_response_content of the large model streaming output content and set its value to an empty string.
[0059] The intelligent agent adaptively parses the incomplete JSON structure response body output during the streaming output of the large model, dynamically parses the correct content, and backfills it into the predefined JSON response body to achieve streaming output of semi-structured response.
[0060] The specific operation steps of step 2 are:
[0061] B1. The agent iteratively obtains the incremental content of a streaming output from the large model;
[0062] B2. The agent adds the acquired incremental content to llm_response_content;
[0063] B3. The agent calls the adaptive JSON parsing algorithm to parse all the content that has been output by the large model. If it can be parsed correctly dynamically, it will be stored in cur_semi_structured_response and enter step B4, otherwise it will enter step B1;
[0064] B4. Push the parsing results to the agent to output a semi-structured response module in a streaming format.
[0065] The operating steps of the adaptive JSON parsing algorithm are:
[0066] C1. Parse the currently input unparsed string according to the standard JSON syntax requirements and try to parse out a JSON structure element. If the parsing is successful, continue to iterate and execute step C1. If the parsing is unsuccessful, proceed to step C2.
[0067] C2. Stop parsing and return a json structure object containing all successfully parsed elements.
[0068] Adaptive JSON parsing algorithm: mainly based on the optimization of the standard JSON parsing algorithm, adding adaptive fault tolerance processing. The main improvement is: when parsing the string to be parsed according to the JSON standard syntax, if it is found that the complete elements that meet the JSON syntax requirements cannot be parsed, the parsing will be stopped immediately, and a JSON object containing all successfully parsed elements will be returned, instead of returning parsing failure.
[0069] The specific operation steps of step three are:
[0070] D1. Compare the JSON result parsed this time with the result parsed last time;
[0071] D2. Extract the fields with different JSON field values in the response structure and the content of the difference (increment);
[0072] D3. Copy the predefined semi_structured_response variable and update the different fields and contents.
[0073] That is, a comparison is made between each successfully parsed semi-structured JSON response body cur_semi_structured_response received and the saved response body pre_semi_structured_response received last time, and the fields and contents (increments) with different JSON field values in the response structure are extracted. The predefined semi_structured_response variable is copied, and the different fields and contents are stored therein to construct the response body required for streaming output of the semi-structured response. Finally, the intelligent agent performs streaming output and sets pre_semi_structured_response = cur_semi_structured_respons.
[0074] The json in step 1 is the abbreviation of "JavaScript Object Notation", which is a text-based format and is structured data containing key-value mapping, nested objects and array information.
[0075] The intelligent agent in step 1 is an intelligent entity, which is based on the cloud and takes AI as the core to build a three-dimensional perception, global coordination, accurate judgment, continuous evolution, and open intelligent system;
[0076] In the context of LLM, an agent is an agent that can autonomously understand, plan and make decisions, and perform complex tasks.
[0077] The LLM refers to a large model, which is a language model used for natural language processing. It is a pre-trained deep learning model that learns language representation and semantic understanding through large-scale text data.
[0078] The LLM model adopts the Transformer architecture, which consists of multiple encoder layers and decoder layers;
[0079] In the pre-training stage, the LLM model learns the statistical features and semantic information of the language from large-scale text data through unsupervised learning;
[0080] In the application stage, LLM models are used for various natural language processing tasks.
[0081] The overall operation steps are as follows: S1, initializing the semi-structured response body variables of the agent streaming output;
[0082] S2. Initialize the accumulation variable llm_response_content of the large model streaming output content to an empty string;
[0083] S3, build prompt words and require the big model to output the response according to the defined JSON structure;
[0084] S4, output large model;
[0085] S5, obtaining the content of the large model streaming output in step S4;
[0086] S6, determine whether the streaming output is finished, if finished, execute step S71, if not finished, execute step S72;
[0087] S71, the procedure is terminated;
[0088] S72, adding the acquired content to llm_response_content;
[0089] S8, taking the string llm_response_content in step S72 as the input string, and adaptively parsing the json string;
[0090] S9, extracting the content to be output when the semi-structured response body is output in streaming mode;
[0091] S10, construct the response body required for streaming output of the semi-structured response, and execute S11 and S12 respectively according to the streaming output;
[0092] S11, streaming output receiver;
[0093] The streaming output data of S12 and S10 is directly returned to S5, and the execution continues in sequence until S71 is executed.
[0094] Although the specific embodiments of the present invention are described above, it should be understood by those skilled in the art that this is only for illustration and the protection scope of the present invention is defined by the appended claims. Those skilled in the art may make various changes or modifications to these embodiments without departing from the principles and essence of the present invention, but these changes and modifications all fall within the protection scope of the present invention.
Claims
1. A method for an agent to stream output a semi-structured response, characterized in that: The method comprises the following steps: Step 1: predefine the streaming response body, predefine the calling method of the intelligent agent, and obtain the streaming output of the corresponding JSON structure field; Step 2: Dynamically adapt to the fragments of the streaming output of the large model. The intelligent agent adaptively parses the incomplete JSON structure response body output during the streaming output of the large model, dynamically parses the correct content, and fills it back into the predefined JSON response body to achieve streaming output of semi-structured response; Step 3: The agent dynamically streams the semi-structured response, and the agent performs streaming output.
2. The method for streaming output of semi-structured responses by an agent as claimed in claim 1, characterized in that: The specific operation steps of step one are: A1. Use the common lightweight Json format as the semi-structured data mode, combine the needs of specific scenarios, define the semi-structured response body of the agent's streaming output, and define the complete json output structure; A2. Define the output requirements of the large model prompt words. In the response output part of the prompt words, require the large model to output the response content in the predefined json format.
3. The method for streaming output of semi-structured responses by an agent as claimed in claim 1, characterized in that: The process of step 2 is to initialize the streaming response body variable semi_structured_response according to the predefined semi-structured JSON streaming response body, construct a complete JSON response body, set the predefined JSON field values in the response body structure to corresponding empty values, initialize the accumulation variable llm_response_content of the large model streaming output content, and set its value to an empty string.
4. The method for streaming output of semi-structured responses by an agent as claimed in claim 1, characterized in that: The specific operation steps of step 2 are: B1. The agent iteratively obtains incremental content of a streaming output from the large model; B2. The agent adds the acquired incremental content to llm_response_content; B3. The agent calls the adaptive JSON parsing algorithm to parse all the content that has been output by the large model. If it can be parsed correctly dynamically, it will be stored in cur_semi_structured_response and enter step B4, otherwise it will enter step B1; B4. Push the parsing results to the agent to output a semi-structured response module.
5. The method for streaming output of semi-structured responses by an agent as claimed in claim 4, characterized in that: The operating steps of the adaptive JSON parsing algorithm are: C1. Parse the currently input unparsed string according to the standard JSON syntax requirements and try to parse out a JSON structure element. If the parsing is successful, continue to iterate and execute step C1. If the parsing is unsuccessful, proceed to step C2. C2. Stop parsing and return a json structure object containing all successfully parsed elements.
6. The method for streaming output of semi-structured responses by an agent as claimed in claim 1, characterized in that: The specific operation steps of step three are: D1. Compare the JSON result parsed this time with the result parsed last time; D2. Extract the fields with different JSON field values in the response structure and the content of the difference (increment); D3. Copy the predefined semi_structured_response variable and update the different fields and contents.
7. The method for streaming output of semi-structured responses by an agent as claimed in claim 1, characterized in that: The json in step 1 is the abbreviation of "JavaScript Object Notation", which is a text-based format and is structured data containing key-value mapping, nested objects, and array information.
8. The method for streaming output of semi-structured responses by an agent as claimed in claim 1, characterized in that: The intelligent agent in step 1 is an intelligent entity, which is based on the cloud and takes AI as the core to build a three-dimensional perception, global coordination, accurate judgment, continuous evolution, and open intelligent system; In the context of LLM, an agent is an agent that can autonomously understand, plan and make decisions, and perform complex tasks.
9. The method for streaming output of semi-structured responses by an agent as claimed in claim 8, characterized in that: The LLM refers to a large model, which is a language model used for natural language processing. It is a pre-trained deep learning model that learns language representation and semantic understanding through large-scale text data.
10. The method for streaming output of semi-structured responses by an agent as claimed in claim 9, characterized in that: The LLM model adopts the Transformer architecture, which consists of multiple encoder layers and decoder layers; In the pre-training stage, the LLM model learns the statistical features and semantic information of the language from large-scale text data through unsupervised learning; In the application stage, LLM models are used for various natural language processing tasks.
Citation Information
Cited By
Large model assistant dynamic output rectification method and system in network target range
CN119557526A
Real-time processing and displaying method for large-model streaming JSON (JavaScript Object Notation) return result
CN120162104A
A real-time processing display method for large model streaming JSON return results
CN120162104B
AI streaming output rich text real-time rendering method based on minimum rendering unit recognition
CN120763417A
AI streaming output rich text real-time rendering method based on minimum rendering unit identification
CN120763417B
