System for improving large language model tool invocation effect
Through systematic session management, task planning, and tool management, the efficiency and accuracy of tool invocation in large language models under complex tasks and multi-turn dialogue scenarios have been improved. This solves the problems of low context utilization efficiency and slow tool invocation speed in existing technologies, and achieves efficient tool invocation and inference speed.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- 亚信安全科技股份有限公司
- Filing Date
- 2026-03-19
- Publication Date
- 2026-06-12
AI Technical Summary
Large language models are inefficient in utilizing context when processing long texts or multi-turn interactions, lack planning in task execution, and have low efficiency and limited speed in calling tools, making it difficult to support efficient access and rapid response of various external tools.
It employs a collaborative approach involving a session management layer, a task planning layer, a tool management layer, a model layer, and an inference engine layer. The session management layer records and compresses contextual information, the task planning layer decomposes user requirements, the tool management layer uniformly registers external tools, the model layer performs context understanding and tool invocation decisions, and the inference engine layer achieves efficient inference and cache management.
It significantly improves tool invocation efficiency and model inference speed, enhances answer accuracy and execution precision for complex tasks, and supports the scalability of multiple tools and high-concurrency scenarios.
Smart Images
Figure CN122195528A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of large-scale language model application technology, and in particular to a system for improving the performance of large-scale language model tool calls. Background Technology
[0002] In recent years, deep learning technology has developed rapidly, and large-scale pre-trained language models (LLMs) have demonstrated outstanding performance in natural language processing, multimodal tasks, and other fields, possessing powerful language understanding and generation capabilities. They have been widely applied in scenarios such as dialogue systems, knowledge-based question answering, and automated office work. Building on this foundation, agent systems that combine LLMs as the core inference engine have emerged, enabling the execution of more complex tasks. Tool invocation is a key link for agent systems to complete complex tasks. By invoking external tools, the functional boundaries of LLMs can be expanded, realizing the implementation of practical business needs.
[0003] In real-world large-scale model product scenarios, it is often necessary to continuously track changes in the user's true intent during multi-turn dialogues. This can lead to a very large context input to the model, even exceeding the length of the context window supported by the model. In order to enable large models to help humans complete tasks, hundreds or even thousands of tools are often provided for the large models to call. Existing large language models have the following problems in the execution of complex tasks: (1) Low context utilization efficiency: When the model processes long texts or multi-turn interactions, it cannot effectively compress and extract the context, making it difficult to accurately capture the user's true intent; (2) Lack of planning in task execution: Existing technologies usually respond directly to user input, lacking systematic decomposition of user intent and task planning, resulting in unstable execution results; (3) Low tool calling efficiency: When multiple external tools or agents are involved, existing methods need to retain a large amount of redundant information in the context, making it difficult to support large-scale and diverse tool access; (4) Limited calling speed: Insufficient cache management during model inference, unable to efficiently utilize memory to store more KV-Cache, resulting in slow tool calling speed and affecting the overall system performance. Summary of the Invention
[0004] The purpose of this invention is to overcome the shortcomings of the prior art and provide a system that improves the performance of calling large language model tools.
[0005] The objective of this invention is achieved through the following technical solution: A system for improving the performance of calling large language model tools, comprising a session management layer, a task planning layer, a tool management layer, a model layer, and an inference engine layer;
[0006] The conversation management layer is used to record and maintain contextual information about user interactions with large language models, continuously tracking user input, model output, and intermediate results of tool calls to form a complete dialogue chain;
[0007] The task planning layer is used to parse and decompose user needs, transform natural language instructions into executable subtask flows, and complete the semantic relevance matching between the current problem and the tool through the embedding model, and provide tool selection results;
[0008] The tool management layer serves as the unified registration and invocation interface for the tool ecosystem, defining all external tools that can be invoked by the large language model.
[0009] The model layer consists of a large language model and an embedding model. The large language model performs contextual understanding, task planning and reasoning, tool invocation decision-making, and result generation. The embedding model is used to transform the current problem and tool description in natural language form into semantic embedding vectors, and to match the semantic relevance between the problem and the tool by calculating vector similarity.
[0010] The inference engine layer includes model API inference and a distributed cache database.
[0011] Preferably, the session management layer uses a linked list to record the interaction history and presets the session compression trigger conditions. When the length of the historical dialogue exceeds the preset threshold, a large language model is called to generate a compressed summary, retaining key semantic information.
[0012] Preferably, the embedding model uses three strategies to match and select the current problem and tools: directly selecting the tool with the highest similarity based on the semantic similarity score; selecting the top k tools with the highest semantic similarity and then having a large language model make a secondary selection; if the maximum similarity value is greater than a preset threshold, the tool is directly selected; if the maximum similarity value is less than the preset threshold, the top-k tools are selected and then a large language model makes a secondary selection.
[0013] Preferably, the model API inference of the inference engine layer uses the open-source distributed inference frameworks vllm or sglang; the distributed cache database uses the open-source distributed in-memory database etcd.
[0014] The present invention has the following advantages:
[0015] 1. This invention achieves end-to-end optimization of large-scale language model tool calls through the coordinated operation of the conversation management layer, task planning layer, tool management layer, model layer, and inference engine layer, from multiple dimensions including the input layer, inference layer, and tool management layer. While ensuring the accuracy of task execution, it achieves scalable support for multiple tools, significantly improves tool call efficiency and model inference speed, and is suitable for large-scale language model application scenarios with high concurrency, multi-turn dialogue, and complex tasks.
[0016] 2. This invention performs semantic compression on earlier sessions through the session management layer. While ensuring that the core semantics are not lost, it significantly reduces the input length, thereby improving the inference speed. It allows the model to focus more on the user's current question within a limited context window, improving the accuracy of the answer and solving the information loss problem caused by traditional truncation methods.
[0017] 3. The task planning layer of this invention transforms complex tasks into executable subtask flows, improving the parallelism and controllability of task execution, solving the problem of lack of planning in task execution in the prior art, and improving the execution accuracy under complex tasks; at the same time, it combines the embedding model to match the semantic relevance between the problem and the tool, and through three tool selection strategies, it significantly reduces the reasoning cost of large language models in the tool selection stage, reduces the invocation of irrelevant tools, and improves the accuracy and efficiency of tool invocation. Attached Figure Description
[0018] Figure 1 A schematic diagram of the system architecture for improving the performance of large language model tools. Detailed Implementation
[0019] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.
[0020] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.
[0021] It should be noted that, unless otherwise specified, the embodiments and features described in this invention can be combined with each other.
[0022] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.
[0023] In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, or the orientation or positional relationship commonly used when the product of this invention is in use, or the orientation or positional relationship commonly understood by those skilled in the art. They are only used for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this invention. In addition, the terms "first," "second," etc., are only used to distinguish descriptions and should not be construed as indicating or implying relative importance.
[0024] In the description of this invention, it should also be noted that, unless otherwise explicitly specified and limited, the terms "set," "install," "connect," and "link" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.
[0025] In this embodiment, as Figure 1 As shown, a system for improving the performance of large language model tool calls includes a session management layer, a task planning layer, a tool management layer, a model layer, and an inference engine layer.
[0026] The conversation management layer is used to record and maintain contextual information about user interactions with large language models, continuously tracking user input, model output, and intermediate results of tool calls to form a complete dialogue chain;
[0027] The task planning layer is used to parse and decompose user requirements, transforming natural language instructions into executable subtask flows. It uses an embedding model to match the semantic relevance of the current problem with tools and provides tool selection results. Specifically, the embedding model uses three strategies to match and select tools for the current problem: directly selecting the tool with the highest semantic similarity score; selecting the top k tools with the highest semantic similarity, followed by a secondary selection by a large language model; and directly selecting the tool if the maximum similarity score is greater than a preset threshold, and selecting the top-k tools if the maximum similarity score is less than the preset threshold, followed by a secondary selection by a large language model. This task planning layer transforms complex tasks into executable subtask flows, improving the parallelism and controllability of task execution, solving the problem of lack of planning in task execution in existing technologies, and improving execution accuracy under complex tasks. Simultaneously, by combining the embedding model to match the semantic relevance of the problem with tools and using the three tool selection strategies, it significantly reduces the inference cost of the large language model in the tool selection stage, reduces the invocation of irrelevant tools, and improves the accuracy and efficiency of tool invocation. In this embodiment, a third strategy is adopted to match and select the current problem and the tool. That is, a tool matching strategy of comprehensive threshold judgment and top-k selection is adopted. The preset similarity threshold is 0.95 and the k value is 3. If the maximum similarity between a tool and the problem is ≥0.95, the tool is directly selected for invocation. If the maximum similarity is <0.95, the top 3 tools in terms of similarity are selected. The capability descriptions and invocation parameters of these 3 tools are sent to the large language model. The large language model performs a secondary selection based on the sub-task requirements to determine the final tool to be invoked.
[0028] The tool management layer serves as the unified registration and invocation interface for the tool ecosystem, defining all external tools that can be invoked by the large language model. Specifically, the tool management layer pre-registers more than 20 external tools, such as log extraction, text analysis, report generation, and data visualization, to achieve standardized registration and management of tools.
[0029] The model layer consists of a large language model and an embedding model. The large language model performs contextual understanding, task planning and reasoning, tool invocation decision-making, and result generation. The embedding model is used to transform the current problem and tool description in natural language form into semantic embedding vectors, and to match the semantic relevance between the problem and the tool by calculating vector similarity.
[0030] The inference engine layer includes model API inference and a distributed cache database. This invention achieves end-to-end optimization of large-scale language model tool calls through the collaborative work of the session management layer, task planning layer, tool management layer, model layer, and inference engine layer, spanning multiple dimensions from the input layer, inference layer, and tool management layer. While ensuring task execution accuracy, it achieves scalable support for multiple tools, significantly improving tool call efficiency and model inference speed. It is suitable for large-scale language model applications with high concurrency, multi-turn dialogues, and complex tasks.
[0031] Furthermore, the conversation management layer uses a linked list to record the interaction history and presets conversation compression trigger conditions. When the length of the historical dialogue exceeds a preset threshold, a large language model is invoked to generate a compressed summary, preserving key semantic information. Specifically, the conversation management layer supports multi-level storage: near-field conversations retain the original text, while far-field conversations retain the summary, thus ensuring global consistency while saving computational resources. Here, near-field conversations are the conversations with the preset number of turns closest to the current question, and the rest are far-field conversations. This invention performs semantic compression on earlier conversations through the conversation management layer, significantly reducing the input length while ensuring that core semantics are not lost. This improves inference speed, allows the model to focus more on the user's current question within a limited context window, improves answer accuracy, and solves the information loss problem caused by traditional truncation methods.
[0032] Furthermore, the model API inference in the inference engine layer adopts the open-source distributed inference frameworks vllm or sglang; the distributed cache database adopts the open-source distributed in-memory database etcd. Specifically, the model API inference combines hardware acceleration and the distributed inference framework to achieve efficient inference computation, supporting cache hits, parallel decoding, and reuse of inference results; the distributed cache database is an in-memory database that caches the KV-Cache generated by the large language model during the prefilling process in real time, supports fast synchronization of KV-Cache between multiple nodes, and directly calls the cached data when the same prefix prompt word is received, and uses a hybrid storage method of video memory and in-memory to store the KV-Cache. In this embodiment, the hardware acceleration means adopt GPU or TPU hardware acceleration.
[0033] Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A system for improving the performance of calling large language model tools, characterized in that: It includes a session management layer, a task planning layer, a tool management layer, a model layer, and an inference engine layer; The conversation management layer is used to record and maintain contextual information of user interaction with large language models, and continuously track user input, model output and intermediate results of tool calls to form a complete dialogue chain; The task planning layer is used to parse and decompose user needs, transform natural language instructions into executable subtask flows, and complete the semantic relevance matching between the current problem and the tool through the embedding model to provide tool selection results; The tool management layer is the unified registration and calling interface for the tool ecosystem, defining all external tools that can be called by the large language model; The model layer consists of a large language model and an embedding model. The large language model performs context understanding, task planning and reasoning, tool invocation decision-making, and result generation. The embedding model is used to transform the current problem and tool description in natural language form into semantic embedding vectors, and to match the semantic relevance between the problem and the tool by calculating vector similarity. The inference engine layer includes model API inference and a distributed cache database.
2. The system for improving the invocation effect of large language model tools according to claim 1, characterized in that: The session management layer uses a linked list to record the interaction history and presets session compression trigger conditions. When the length of the historical dialogue exceeds a preset threshold, a large language model is called to generate a compressed summary, retaining key semantic information.
3. The system for improving the invocation effect of large language model tools according to claim 2, characterized in that: The embedding model uses three strategies to match and select the current problem and tools: directly selecting the tool with the highest similarity based on the semantic similarity score; selecting the top k tools with the highest semantic similarity and then having a large language model make a secondary selection; if the maximum similarity value is greater than a preset threshold, the tool is directly selected; if the maximum similarity value is less than the preset threshold, the top-k tools are selected and then a large language model makes a secondary selection.
4. The system for improving the invocation effect of large language model tools according to claim 3, characterized in that: The model API inference of the inference engine layer uses the vllm or sglang open-source distributed inference framework; the distributed cache database uses the etcd open-source distributed in-memory database.