A tool calling management method and system for a large language model intelligent agent
By setting deterministic checkpoints and external approval mechanisms outside the reasoning action loop of a large language model agent, the uncertainty problem of sensitive action control in existing technologies is solved, and real-time reliable control and unique approval results for sensitive actions are achieved.
Patent Information
- Application Number
- CN202611257740.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-19
- Publication Date
- 2026-11-17
AI Technical Summary
In existing technologies, large language model agents lack reliable, real-time sensitive action control during tool invocation, which can easily lead to irreversible business consequences due to uncertainty in model output and omissions or misjudgments, and lack effective external deterministic constraints.
Deterministic checkpoints are set outside the reasoning action loop of the large language model agent to determine whether the tool call request is a sensitive action, generate an approval record and block execution, and conduct manual approval through an external approval terminal to ensure the uniqueness and validity of the decision.
It enables real-time and reliable control over sensitive actions, avoids misjudgments caused by uncertainty in model output, and ensures the security and uniqueness of business operation approval results.
Smart Images

Figure CN123116265A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of artificial intelligence technology, and in particular to a tool invocation control method and system for large language model intelligent agents. Background Technology
[0002] With the rapid development of internet technology, Large Language Model (LLM) driven agents are being used more and more widely in enterprise business scenarios. These agents possess autonomous reasoning and action capabilities, and can call external tools as needed during the reasoning-action loop, such as calling business system interfaces, performing database operations, and triggering external message notifications, to complete complex business tasks.
[0003] However, during the use of this tool, there may be sensitive actions, such as making external payments, deleting business data, or sending notifications in bulk. If these sensitive actions are triggered incorrectly or maliciously, they will have irreversible business consequences and cause actual losses.
[0004] Currently, several solutions have been proposed for the security management of intelligent agents. Common solutions involve pre-setting security rules within the system, with a rule engine automatically adjudicating the operational instructions generated by the agent, or automatically analyzing control instructions based on a security constraint graph. A common characteristic of these solutions is that the decision-making body for security judgments is the machine itself; that is, the pre-set rules or graph model automatically determine and handle the agent's operations. However, the reliability of this machine-driven approach depends on the completeness and accuracy of the pre-set rules or graph model. In practical applications, pre-set rules are insufficient to cover all business scenarios, and the coverage of graph models is limited, making them prone to missed or false judgments. Furthermore, the agent's own reasoning output is uncertain. If security judgments are embedded in the reasoning process of a large language model, allowing the model to decide whether control is needed, it may be susceptible to malicious injection, leading to control failure. In other words, entrusting security judgments to the model's own reasoning essentially lacks external, deterministic constraints independent of the model's output.
[0005] In summary, existing technologies have not yet provided a complete solution for reliably and in real-time controlling an intelligent agent when it performs a single sensitive action, and for ensuring the validity and uniqueness of the control results. Therefore, there is an urgent need in this field for a new technical solution to address these issues. Summary of the Invention
[0006] To address the shortcomings of existing technologies, this application provides a tool invocation control method and system for large language model intelligent agents. The purpose is to solve or improve the technical problems in existing technologies, such as insufficient control reliability due to reliance on machine autonomous decision-making, susceptibility to model output uncertainty, difficulty in deterministic real-time control of individual sensitive actions, and lack of guarantee of the validity and uniqueness of control results.
[0007] To achieve the above objectives, the first aspect of this application provides a tool invocation control method for large language model intelligent agents, the tool invocation control method comprising:
[0008] By setting deterministic checkpoints outside the reasoning action loop of the large language model agent, the tool call request is judged before the tool call request is executed, so as to determine whether the corresponding tool action is marked as a sensitive action that requires manual approval.
[0009] When it is determined that the tool action corresponding to the tool call request requires manual approval, an approval record with a unique identifier and a pending status is generated, the execution of the tool call request is blocked, and a manual approval request containing a unique identifier is sent to the designated approval end.
[0010] In addition, it receives a decision instruction from the approval end and updates the status of the approval record from pending to one of approved or rejected by performing an atomic update operation on the approval record, so as to decide whether to continue execution or block the blocked tool call request based on the updated status.
[0011] Optionally, determine whether the corresponding tool action is marked as a sensitive action requiring manual approval, including:
[0012] Based on the sensitive action flags pre-configured for the actions of the invoked tool, determine whether approval is required;
[0013] In this process, the inference output is determined independently of the large language model agent, and therefore cannot be bypassed by prompts.
[0014] Optionally, deterministic checkpoints employ a strategy of blocking upon execution failure, directly intercepting tool call requests when an exception occurs at a deterministic checkpoint.
[0015] Optionally, an approval record with a unique identifier and a pending status is generated, including:
[0016] The approval record records the invoked tools and actions, the organization to which they belong, and the approval qualification information. The approval qualification information is used to limit the scope of personnel with decision-making authority. This scope is configured as a set of one or more identity types selected from the initiator, the agent owner, and the organization member.
[0017] Optionally, the method further includes: before receiving the adjudication instruction from the approval end, determining whether the approving user submitting the adjudication belongs to the personnel scope limited by the approval qualification information in the approval record, and rejecting submissions that do not belong to the scope.
[0018] Optionally, a manual approval request containing a unique identifier is sent to a designated approval authority, including:
[0019] Select the approval channel based on the approval routing pattern pre-configured for the large language model agent;
[0020] The approval routing modes include:
[0021] In the channel-only mode, manual approval requests are sent only through external instant messaging channels.
[0022] Inline mode only, sending manual approval requests only within the dialogue interface interacting with the large language model agent;
[0023] In automatic mode, manual approval requests are sent via external instant messaging channels first. If no external instant messaging channels are available, the system will revert to sending manual approval requests within the chat interface.
[0024] Optionally, when the approval routing mode is channel-only or automatic, a manual approval request containing a unique identifier is sent to the designated approval end, specifically including:
[0025] Construct an approval card that includes approval and rejection interaction buttons, with the return value of the interaction button carrying a unique identifier of the approval record;
[0026] Based on the external instant messaging channel, the approval card is converted into a message format compatible with that channel, and then pushed to the instant messaging group of the approval user associated with the approval end through the channel's open interface.
[0027] Optionally, after the execution of the blocking tool invocation request, the method further includes:
[0028] In a non-busy waiting mode, the status of the approval records is polled at set time intervals until the status is updated to the final state of approval or rejection, or the preset timeout period is reached.
[0029] When the timeout period is reached and no ruling is received, the status of the approval record is set to expired, and the tool call request is blocked.
[0030] Optionally, an adjudication instruction may be received from the approval authority, including at least one of the following two sources:
[0031] When the adjudication instruction originates from the dialogue interface interacting with the large language model agent, the adjudication instruction is received after the user's identity has been authenticated by the system.
[0032] When an adjudication order originates from a callback notification via an external instant messaging channel, the callback notification is received after verifying the authenticity of its source and that it has not been tampered with by signing the content of the notification.
[0033] Optionally, the method further includes: encrypting and storing the channel credentials required to access the channel before communicating with the external instant messaging channel;
[0034] When it is necessary to call the channel's open interface, decrypt the channel credentials and use them.
[0035] Optionally, performing an atomic update operation on the approval record includes:
[0036] When a decision instruction is received from the approval end, check the current status of the approval record;
[0037] The decision will be considered a valid decision and the status of the approval record will be updated to approved or rejected only if the current status is still pending.
[0038] If the current status has been updated to approval or rejection, the ruling instruction shall be ignored as an invalid ruling;
[0039] This ensures that the same approval record is updated only by one valid ruling, thus achieving deduplication of concurrent rulings from different sources.
[0040] Optionally, after the status of the approval record is updated to approved or rejected, the method further includes:
[0041] The approval record uniformly records the adjudication result, adjudication time, and adjudicator identifier; for adjudications from the chat interface, the recorded adjudicator identifier is the authenticated user identifier; for adjudications from external instant messaging channels, the recorded adjudicator identifier is the channel user identifier.
[0042] Optionally, the method further includes: generating a blocking description when the blocking tool is invoked according to the updated state, and injecting the blocking description as observation information back into the context of the large language model agent;
[0043] When the tool call request continues to be executed based on the updated state, the result of the execution is injected back into the context of the large language model agent as observation information.
[0044] A second aspect of this application provides a tool invocation control system for large language model intelligent agents, the system comprising:
[0045] The deterministic check module is used to judge the tool call request before it is executed to determine whether the corresponding tool action is marked as a sensitive action that requires manual approval.
[0046] The approval record management module is used to generate approval records with unique identifiers and pending status when it is determined that a tool call request requires manual approval.
[0047] The blocking and waiting module is used to block the execution of tool call requests and wait for the status update of the approval record;
[0048] The approval request sending module is used to select an approval channel based on the approval routing mode pre-configured for the large language model agent, and send a manual approval request containing a unique identifier to the designated approval end.
[0049] The atomic decision module is used to receive decision instructions from the approval end and update the status of the approval record from pending to one of approved or rejected by performing atomic update operations on the approval record;
[0050] The blocking and waiting module decides whether to continue execution or intercept the blocked tool call request based on the updated state from the atomic decision module.
[0051] A third aspect of this application provides an electronic device including a memory and a processor, wherein the memory stores a computer program and the processor is configured to implement the method as described in any of the first aspects when executing the computer program.
[0052] The fourth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method as described in any one of the first aspects.
[0053] This application has the following beneficial effects:
[0054] This application establishes deterministic checkpoints outside the reasoning action loop of a large language model agent, making the judgment of sensitive actions independent of the model output, thus avoiding control failures caused by prompt injection or model bias. By blocking tool calls and generating approval records and sending approval requests to the approval end, manual approval is introduced into the tool call chain, realizing real-time human-in-the-loop control of individual sensitive actions. By using atomic update operations to adjudicate approval records, the uniqueness of the adjudication results is guaranteed, avoiding the risk of concurrent duplicate adjudication.
[0055] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application, it can be implemented according to the contents of the specification. In order to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description
[0056] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the embodiments of this application or the background art will be described below.
[0057] Figure 1 This is a flowchart illustrating the steps of the tool call control method for large language model intelligent agents provided in the embodiments of this application;
[0058] Figure 2 This is an architectural block diagram of a tool call management system for large language model intelligent agents provided in an embodiment of this application;
[0059] Figure 3 This is a schematic diagram of the overall process of the tool call control method for large language model intelligent agents provided in the embodiments of this application. Detailed Implementation
[0060] The specific embodiments of this application will now be described in detail with reference to the accompanying drawings. It should be noted that the described embodiments are only a part of the embodiments of this application, and not all of them. These embodiments are provided to make the disclosure of this application clearer and more complete, so that those skilled in the art can understand and implement this application, and should not be construed as limiting the scope of protection of this application. In all the drawings, the same reference numerals indicate the same or similar parts or elements.
[0061] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. Unless the context clearly indicates otherwise, the singular forms “a,” “an,” and “the” used herein are also intended to cover the plural forms. For example, reference to “a component” refers not only to a single entity but also to two or more entities of that type. The term “and / or” is intended to include any and all possible combinations of one or more of the associated listed items.
[0062] Furthermore, the terms “comprising,” “including,” or any other variations thereof as used herein are intended to cover non-exclusive inclusion. This means that a system, apparatus, or method that includes a list of features, steps, operations, elements, or components includes not only those elements expressly listed but also other elements not expressly listed or inherent to the system, apparatus, or method. Unless otherwise specified, an element defined by the phrase “comprising one…” does not exclude the presence of other identical elements in the system or method that includes that element.
[0063] The numbering of the method steps in this application specification is only used to distinguish different operations or states to make the description clearer, and does not necessarily constitute a limitation on the order of their execution.
[0064] To facilitate understanding of the technical solutions in this application, the following explanations are provided for some of the technical terms and jargon used in this document:
[0065] A large language model intelligent agent refers to an intelligent system that uses a large language model as its core reasoning engine and possesses autonomous reasoning and action capabilities. It can autonomously plan tasks in the reasoning-action cycle, invoke external tools as needed, and perform the next round of reasoning based on the tool's execution results.
[0066] The reasoning action loop refers to the autonomous workflow of a large language model agent: "reasoning, calling tools, obtaining results, and then reasoning again." The deterministic checkpoints in this application are set outside this loop, i.e., intercepted at the tool call distribution point, and do not participate in the model's reasoning process.
[0067] Deterministic checkpoints are interception points set outside the inference action loop of a large language model and before tool calls are executed. Their determination of whether a tool call is a sensitive action is independent of the model output and cannot be bypassed by hint injection or model bias.
[0068] Sensitive actions refer to tool actions that will have irreversible business consequences after execution, such as making external payments, deleting business data, and sending notifications in bulk. Sensitive actions must be processed through the control mechanism before they are executed.
[0069] An atomic update operation refers to a single, indivisible update to the status of an approval record. This operation is only executed when the current status of the approval record is pending, ensuring that only one of the concurrent decisions from multiple sources actually takes effect.
[0070] Approval routing mode refers to the strategy pre-configured for the agent that determines which channel the approval request will be sent through. This includes channel-only mode (sent only through external instant messaging channels), inline-only mode (sent only within the dialog interface), and automatic mode (prioritizes sending through external instant messaging channels, and reverts to the dialog interface if unavailable).
[0071] External instant messaging (IM) channels refer to instant messaging platforms used by enterprises on a daily basis, such as Lark, Slack (a team collaboration and instant messaging platform), Microsoft Teams, WeChat Work, and DingTalk. Approval requests can be pushed through these channels in the form of interactive cards, and the approving user can complete the processing directly in the group.
[0072] The following embodiments are only used to illustrate the technical solutions of this application more clearly, and are therefore only examples and should not be used to limit the scope of protection of this application.
[0073] For the security management of tool calls by large language model agents, the above-mentioned solutions include automatic adjudication through rule engines or constraint graphs. However, these solutions still have shortcomings in handling real-time control of sensitive actions. Specifically:
[0074] First, if the security judgment of tool calls is left to the large language model to handle by prompt words, the judgment will fail due to the uncertainty of the model output. It can be bypassed by prompt injection or model bias, and cannot form deterministic control.
[0075] Second, automatic adjudication schemes based on predefined rules or constraint graphs rely on machines for decision-making. When encountering edge scenarios where the rules are not covered or the graph boundaries are unclear, they are prone to missed or incorrect judgments.
[0076] Third, the existing solution does not provide a closed-loop mechanism that blocks a synchronous tool call, sends an approval request through an external instant messaging channel, waits for asynchronous processing, and then allows or blocks the request based on the result.
[0077] Fourth, when the processing instructions may come from both the dialogue interface and external instant messaging channels, there is a lack of a mechanism to deduplicat the same approval request and uniformly record the processor, processing time and processing result, which can easily lead to concurrent duplicate processing and fragmented audit records.
[0078] Therefore, the first aspect of this application provides a tool call control method for large language model agents. Outside the reasoning action loop of the large language model agent, a deterministic interception point is set at the tool call distribution point to control sensitive actions in a way that cannot be bypassed by prompts. When approval is required, the tool call is blocked, an approval record is generated, and an approval request is sent to the approval end. Finally, the decision result is uniformly processed through atomic update operations. Specifically, such as... Figure 1 As shown, the method includes the following steps:
[0079] S101: By setting a deterministic checkpoint outside the reasoning action loop of the large language model agent, the tool call request is judged before the tool call request is executed to determine whether the corresponding tool action is marked as a sensitive action that requires manual approval.
[0080] Specifically, deterministic checkpoints are set outside the inference-action loop of the large language model agent, at the tool invocation distribution point. When the agent constructs a tool invocation request in the inference-action loop, and the request arrives at the tool invocation distribution point ready for execution, the deterministic checkpoint runs before the actual execution of the tool invocation. The checkpoint determines whether the invoked tool action requires approval based on pre-configured sensitive action flags. This determination is independent of the large language model agent's inference output; that is, the sensitive action flags are pre-configured and do not rely on the model's own determination of approval requirements during inference. Therefore, this determination cannot be bypassed by hints.
[0081] If the judgment result is that no approval is required, the tool call request is directly allowed, executed normally, and the result is returned.
[0082] S102: When it is determined that a tool call request requires manual approval, an approval record with a unique identifier and a pending status is generated, the execution of the tool call request is blocked, and a manual approval request containing a unique identifier is sent to the designated approval end.
[0083] Specifically, when a deterministic checkpoint determines that the tool action corresponding to the current tool call request is a sensitive action requiring approval, the system generates an approval record. This approval record has a unique identifier and a pending status, used to uniquely identify this approval request and serving as the status carrier for the entire subsequent approval process. Simultaneously, the current synchronous tool call request is blocked; that is, the tool call is temporarily suspended, awaiting approval before a decision is made to allow or block it based on the ruling. Subsequently, a manual approval request is sent to the designated approval end, carrying the unique identifier of the approval record, so that the approval end can make a ruling on this approval request accordingly.
[0084] S103: Receive the decision instruction from the approval end, and update the status of the approval record from pending to one of approved or rejected by performing an atomic update operation on the approval record, so as to decide whether to continue execution or block the blocked tool call request based on the updated status.
[0085] Specifically, the system receives a ruling instruction from the approval end, which can instruct the system to approve or reject the current tool call. The system performs atomic update operations on the approval record. The principle of this operation is that the status of the approval record is updated to approved or rejected only if the current status of the approval record is still pending; if the current status has already been updated to approved or rejected (for example, it has been updated by a ruling from another source), then the current update is invalid. This ensures that the same approval record is updated by only one valid ruling.
[0086] After the update is completed, the subsequent processing of the tool call is determined based on the final status of the approval record: if the status is approved, the blocking is lifted and the blocked tool call request continues to be executed; if the status is rejected, the tool call request is intercepted and the sensitive action is not executed.
[0087] The main steps of the method described above are presented in this application. The specific implementation of the above method will be further explained in detail below, taking into account the system architecture.
[0088] The method described in this application can be executed by a tool that invokes a control system. For example... Figure 2 As shown, the system includes a deterministic check module, an approval record management module, a blocking and waiting module, an approval request sending module, and an atomic decision module. It should be noted that... Figure 2 To clearly illustrate the process of sending and receiving approval requests, the functions of approval routing, approval card construction, message format conversion and push, channel callback signature verification, and encrypted storage of credentials are shown in the diagram under module names such as Approval Routing Module, Channel Interaction Card Module, and IM Channel Adaptation Layer. For simplicity, [the remaining text is incomplete and requires further context]. Figure 2 The external instant messaging channel adaptation layer is referred to as the IM channel adaptation layer, and IM channels and external instant messaging channels have exactly the same meaning. At the system level, all the above functional steps are uniformly carried out by the approval request sending module in this application system. The collaborative workflow of each module is as follows:
[0089] In one embodiment, the large language model agent, in its reasoning-action loop, first performs large model reasoning and tool selection, generates a tool invocation request, and sends the request to the tool invocation dispatcher.
[0090] Before executing the request, the tool invocation dispatch center intercepts and judges it through a deterministic check module. This deterministic check module is set outside the reasoning-action loop of the large language model agent, and its judgment logic is independent of the reasoning output of the large language model, thus forming a deterministic checkpoint. In a specific implementation, this deterministic check module can be deployed as an out-of-loop deterministic hook runtime. Here, "hook" can be understood as a hook or hook, referring to an interception point pre-embedded in the normal execution flow of the program.
[0091] In one embodiment, the deterministic inspection module can maintain a Hook registry internally, which is used to organize multiple hooks according to tool filtering rules and priorities. Specifically, the hook registry allows the system to organize and manage multiple hooks with different responsibilities in a modular manner. It schedules them uniformly according to the tool filtering rules and preset priorities configured for each hook. For example, the system may need a hook specifically for determining whether manual approval is required, as well as a hook for audit logging; the tool filtering rules can limit the previous hook to be triggered only when a specific tool name or action pattern is matched, while the audit hook applies to all tool calls; the priority can stipulate that the approval hook must be executed before the audit hook, or that the audit hook still needs to be executed to complete the logging after the approval hook makes an interception decision.
[0092] Among the multiple hooks managed by the Hook registry, the deterministic approval hook is responsible for performing the approval judgment. When scheduled for execution by the Hook registry, this deterministic approval hook determines whether the action requires approval based on the pre-configured sensitive action flags of the called tool's action. The entire judgment process is independent of any output generated by the large language model agent during the inference phase, and therefore cannot be bypassed by hints.
[0093] If the judgment result is that no approval is required, the tool call request is directly approved, the tool call distribution point executes the tool call normally, and returns the result to the large language model agent.
[0094] If the determination result indicates that approval is required, the tool call request and related information are passed to subsequent modules for processing.
[0095] Furthermore, this deterministic check module employs a security strategy of blocking execution upon failure. This means that when any anomaly occurs during the operation of this deterministic check module—whether it's a failure to read the configuration of sensitive action flags, an error in the scheduling logic of the Hook registry, or an unexpected internal failure encountered during the determination process of the deterministic approval Hook—the deterministic check module will not choose the lenient approach of allowing the request to proceed to ensure uninterrupted business operations, but will directly block the current tool call request. This strategy ensures that a failure in the control link itself will not become a breakthrough in the entire security system, avoiding the risk that sensitive actions will be executed without approval due to checkpoint failure.
[0096] When the deterministic check module determines that the current tool call request requires approval, the request is not executed immediately but is blocked and enters the subsequent manual approval process. This design transforms security control from relying on the uncertain perception of the model itself to relying on the deterministic guarantees of the system architecture, serving as the entry point for the entire approval loop.
[0097] When entering the manual approval process, the approval request sending module determines the sending channel for this approval request based on the approval routing mode pre-configured for the current large language model agent. This mode can be an inline-only mode, where the manual approval request is sent only within the user-agent interaction interface; a channel-only mode, where the request is sent only through external instant messaging channels (such as WeChat Work, Lark, etc.); or a more flexible automatic mode, which prioritizes sending through external instant messaging channels and automatically reverts to sending through the dialogue interface if no available external channel is detected, ensuring the approval process always reaches the approver. This design provides flexible and configurable approval channel selection capabilities for different business scenarios and deployment forms. In particular, the automatic mode, through its fallback mechanism, ensures that the approval process is always available under various circumstances, avoiding process interruptions due to channel configuration issues.
[0098] Once the approval channel is established, the approval record management module generates an approval record with a unique identifier and an initial pending status for this approval. This approval record includes information such as the invoked tools and actions, the affiliated organization, and approval eligibility information. This eligibility information limits the scope of personnel with decision-making authority, and this scope can be flexibly configured to be a set of one or more identity types selected from the initiator, agent owner, or organization members. Furthermore, to address the extreme situation where all personnel within the approval scope are unable to process the approval in a timely manner, the system administrator can intervene as an emergency channel. That is, the administrator is granted decision-making authority at the system level that is not limited by the aforementioned eligibility scope, enabling them to handle approval requests as a fallback when no one responds.
[0099] After the approval record is generated, two operations occur simultaneously: First, the original tool call request is blocked and suspended by the waiting module. This module decouples and effectively connects a synchronous tool call request with an asynchronous, unpredictable manual decision-making process from an external source. The module uses a non-busy waiting method, polling the status of the approval record at set time intervals until it reaches a final state such as approved or rejected, or until a preset timeout is reached. As an example configuration, the polling interval can be set to approximately 1.5 seconds, and the timeout can be set to approximately 120 seconds. If no decision is received within the timeout period, the module sets the record status to expire and decides to block the call. This implementation reduces the unnecessary use of system resources during the waiting period.
[0100] On the other hand, approval requests are sent out based on the routing results. If routed to an external channel (i.e., channel-only mode or automatic mode), the channel interaction card module constructs an approval card containing approve and reject buttons, embedding the unique identifier of the approval record into the button's feedback value. Subsequently, through the external instant messaging channel adaptation layer, the card is converted to a format compatible with the specific instant messaging channel and pushed to the instant messaging group of the user associated with the approver. This approach allows the user associated with the approver to handle approval requests directly in their usual instant messaging tools without interrupting the current workflow and returning to the dialogue interface with the agent, improving approval efficiency and user experience.
[0101] When communicating through external instant messaging channels, the approval request sending module is also responsible for ensuring communication security. Specifically, the IM channel adaptation layer (i.e., the external instant messaging channel adaptation layer) of the approval request sending module, through unified abstraction and encapsulation, enables the system to access multiple different external instant messaging channels in the same way, enhancing the system's scalability and compatibility. Simultaneously, its built-in credential encryption and signature verification mechanisms ensure the authenticity and security of communication with external instant messaging channels. This adaptation layer is also responsible for encrypting and storing the credentials required to access the channel and decrypting them for use upon invocation. As an exemplary security implementation, channel credentials can be stored using symmetric encryption, and the encryption and decryption keys can be derived from the system key, ensuring the security of communication credentials.
[0102] The approval order may come from at least one of the following two sources:
[0103] The first source is in-line adjudication within the dialogue. When an adjudication instruction originates from a dialogue interface interacting with a large language model agent, the instruction is received after the approving user's identity has been authenticated by the system. In other words, the approving user must complete system login and authentication within the dialogue interface; only after the system confirms their identity will the approval or rejection instruction submitted by that user be considered a valid adjudication.
[0104] The second source is the callback adjudication from external instant messaging channels (corresponding to IM channel callback adjudication in the diagram). When the adjudication instruction originates from a callback notification from an external instant messaging channel, this callback notification needs to undergo signature verification. Only after verifying the signature of the notification content and confirming its authenticity and lack of tampering will it be accepted by the system. In specific implementations, signature verification can be achieved by concatenating the callback timestamp, random string, locally stored channel key, and message body, performing a hash calculation, and comparing the calculation result with the signature carried in the callback in a constant-time manner to prevent timing attacks.
[0105] All rulings from various sources ultimately converge to the atomic ruling module. Before processing any ruling, the system first verifies whether the user submitting the ruling is within the approved record's eligibility range or is a system administrator. Rulings submitted by users who are neither within the eligibility range nor administrators are rejected. For authorized submissions, the module performs an atomic update operation: the ruling is considered valid only if the current status of the approved record is still pending, and the record status is updated to approved or rejected. In practice, this atomic update operation can be implemented through a database update with precondition checks. The number of rows affected by the update operation determines whether the ruling has actually taken effect. A 1-row effective record indicates successful adoption, while a 0-row effective record indicates that the record has been updated by other rulings and this ruling is ignored. This selective update mechanism effectively deduplicates concurrent rulings from different sources and ensures the uniqueness of the approval result.
[0106] After the ruling takes effect, the atomic ruling module also uniformly records the ruling result, ruling time, and ruling identifier in the approval log. Specifically, for rulings originating from the chat interface, the recorded ruling identifier is the authenticated user identifier, i.e., the user identifier of the approving user after successful system authentication; for rulings originating from external instant messaging channels, the recorded ruling identifier is the channel user identifier, i.e., the user identifier of the approving user in the corresponding instant messaging channel, parsed from the channel callback notification. In this way, regardless of the origin of the ruling, a unified and complete audit trail can be formed, facilitating post-event traceability and compliance review.
[0107] Finally, once the blocking waiting module detects that the approval record's status has been updated to the final state, it unblocks. Based on the updated status (approved, rejected, or expired), it returns a final instruction to the tool call dispatcher to either allow or block. If it blocks, the system generates a blocking description as observation information and injects it back into the agent's context; if it allows, the tool call is actually executed, and its execution result is injected back into the agent's context as observation information, thus completing the entire closed loop and enabling the agent to continue its next round of reasoning-action cycle.
[0108] Figure 3 The overall flow of the method in this application is illustrated. First, before the tool call request is actually executed, it is intercepted by a deterministic checkpoint located outside the reasoning-action loop (i.e., an out-of-loop PRE Hook). The deterministic checkpoint judges the request based on pre-configured sensitive action flags. If no approval is required, the request is directly allowed to be executed; if approval is required, it enters the manual approval process.
[0109] Next, an approval channel is selected based on the approval routing pattern pre-configured for the large language model agent, and an approval record with a unique identifier and pending status is generated.
[0110] Subsequently, on the one hand, the original tool call request was blocked, and the system began to poll the status of the approval record in a non-busy wait mode; on the other hand, the approval request was sent to the designated approval end, for example, in the form of an interactive card sent through an external instant messaging channel.
[0111] Approval decisions can originate from an authenticated inline chat interface or from a signed external instant messaging channel callback. All decisions are processed through a single atomic update operation: the decision is only considered valid and the approval record is updated to approved or rejected if the current status of the approval record is still pending. This ensures the uniqueness of decisions and the consistent recording of results.
[0112] Finally, after polling the approval record and finding that the status has been updated to a final state such as approved, rejected, or timed out, the block is lifted, and a decision is made based on the final state to either allow or block the blocked tool call request. The execution result or blocking description is then injected back into the context of the large language model agent as observation information to close the entire loop.
[0113] To make the purpose, technical solution and advantages of this application clearer, the technical solution of this application will be illustrated below with a specific application scenario.
[0114] A company has deployed a large language model intelligent agent system with autonomous tool invocation capabilities. The tools that this intelligent agent can invoke include several sensitive actions, such as making external payments, deleting business data, and sending external notifications.
[0115] In actual deployment, the system configures the following control parameters for this agent: the approval routing mode is set to automatic mode, the approver scope is set to the agent owner, and the agent is bound to an enterprise instant messaging channel group (such as a WeChat group or Lark group). It should be noted that the above configuration is only exemplary; the approval routing mode can also be configured as an inline-only mode or a channel-only mode according to business needs, and the approver scope can also be configured as the initiator, organization members, or a combination of multiple identities.
[0116] During a task processing session, the agent constructs a tool invocation request for outbound payment based on user instructions. Just before this request reaches the tool invocation dispatch point and is about to be executed, it is intercepted by a deterministic checkpoint located outside the inference-action loop. Based on pre-configured sensitive action flags, the deterministic checkpoint determines that the outbound payment action is a sensitive action and requires manual approval before execution.
[0117] Since the system has configured the approval routing mode for this agent to be automatic, and an available external instant messaging channel exists, the approval process is conducted through the external instant messaging channel. At this time, the system generates an approval record with a unique identifier and a pending status. The approver scope in this record is limited to the agent owner, that is, the creator or person in charge of the agent.
[0118] Meanwhile, the original tool call request was blocked. The system polled the status of the aforementioned approval records at approximately 1.5-second intervals in a non-busy wait manner, with a timeout limit of approximately 120 seconds.
[0119] Regarding the sending of approval requests, the system constructs an approval card containing approval and rejection buttons. The feedback value of each button carries a unique identifier for this approval record. After being converted into a message format compatible with the corresponding instant messaging channel, the card is pushed to the approval group bound to the agent.
[0120] The agent owner saw the approval card in the instant messaging group and clicked the approve button. This operation was sent back to the system via a callback notification from an external instant messaging platform. The system first verified the signature of the callback notification: by obtaining the timestamp and random string from the callback, concatenating them with the locally encrypted channel credentials (decrypted upon use) and the message body, a hash calculation was performed, and the calculation result was compared with the signature carried in the callback in a constant-time manner. After successful verification, the system confirmed that the callback notification originated from an authentic source and that its content had not been tampered with, and the decision instruction was received.
[0121] Meanwhile, if other qualified personnel submit a ruling instruction for the same approval request in the dialogue interface interacting with the intelligent agent, the instruction will only be received after the sender's user identity has been authenticated by the system itself.
[0122] All the aforementioned rulings from different sources are ultimately processed through a single atomic update operation. Before processing the ruling, the system first verifies whether the approving user submitting the ruling falls within the scope of approval eligibility defined in the approval record for the agent owner. Submissions that do not fall within this scope are rejected. For authorized submissions, the system performs an atomic update operation: the status is updated to approved only if the current status of the approval record is still pending. In this scenario, the approval ruling submitted by the agent owner via instant messaging arrives and takes effect first, and the status of the approval record is atomically updated from pending to approved. After the ruling takes effect, the ruling result (approval), the ruling time, and the ruling person's identifier (the agent owner's channel user identifier in that instant messaging channel) are all recorded in the approval record.
[0123] If, at the same time the agent owner clicks "Approve," another authorized user clicks "Reject" in the chat interface, the rejection decision will be deemed invalid during the atomic update operation because the current state does not meet the update conditions, and the system will ignore it. Thus, the same approval request is adopted only once with a valid decision, achieving deduplication of concurrent decisions from multiple sources.
[0124] In the next round of polling, the system reads that the approval record status has been updated to the final state of approval, and immediately unblocks it, returning a release instruction to the tool call distribution center. The blocked external payment tool call can then continue execution, actually calling the financial system interface to complete the payment operation, and injecting the successful payment result as observation information back into the context of the large language model agent. Upon receiving this observation information, the agent can then proceed with the next round of reasoning-action loop, such as reporting the payment completion status to the user.
[0125] If the agent owner does not take any action on the approved card within the approximately 120-second timeout period, the system will proactively set the approval record status to expire and instruct the system to block the tool call. At this time, the system will generate a blocking message, such as "Outward payment operation blocked due to approval timeout," and inject this blocking message back into the agent's context as observation information, so that the agent is aware that the operation was not executed.
[0126] The second aspect of this application also provides a tool invocation control system for large language model intelligent agents. This system includes a deterministic checking module, an approval record management module, a blocking waiting module, an approval request sending module, and an atomic decision module. The specific functions of each module and their collaborative working methods have been described in detail in the foregoing method embodiments and system architecture descriptions, and will not be repeated here. This system and the method embodiments are based on the same inventive concept. Through the collaborative cooperation of each module, it achieves deterministic control over intelligent agent tool invocations and a closed-loop system for cross-channel manual approval.
[0127] A third aspect of this application also provides an electronic device including a memory and a processor. The memory stores a computer program, and the processor is configured to execute the computer program to implement the tool invocation control method as described in any of the foregoing embodiments of this application.
[0128] A fourth aspect of this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the tool call control method as described in any of the foregoing embodiments of this application.
[0129] In summary, this application has the following beneficial effects:
[0130] First, it achieves deterministic and unbypassable control. This application sets the control link outside the reasoning-action loop of the large language model agent, intercepting it at the tool call distribution point. The judgment logic is independent of the model output, and the judgment is based on pre-configured sensitive action flags. This avoids the uncertainty and risk of being bypassed by prompt injection that come from writing the approval logic into the prompt words and relying on the model's self-awareness.
[0131] Second, it is human-in-the-loop and aligns with the daily usage habits of approval users. This application delegates the final decision-making power for sensitive actions to personnel with approval authority, and allows approval users to directly view approval requests and complete the approval or rejection process in their usual external instant messaging channels without interrupting the current workflow and returning to the dialogue interface for interacting with the intelligent agent.
[0132] Third, the ruling results are unique, consistent, and auditable. This application uses atomic update operations to selectively update ruling instructions from both the inline dialog interface and external instant messaging channels, ensuring that the same approval request is adopted by only one valid ruling. It also uniformly records the ruling result, ruling time, and ruling person's identifier, thus solving the risks of concurrent and duplicate rulings and the problem of fragmented audit trails.
[0133] Fourth, the approval channels and decision-making permissions are flexibly configurable and securely implemented. The approval routing mode of this application can be configured by intelligent agent as internal only, channel only, or automatic mode. The scope of approvers can be configured as initiator, owner, or organization members. Combined with signature verification and encrypted storage of credentials for channel callbacks, the solution can adapt to the multi-form deployment needs of different enterprises and ensure communication security.
[0134] The above description is only a partial embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the embodiments of this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.
Claims
1. A method for controlling tool invocation in large language model intelligent agents, characterized in that, The tool invocation control method includes: By setting a deterministic checkpoint outside the reasoning action loop of the large language model agent, the tool call request is judged before the tool call request is executed, so as to determine whether the corresponding tool action is marked as a sensitive action that requires manual approval. When it is determined that the tool action corresponding to the tool call request requires manual approval, an approval record with a unique identifier and a pending status is generated, the execution of the tool call request is blocked, and a manual approval request containing the unique identifier is sent to the designated approval end. In addition, it receives a decision instruction from the approval end and updates the status of the approval record from pending to approved or rejected by performing an atomic update operation on the approval record, so as to decide whether to continue execution or block the blocked tool call request based on the updated status.
2. The tool call control method according to claim 1, characterized in that, The process of determining whether the corresponding tool action is marked as a sensitive action requiring manual approval includes: Based on the sensitive action flags pre-configured for the actions of the invoked tool, determine whether approval is required; The judgment is independent of the reasoning output of the large language model agent, and therefore cannot be bypassed by prompts.
3. The tool call control method according to claim 1, characterized in that, The deterministic checkpoint employs a strategy of blocking upon execution failure. When an anomaly occurs at the deterministic checkpoint, the tool call request is directly intercepted.
4. The tool call control method according to claim 1, characterized in that, The generation of approval records with unique identifiers and pending status includes: The approval record records the invoked tool and action information, the organization to which it belongs, and the approval qualification information; wherein, the approval qualification information is used to limit the scope of personnel with decision-making authority, and this scope is configured as a set consisting of one or more identity types selected from the initiator, the intelligent agent owner, and the organization member.
5. The tool call control method according to claim 4, characterized in that, The method further includes: Before receiving a ruling instruction from the approval end, it is determined whether the approving user who submitted the ruling belongs to the personnel range limited by the approval qualification information in the approval record, and submissions that do not belong to the range are rejected.
6. The tool call control method according to claim 1, characterized in that, Sending a manual approval request containing the unique identifier to the designated approval terminal includes: The approval channel is selected based on the approval routing pattern pre-configured for the large language model agent; The approval routing modes include: In the channel-only mode, the manual approval request is sent only via external instant messaging channels. Inline mode only, the manual approval request is sent only within the dialogue interface that interacts with the large language model agent; In automatic mode, the manual approval request is sent first via the external instant messaging channel. If no external instant messaging channel is available, the manual approval request is sent back to the dialog interface.
7. The tool call control method according to claim 6, characterized in that, When the approval routing mode is channel-only mode or automatic mode, sending a manual approval request containing the unique identifier to the designated approval terminal specifically includes: Construct an approval card that includes approval and rejection interaction buttons, wherein the feedback value of the interaction button carries a unique identifier of the approval record; Based on the external instant messaging channel, the approval card is converted into a message format compatible with that channel, and then pushed to the instant messaging group of the approval user associated with the approval end through the channel's open interface.
8. The tool call control method according to claim 1, characterized in that, After blocking the execution of the tool invocation request, the method further includes: In a non-busy waiting mode, the status of the approval record is polled at set time intervals until the status is updated to the final state of approval or rejection, or the preset timeout period is reached; When the timeout period is reached and no ruling is received, the status of the approval record is set to expired, and the tool call request is blocked.
9. The tool call control method according to claim 1, characterized in that, The receipt of the decision instruction from the approval end includes at least one of the following two sources: When the adjudication instruction originates from the dialogue interface interacting with the large language model agent, the adjudication instruction is received after the user's identity has been authenticated by the system. When an adjudication instruction originates from a callback notification via an external instant messaging channel, the callback notification is received after verifying the authenticity of its source and confirming that it has not been tampered with, through signature verification of the notification content.
10. The tool call control method according to claim 9, characterized in that, The method further includes: Before communicating with the external instant messaging channel, the channel credentials required to access the channel are encrypted and stored. When it is necessary to call the channel's open interface, decrypt the channel credentials and use them.
11. The tool call control method according to claim 1, characterized in that, The atomic update operation on the approval record includes: When a decision instruction is received from the approval end, check the current status of the approval record; The decision instruction will be considered a valid decision and the status of the approval record will be updated to approved or rejected only if the current status is still pending. When the current status has been updated to approval or rejection, the ruling instruction shall be ignored as an invalid ruling; This ensures that the same approval record is updated only by one valid ruling, thus achieving deduplication of concurrent rulings from different sources.
12. The tool call control method according to claim 1, characterized in that, Once the status of the approval record is updated to approved or rejected, the method further includes: The approval record uniformly records the adjudication result, adjudication time, and adjudicator identifier; wherein, for adjudications from the dialog interface, the recorded adjudicator identifier is the authenticated user identifier; for adjudications from the external instant messaging channel, the recorded adjudicator identifier is the channel user identifier.
13. The tool call control method according to claim 1, characterized in that, The method further includes: When the tool call request is intercepted based on the updated state, a blocking description is generated, and the blocking description is injected back into the context of the large language model agent as observation information; When the tool call request is executed again based on the updated state, the result of the execution is injected back into the context of the large language model agent as observation information.
14. A tool invocation control system for large language model intelligent agents, characterized in that, The system includes: The deterministic check module is used to judge the tool call request before the tool call request is executed, so as to determine whether the corresponding tool action is marked as a sensitive action that requires manual approval; The approval record management module is used to generate an approval record with a unique identifier and a pending status when it is determined that the tool call request requires manual approval. The blocking and waiting module is used to block the execution of the tool call request and wait for the status update of the approval record; The approval request sending module is used to select an approval channel according to the approval routing mode pre-configured for the large language model agent, and send a manual approval request containing the unique identifier to the designated approval end. The atomic decision module is used to receive decision instructions from the approval end and update the status of the approval record from pending to one of approved or rejected by performing an atomic update operation on the approval record; The blocking waiting module determines whether to continue execution or intercept the blocked tool call request based on the updated state of the atomic decision module.
15. An electronic device comprising a memory and a processor, the memory storing a computer program, the processor being configured to, when executing the computer program, implement the method as described in any one of claims 1-13.
16. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method as described in any one of claims 1-13.
