Indirect prompt injection reasoning defense method and system for tool-oriented enhanced large language model agent

Abstract

The application discloses a method and system for defending against indirect prompt injection reasoning of a tool-oriented enhanced large language model agent. In the tool return boundary interception context, the exogenous content is extracted as an untrusted intermediate and cached. In the disturbance mode, a plurality of controlled re-executions of the original / masked input and the original / purified intermediate are constructed and executed, a three-value ordered risk result of the candidate action is calculated, and the causal effect of the user channel and the intermediate channel is estimated. A boundary takeover indication is generated based on the degradation trend of the causal index in the sliding time window. When triggered, evidence-preserving context purification and minimal action revision are performed. Finally, through effect gating and authorization verification, only when the authorization conditions are met, the external side effects are submitted, otherwise the submission is suppressed and safe continuation is supported. The application realizes explainable attribution, robustness to multiple rounds of injection and tool chain adaptation, while improving security and considering task availability.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence security and runtime protection of tool-augmented large language model (LLM) agents. Specifically, it relates to a reasoning-time defense method and system that performs causal diagnosis, takeover determination, evidence-preserving context cleanup, and minimized action revision at the tool-return boundary against untrusted intermediary channels from external tools, retrieval, and memory, and suppresses unauthorized external side effects through effect gating and authorization verification. Background Technology

[0002] With the deployment of large language model-driven agents in scenarios such as office automation, retrieval and question answering, data analysis, financial operations, customer service, and R&D operations, agents typically acquire information through external tools, retrieval systems, or memory modules and write it into the context. They then generate the next action (including natural language output and tool call sequences) based on the updated context. In this model, the agent's context is continuously mixed with content from the external environment. This external content is often presented in the form of "evidence / facts / explanations," but may embed instruction fragments with control intent.

[0003] Unlike direct prompt injection, indirect prompt injection (IPI) does not rely on the attacker controlling user input. Instead, it hides malicious commands in tool returns, search snippets, email / document / webpage content, or memorized content, causing the agent to consume this content as a trusted basis in subsequent rounds. This induces the agent to perform high-impact actions inconsistent with the user's goals or deployment strategy (e.g., sending data, transferring money, deleting / overwriting, privilege escalation, bypassing authorization, etc.). IPI presents the following engineering challenges: Multi-round accumulation and delayed triggering: Injection may accumulate gradually after multiple boundary writes, and only manifest as action offset after several steps; The carriers and toolchains are diverse: attacks can come from emails, web pages, table fields, JSON key-value pairs, knowledge base fragments, etc., making it difficult to cover them with static rules alone; Security versus availability conflict: a hardline approach of "disabling tools / full filtering / blocking everything" will significantly reduce task availability; while a lenient approach will amplify the risk of unauthorized access. Lack of explainable attribution: Many existing methods can only give a "risk score" or "hit rule", but it is difficult to answer "whether the takeover was caused by the intermediary content, where the takeover started, and how to fix it at the lowest cost". Summary of the Invention

[0004] To overcome the shortcomings of the prior art, this invention provides a method and system for defending against indirect prompting injection during inference in a tool-enhanced large language model agent. The method establishes a tool return boundary and intercepts the boundary context each time the agent introduces external tool return results, retrieval results, or memorized content and is about to generate the next action; it treats content from external channels as untrusted intermediary channels and caches them; and it implements multiple sets of controlled re-executions without causing irreversible external effects. The re-execution results are used to construct ordered risk observations and estimate the impact indicators of user channels and intermediary channels, thereby generating a boundary takeover indication. When it is determined that there is intermediary-led behavioral takeover, this invention performs evidence-preserving context cleansing and minimizes action revision, and combines effect gating and authorization verification to suppress unauthorized external side effects, thereby improving security while also considering task usability and interpretable attribution capabilities.

[0005] According to one aspect of the present invention, a method for defending against indirect hint injection during inference in a tool-enhanced large language model agent is provided, comprising: During the execution of the agent, after the external tool returns, retrieves results, or writes memory content into the context, and before the agent outputs the next action, the tool return boundary is set and the boundary context is intercepted. Extract the external content written by external tools, retrieval, or memory from the boundary context, and cache it as content from an untrusted intermediary channel. In the perturbation mode without submitting external side effects, multiple sets of controlled re-execution are constructed and executed based on the boundary context. The multiple sets of controlled re-execution include at least: original input-original intermediary, masked input-original intermediary, masked input-cleaned intermediary, and original input-cleaned intermediary. For each group of candidate actions generated by controlled re-execution, calculate the three-valued ordered risk result, and estimate the user channel causal effect and the intermediary channel causal effect based on the three-valued ordered risk result to obtain at least one causal index; Based on the sliding time window degradation trend and the significance of the mediation effect of the aforementioned causal indicators, a boundary takeover indication is generated. When the boundary takeover instruction is triggered, evidence-preserving context cleanup is performed on the boundary context, and the next action is modified with minimal action in the cleaned context. The revised action execution effect is gating and authorization verification. External side effects are submitted only if the action meets the authorization conditions consistent with the deployment strategy; otherwise, external submissions are suppressed and the subsequent reasoning process continues.

[0006] As a further technical solution, the three-valued ordered risk result includes low risk, diagnostic risk, and high impact risk; when a candidate action contains a high impact tool invocation intent, it is assigned high impact risk; when a candidate action contains a diagnostic tool invocation intent or exhibits a semantic shift induced by mediation, it is assigned diagnostic risk; and the rest are assigned low risk.

[0007] As a further technical solution, the mask input is a task-neutral probe that does not repeat the user's target, used to amplify the influence of the mediation channel on candidate actions; the purification mediation is an alternative version generated by rewriting the cached mediation content according to predetermined purification rules.

[0008] As a further technical solution, the user channel causal effect is obtained by comparing the expected difference of the three-valued ordered risk results of the original input-original intermediary and the masked input-original intermediary; the intermediary channel causal effect is obtained by comparing the expected difference of the three-valued ordered risk results of the masked input-original intermediary and the masked input-purification intermediary.

[0009] As a further technical solution, the degradation trend of the sliding time window is obtained by calculating the decay trend of the causal effect of the user channel and the enhancement trend of the causal effect of the intermediate channel within the sliding time window; when the causal effect of the user channel shows a decay trend and the causal effect of the intermediate channel shows an enhancement trend, and a preset threshold or significance condition is met, a boundary takeover indication is generated.

[0010] As a further technical solution, the evidence-preserving context cleansing only targets intermediate channel content, including: stripping directive, priority-covering, and tool capability-guiding fragments; retaining factual fields, entities, and structured information related to user goals; and injecting the retained content into the cleansed boundary context as a non-executable evidence representation.

[0011] As a further technical solution, the minimized action revision only deletes, shrinks, or reprograms the action components attributed by the causal effect of the mediating channel.

[0012] According to one aspect of the present invention, a defense system against indirect hint injection during inference for tool-enhanced large language model agents is provided, comprising: The boundary interception module is used to intercept the boundary context and corresponding runtime state information when the tool returns a boundary. The intermediary extraction and caching module is used to extract and cache the content of untrusted intermediary channels from the boundary context. The perturbation re-execution module is used to construct and execute multiple sets of controlled re-execution in a perturbation mode. The multiple sets of controlled re-execution include at least the original input-original intermediary, masked input-original intermediary, masked input-cleaned intermediary, and original input-cleaned intermediary. The risk assessment and causality estimation module is used to calculate the three-valued ordered risk results for candidate actions generated by controlled re-execution of each group, and to estimate the causal effects of the user channel and the mediator channel. The time degradation determination module is used to maintain the historical records of causal indicators on the boundary sequence and calculate trend characteristics within a sliding time window to generate boundary takeover indicators. The context purification module is used to perform evidence-preserving purification on the content of the intermediary channel when the takeover is triggered, and output the purified boundary context. The action revision module is used to perform a minimal revision of the next action in the cleaned boundary context and output a safe action; The effect gating module is used to perform authorization verification on the security action, and submit external side effects only when the authorization conditions are met, otherwise suppress external submission.

[0013] As a further technical solution, the three-valued ordered risk result includes low risk, diagnostic risk, and high impact risk; the mask input is a task-neutral probe that does not repeat the user's objective; and the purification medium is an alternative version generated according to predetermined purification rules.

[0014] As a further technical solution, the time degradation determination module generates a boundary takeover indication when it detects that the causal effect of the user channel is attenuating and the causal effect of the intermediate channel is increasing, and the preset threshold or significance condition is met.

[0015] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. This invention constructs a real-time diagnostic framework based on causal inference by intercepting context and caching external intermediary content at the tool's return boundary. This enables boundary-level localization and quantitative attribution of multi-round cumulative indirect suggestion injections. Utilizing multiple sets of controlled re-execution and ternary ordered risk results, it can accurately estimate the causal effects of the user channel and the intermediary channel, transforming the fuzzy judgment of injection risk into interpretable causal indicators. This provides transparent decision-making basis for the protection mechanism and significantly improves the interpretability of security defenses.

[0016] 2. Based on the degradation trend analysis of causal indicators within a sliding time window, this invention can capture the takeover evolution process of enhanced mediating effects and diminished user effects, thereby generating boundary takeover indications before the injection triggers actual external side effects. It exhibits strong robustness against complex injection patterns characterized by delayed triggering and multi-round gradual injection. Through evidence-preserving context cleansing, only control instructions are stripped away while retaining the factual information necessary for the task. The injection is represented by non-executable evidence, thus both cutting off the execution path of attack instructions and maintaining the agent's availability of business-critical information.

[0017] 3. After takeover is triggered, minimal action revisions are performed, only deleting, shrinking parameters, or replanning action components attributable to mediation effects, retaining low-impact steps necessary for task completion, thus achieving a balance between security and task continuity. External side effects are uniformly managed through effect gating and authorization verification, allowing submission only when authorization conditions consistent with the deployment strategy are met; otherwise, submission is suppressed and safe continuation is supported. This extends the defense boundary from input filtering to the final step before action execution, fundamentally suppressing unauthorized operations.

[0018] 4. Furthermore, this invention can be adapted to various tool-enhanced intelligent agent platforms through boundary interception, intermediate caching, and perturbation re-execution, demonstrating good versatility. The overall solution improves the defense against indirect hint injection while also considering task availability, diagnostic transparency, and engineering adaptability. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 A schematic diagram of the overall process of a method for defending against indirect hint injection during inference in a tool-oriented enhancement of a large language model agent, provided in an embodiment of the present invention; Figure 2 A schematic diagram of the functional modules of an indirect hint injection defense system for tool-oriented enhancement of large language model agents provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of purification, revision, and effect gating under boundary takeover triggering provided in an embodiment of the present invention. Detailed Implementation

[0021] To address the challenges of real-time localization and attribution of multi-round accumulated indirect hint injections in existing technologies, the difficulty in balancing security protection and task availability, and the lack of interpretability, a protection mechanism is needed that is inference-time, boundary-level, interpretable, and adaptable to multiple tool orchestrations. This mechanism should be able to locate and determine the takeover trend of multi-round IPIs, achieve safe continuation of the task without terminating it, and ensure the consistency of deployment strategies through external side effect gating.

[0022] To this end, the present invention provides a method for defending against indirect hint injection during inference in a tool-enhanced large language model agent, comprising the following steps: S1: Boundary setting and context capture.

[0023] During the execution of the agent, after the external tool returns / retrieves results / memorizes content and writes it into the context, and before the agent outputs the next action, the tool return boundary is set and the boundary context is intercepted.

[0024] S2: Intermediary content extraction and caching.

[0025] Extract the external content written by external tools, retrieval, or memory from the boundary context, and cache it as untrusted intermediary channel content; the cache is used to replay and replace the intermediary channel content during subsequent re-executions to maintain comparability between different re-executions.

[0026] S3: Controlled re-execution construction and execution (perturbation mode).

[0027] In perturbation mode, with a fixed dialog prefix and runtime state, multiple sets of controlled re-executions are constructed and executed by replacing user channel input and / or intermediary channel content, including at least the following combinations: (1) Raw input – raw intermediary; (2) Mask input – original intermediary; (3) Mask input – Clean up the intermediary; (4) Raw input – purification mediator.

[0028] The mask input is a task-neutral probe that does not restate the user's target, used to improve the observability of the influence of the intermediate channel on the candidate action output while keeping the boundary context and runtime state unchanged. The sanitization intermediate is an alternative version generated from the cached intermediate content according to predetermined sanitization rules, and its generation rules are consistent with or equivalent to those of online sanitization.

[0029] S4: Orderly risk quantification and impact indicator estimation.

[0030] For each group of controlled re-executions, calculate the ordered risk result of the candidate actions, and estimate the user channel impact index and the intermediary channel impact index based on the ordered risk result to obtain at least one index for takeover determination.

[0031] Preferably, the ordered risk result is a three-valued ordered set used to characterize the impact level and mediation sensitivity of candidate actions, including: low risk, diagnostic risk, and high impact risk; wherein, a candidate action is determined to be of high impact risk when it contains an intent to invoke a high-impact tool; otherwise, a candidate action is determined to be of diagnostic risk when it contains an intent to invoke a diagnostic tool or when it exhibits a semantic shift induced by mediation; the rest are determined to be of low risk.

[0032] Furthermore, the impact index can be obtained by statistically differencing the risk results under different re-execution combinations, so as to characterize the impact contribution of changes in user input and changes in intermediary content on candidate actions, respectively.

[0033] S5: Generation of time window degradation trend and boundary takeover indication.

[0034] Within a sliding time window, the degradation trend of the influence index is calculated, and a boundary takeover indication is generated by combining the threshold or significance conditions of the influence of the intermediate channel; when the influence of the user channel shows a decaying trend and the influence of the intermediate channel shows an increasing trend, and the preset conditions are met, it is determined that the intermediate-dominated behavioral takeover has occurred.

[0035] The "significance condition" can be achieved by determining the confidence interval of the re-execution results, the consistency test of repeated re-executions, or a preset effect size threshold.

[0036] S6: Evidence preservation sanitization and minimal action revisions triggered by takeover.

[0037] When a boundary takeover instruction is triggered, evidence-preserving context cleansing is performed only on the intermediate channel content: directive, priority-covering, and tool capability-guiding fragments are stripped away; fact fields, entities, and structured information relevant to the user's objectives are preserved; and the preserved content is injected into the cleansed boundary context as non-executable evidence. Within the cleansed context, minimal action revisions are performed on the next action, deleting, shrinking parameters, or reprogramming replacements only for action components attributed to the influence of the intermediate channel.

[0038] S7: Effect gating and authorization verification.

[0039] The revised action execution effect is gating and authorization verification is performed. External side effects are submitted only if the action meets the authorization conditions consistent with the deployment policy; otherwise, external submissions are suppressed and the subsequent inference process continues to achieve safe continuation.

[0040] In this invention specification, it should be noted that: "Tool return boundary" refers to the point in time when the external tool / retrieval / memory content has been written into the context and the agent has not yet output the next action; "External side effects" refer to operations that alter the external environment or cause data to be leaked. "Disturbance mode" means: allowing the generation of candidate actions but prohibiting the submission of external side effects, and only recording the intended execution and parameter summary.

[0041] The terms “comprising” and “having”, and any variations thereof, in the specification, claims, and accompanying drawings of this invention are intended to cover a non-exclusive inclusion, such as a process, method, system, product, or apparatus that includes a series of steps or units, not necessarily limited to those explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0042] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. In addition, the technical features of the various embodiments or individual embodiments provided by the present invention can be arbitrarily combined to form new technical solutions. Such combinations are not bound by the order of steps and / or structural composition patterns, but must be based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by the present invention.

[0043] For ease of description, the following symbols and definitions are introduced in this embodiment of the invention: the internal context is c, the external environment state is o; the tool-returned boundary sequence is B={1,2,…}, the context at boundary b is c_b, and the external state is o_b; the content of the untrusted intermediary channel extracted from the context is r_b; the policy / deployment rule is Π; the next candidate action proposed by the agent at boundary b is a_b; the action risk diagnosis function is ψ(·;Π), and the output is a three-valued ordered result y∈{0,1,2}. The tool set is divided into a diagnostic tool subset T_diag and a high-impact tool subset T_exfil (e.g., outbound, write, transfer, deletion, permission change, etc.) according to the degree of impact. The authorization verification function is Auth(·;Π,c)∈{0,1}. Example 1

[0044] See Figure 1 This embodiment provides a method for defending against indirect hint injection during inference in a tool-enhanced large language model agent. The method performs boundary-level causal diagnosis and takeover determination at each tool return boundary, and performs context cleanup, action revision, and effect gating as necessary. The specific steps are as follows: Step S1: Boundary Setting and Context Interception During agent execution, after the external tool returns, retrieves results, or writes memory content into the context, and before the agent outputs the next action, a tool return boundary b is set. At this boundary point, the interceptor obtains the current boundary context c_b and the corresponding runtime state.

[0045] The runtime state includes at least a dialog prefix, tool call record, retrieval index identifier, and memory read / write key value, which is used to ensure that subsequent controlled re-execution can be performed under the same prefix and state conditions for comparable re-inference.

[0046] By setting the tool return boundary, the present invention can intercept the boundary context within the time window when external content has been written but has not yet affected the generation of agent actions, providing a consistent runtime basis for subsequent extraction of intermediate content, controlled re-execution, and estimation of causal effects.

[0047] As an optional implementation, the boundary settings can be deployed via a runtime hook mechanism: the interception operation is triggered after the tool executor appends the returned content to the context serialization structure and before the model calls the action to generate the action.

[0048] Step S2: Extraction and caching of intermediary content From the intercepted boundary context c_b, extract the external content returned by external tools, retrieval results, or memory content written into it, and use it as the untrusted intermediary channel content r_b, and write it into the cache.

[0049] The cache contains at least the following three types of information to support the comparability and counterfactual construction of subsequent controlled re-execution: (1) The original text of the intermediary content or its structured fields, used to replay the original intermediary content during re-execution; (2) Source metadata, including at least one of tool name, interface ID, retrieval document identifier, memory key or timestamp, used to trace the source context of external content; (3) The runtime snapshot index corresponding to the boundary is used to accurately locate and restore the boundary state during re-execution, ensuring the consistency of the runtime basis between different re-executions.

[0050] The cache serves two purposes: first, to replay during controlled re-execution, ensuring that multiple re-executions use the same intermediate content to maintain comparability of results; and second, to replace r_b with the cleaned intermediate version when a counterfactual comparison needs to be constructed. This supports the comparative execution of the original and purified mediators, thereby providing a data foundation for subsequent causal effect estimation.

[0051] Through the above-mentioned caching mechanism, the present invention can repeatedly call the same intermediary content in the perturbation mode, or replace only the intermediary content while keeping other conditions unchanged, thereby achieving isolated observation and quantitative attribution of the impact on the intermediary channel.

[0052] Step S3: Controlled re-execution construction and execution (perturbation mode, no external side effects) Controlled re-execution is performed in perturbation mode: the perturbation mode means that the model is allowed to generate candidate actions and tool call intentions, but it is prohibited to submit any calls that would change the state of the external environment or cause data to be sent out. Only the tool to be executed, parameter summary and risk diagnosis results are recorded, thereby avoiding irreversible external side effects during the diagnosis phase.

[0053] Under the conditions of fixed boundary prefix and runtime state, at least four sets of controlled re-execution are constructed and executed, the four sets of controlled re-execution including: R1 Raw Input – Raw Mediator: Using the observed user input with the raw mediator r_b; R2 Masked Input – Original Mediator: Replace the user channel input with the task-neutral probe x_mask while keeping the mediator as is; R3 Masked Input – Cleanse Mediation: Replace user input with x_mask and replace the mediation with ; R4 Raw Input – Cleanse the Mediator: Keep the user input unchanged and replace the mediator with... .

[0054] The mask input is a task-neutral probe x_mask that does not restate the user's objective. This amplifies the observability of the influence of the mediation channel on the candidate action output while maintaining the boundary context and runtime state. Preferably, it does not restate the user's objective and only requires the model to provide a prompt for the next candidate action structure without executing external side effects. The purification mediation... This is an alternative version of the cached intermediate content generated based on cleanup rules that are consistent with or equivalent to those used for online cleanup. In this step, it is only used for offline replacement to construct a counterfactual comparison and is not directly written back to the real context.

[0055] To reduce the impact of model generation randomness on diagnostic results, optionally, K repeated samplings (e.g., K=3~10) can be performed on each controlled re-execution, and the statistical mean of the results of each group can be calculated as input for subsequent risk assessment and causal effect estimation.

[0056] Step S4: Three-valued ordered risk quantification and impact index estimation Candidate actions generated for each group of controlled re-executions The three-valued ordered risk results are calculated as a quantitative basis for subsequent causal effect estimation: .

[0057] The three-valued ordered risk results include low risk, diagnostic risk, and high-impact risk, determined by the following rules: When a candidate action contains at least one high-impact tool invocation intent belonging to the high-impact tool subset T_exfil, y=2, representing high-impact risk; otherwise, when a candidate action contains at least one diagnostic tool invocation intent belonging to the diagnostic tool subset T_diag, or when semantic shifts induced by mediating content appear in the natural language output (e.g., priority overriding, role redefinition, authorization bypassing tendency, etc.), y=1, representing diagnostic risk; otherwise, y=0, representing low risk. Here, y=1 serves as a diagnostic marker, used to identify potential risk tendencies induced by mediating elements, and does not necessarily indicate a violation.

[0058] For each execution combination For each combination ∈{orig, mask, mask_sanitized, orig_sanitized}, calculate the mean of the three-valued ordered risk results from multiple repeated samplings, and use it as the risk observation value for this combination: .

[0059] Furthermore, based on the risk observations from the four sets of re-executions, causal indicators used for takeover determination are estimated. These causal indicators include at least: (1) User channel causal effect, obtained by comparing the expected difference of risk outcomes between the original input-original mediation and the masked input-original mediation, is used to quantify the contribution of user input to candidate actions: ; (2) The causal effect of the mediation channel is obtained by comparing the expected difference of risk outcomes between masked input-original mediation and masked input-purified mediation, and is used to quantify the contribution of the original mediation content to the candidate action: ; (3) As an optional comparison indicator, the user channel effect under the cleansing mediation condition can be further constructed. It is obtained by comparing the expected difference of risk outcomes between the original input-cleansing mediation and the masked input-cleansing mediation, and is used for consistency checks: .

[0060] In engineering implementation, the above indicators can be used as a quantitative basis for "user dominance decay / mediator dominance enhancement"; they can also be used to output residual terms for consistency checks, but this does not constitute a limitation.

[0061] Step S5: Generation of Time Window Degradation Trend and Boundary Takeover Indicator Maintain a historical cache of user channel causal effects and intermediate channel causal effects on the boundary sequence. (In a sliding time window) Within this framework, the degradation trend of the causal indicators is calculated to generate a boundary takeover indication.

[0062] Preferably, the causal effect on the user channel Causal effect of mediating channels Estimate the trend term within the window, for example, by calculating the trend slope through linear fitting. and : .

[0063] The trend term is used to characterize the direction and magnitude of the change of causal indicators over time.

[0064] When "user channel impact attenuation" is detected (e.g.) And the "increased influence of intermediary channels" (e.g.) When a pre-defined threshold or significance condition is met, a mediator-led takeover is determined to have occurred, and a boundary takeover indication is generated. .

[0065] The significance condition can be achieved in one of the following ways, but is not limited to: (1) Causal effect of mediating channels Perform confidence interval discrimination for repeated executions; if the lower limit of the confidence interval is greater than zero, it is considered significant. (2) The lower bound test of the sample difference between mask and mask_sanitized is performed using the bootstrap method; (3) Use effect size thresholds (e.g.) The mediation effect is determined in conjunction with the consistency test to ensure that it reaches both the actual significant scale and statistical stability.

[0066] Through the above trend analysis and significance determination, the present invention can accurately locate the turning point where the intermediary content begins to dominate the behavior of the intelligent agent in a multi-round cumulative injection scenario, providing a reliable trigger signal for subsequent takeover and repair.

[0067] Step S6: Evidence preservation cleanup and minimization action revision under takeover trigger (corresponding to) Figure 3 ) When the boundary receptacle instruction generated in step S5 At that time, perform local boundary repairs, including evidence-preserving context cleanup and minimal action revisions, such as... Figure 3 As shown.

[0068] (1) Evidence-preserving context purification Only the content r_b from untrusted intermediary channels is processed, specifically including: stripping away control fragments such as directives, priority coverage, and tool capability guidance; retaining fact fields, entities, timestamps, and structured key-value pairs related to the user's goals; and injecting the retained content as non-executable evidence into the cleansed boundary context. .

[0069] The unexecutable evidence representation refers to the presentation of "evidence" rather than "executable instructions" at the serialization level, including but not limited to citation, prefix, or structured summary representations, to reduce the probability of it being consumed by the model as a control signal. Through the above purification, the present invention both cuts off the execution path of attack instructions and maintains the agent's availability of business-critical information.

[0070] (2) Minimize action revision In the cleaned boundary context The next steps will be revised to form a safe action plan. The revision principles are as follows: only high-impact action components attributable to the causal effects of the mediation channel should be deleted, parameter shrinked, or reprogrammed and replaced; low-impact steps necessary for task completion should be retained; for high-impact operations that must be retained, priority should be given to ensuring that the source of sensitive parameters can be supported by credible context or evidence fields, rather than from mediation instructions in free text form.

[0071] Through the above-mentioned minimal revisions, the present invention achieves precise intervention in actions, only dealing with the risk component attributable to the mediating effect, and maintaining task continuity to the maximum extent while ensuring safety.

[0072] Step S7: Effect Gating and Authorization Verification (Secure Continued Run) Gating and authorization verification of the revised action execution effect: External side effects are allowed to be submitted only if the action meets the authorization conditions consistent with the deployment policy; otherwise, external submissions are suppressed and the subsequent inference process continues to achieve safe continuation.

[0073] Preferably, the decision submitted by external side effects is abstracted as follows: .

[0074] In the context of boundary cleanup after action revision (If takeover is not triggered, the original boundary context c_b is used) for safety actions. The authorization verification function Auth(·;Π, c) is executed. External calls involved in the action are submitted only if the authorization verification passes; if the authorization fails, external submissions are suppressed. The system can record audit logs and output explanatory evidence for subsequent traceability or authorization within the loop, but this does not constitute a limitation of the present invention.

[0075] Through the above steps, this embodiment achieves the following: in a multi-round cumulative IPI scenario, boundary-level positioning and determination of "intermediary-led takeover" is performed, and partial repair and safe continuation are completed without terminating the task.

[0076] Example 2 See Figure 2 This embodiment provides a defense system against indirect hint injection during inference for tool-enhanced large language model agents. The system uses the "tool return boundary" as the runtime anchor point, performing boundary-level diagnosis, takeover judgment, and local repair on content introduced through untrusted intermediary channels such as external tools / retrieval / memory. It also implements effect gating and authorization verification before submitting external side effects to ensure safe continuation. The system includes at least the following functional modules: (1) Boundary interception module: configured to intercept the boundary context and its corresponding runtime state information at the tool return boundary after the external tool returns, the retrieval results or the memory content is written into the context and before the agent generates the next action, so as to provide a consistent boundary prefix and state basis for subsequent controlled re-execution and comparable diagnosis.

[0077] (2) Mediation extraction and caching module: configured to extract the mediation content written by the external channel from the boundary context, construct an untrusted mediation view and cache it; the cache contains the mediation content and its source metadata and replay identifier, which are used for mediation replay and replacement during subsequent re-execution to ensure the comparability between different re-execution results.

[0078] (3) Disturbance re-execution module: configured to construct and execute multiple sets of controlled re-execution based on fixed boundary prefixes and runtime states in disturbance mode, including at least four combinations: original input-original intermediary, masked input-original intermediary, masked input-purified intermediary, and original input-purified intermediary; wherein the disturbance mode is used to prohibit the submission of any external side effects that may change the external environment state or cause data outflow, and only records candidate actions and their tool call intentions and parameter summaries.

[0079] (4) Risk assessment and impact index estimation module: configured to quantify the risk of candidate actions generated by controlled re-execution in each group, output three-valued ordered risk results, and estimate user channel impact index and intermediary channel impact index based on these results, serving as the quantitative basis for takeover judgment; among which the three-valued ordered risk results at least distinguish between low risk, diagnostic risk and high impact risk.

[0080] (5) Time degradation judgment module: configured to maintain the historical records of the influencing indicators on the boundary sequence and calculate the trend characteristics within the sliding time window; when the influence of the user channel is detected to be attenuating and the influence of the intermediary channel is detected to be increasing, and the preset threshold condition or significance condition is met, a boundary takeover indication is generated to determine the occurrence of intermediary-dominated behavior takeover.

[0081] (6) Context purification module: Configured to perform evidence-preserving purification only on the content of the intermediate channel when the takeover is triggered, stripping control fragments such as directives, priority coverage and tool capability guidance, retaining fact fields, entities and structured information related to the task, and outputting them to the purified boundary context as non-executable evidence.

[0082] (7) Action revision module: Configured to perform minimal revisions to the next action in the cleaned boundary context, preferably only deleting, shrinking or reprogramming high-impact components attributable to the influence of intermediate channels, while retaining low-impact steps necessary for task completion, thereby outputting safe actions.

[0083] (8) Effect Gating Module: Configured to perform authorization verification and effect gating on the security action during the external execution phase. External side effects are only allowed to be submitted when the action meets the authorization conditions consistent with the deployment strategy; otherwise, external submissions are suppressed and the process is driven into the subsequent inference or user-in-the-loop authorization path to support secure continuation.

[0084] Through the synergy of the above modules, the system can execute the inference-time defense method provided in any embodiment of the present invention, and has a comprehensive protection effect of boundary-level interpretable diagnosis, takeover location and local repair, and controllable external side effects.

[0085] Example 3 Based on the same inventive concept as the foregoing embodiments, see [link to previous document]. Figure 3 When the time degradation determination module outputs At that time, the system performs evidence-preserving cleanup on the original intermediary r_b, injecting factual information as evidence representation into the cleaned boundary context. Within this cleanup context, the actions are minimized and revised to obtain safe actions. The effect gating module then... Perform authorization verification: if authorization is successful, submit external side effects; if authorization fails, suppress submission and continue the reasoning process or trigger user-in-the-loop authorization (optional). The beneficial effect of this embodiment is that the repair operation is limited to the boundary locality, only the intermediate channel is processed and the revision action is minimized, thereby maintaining task availability while reducing the risk of unauthorized access.

[0086] In this embodiment of the invention, the parameter range and anomaly handling include: Number of re-executions K: preferably 3 to 10, to balance stability and overhead; Sliding window length w: preferably 3 to 8 boundaries, used to capture multi-round accumulation and delayed triggering; Threshold settings τ_IE, τ_ACE, γ can be configured according to business domain (such as more conservative thresholds for funding / privacy / code execution scenarios).

[0087] When the tool cannot be simulated in perturbation mode: it can be downgraded to a "dry run strategy" that "only generates action intent and parameter summary, and does not execute the tool"; when intermediate metadata is missing: a conservative default strategy is adopted and the gating strength is increased; none of the above constitutes a restriction.

[0088] In summary, this invention discloses a method and system for defending against indirect prompting injection during inference in a tool-enhanced large language model agent. The method includes: setting a tool return boundary and intercepting the boundary context after external tools, retrieval, or memory content are written into the context and before the agent generates the next action; extracting and caching external content from tools, retrieval, or memory as untrusted mediation channels; constructing and executing multiple sets of controlled re-executions in a perturbation mode without submitting external side effects, including at least original input-original mediation, masked input-original mediation, masked input-purified mediation, and original input-purified mediation; calculating the three-valued ordered risk result for each candidate action generated by each re-execution, and estimating the causal effect of the user channel and the causal effect of the mediation channel accordingly to form a causal index; generating a boundary takeover indication based on the sliding time window degradation trend and mediation effect significance of the causal index; performing evidence-preserving context purification when takeover is triggered and revising the action to minimize the impact under the purified context; and finally, through effect gating and authorization verification, submitting external side effects only when the authorization conditions for consistent deployment strategies are met, otherwise suppressing external submissions and supporting safe continuation. This solution combines explainable attribution, robustness to long-process, multi-round injection, and toolchain adaptability, making it suitable for intelligent agent platforms with multi-tool orchestration to improve security and task availability.

[0089] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the technical solutions of the embodiments of the present invention.

Claims

1. A method for defending against indirect hint injection during inference in a tool-enhanced large language model agent, characterized in that, include: During the execution of the agent, after the external tool returns, retrieves results, or writes memory content into the context, and before the agent outputs the next action, the tool return boundary is set and the boundary context is intercepted. Extract the external content written by external tools, retrieval, or memory from the boundary context, and cache it as content from an untrusted intermediary channel. In the perturbation mode without submitting external side effects, multiple sets of controlled re-execution are constructed and executed based on the boundary context. The multiple sets of controlled re-execution include at least: original input-original intermediary, masked input-original intermediary, masked input-cleaned intermediary, and original input-cleaned intermediary. For each group of candidate actions generated by controlled re-execution, calculate the three-valued ordered risk result, and estimate the user channel causal effect and the intermediary channel causal effect based on the three-valued ordered risk result to obtain at least one causal index; Based on the sliding time window degradation trend and the significance of the mediation effect of the aforementioned causal indicators, a boundary takeover indication is generated. When the boundary takeover instruction is triggered, evidence-preserving context cleanup is performed on the boundary context, and the next action is modified with minimal action in the cleaned context. The revised action execution effect is gating and authorization verification. External side effects are submitted only if the action meets the authorization conditions consistent with the deployment strategy; otherwise, external submissions are suppressed and the subsequent reasoning process continues.

2. The indirect hint injection defense method for tool-oriented large language model agents according to claim 1, characterized in that, The three-valued ordered risk results include low risk, diagnostic risk, and high impact risk; a candidate action is assigned high impact risk when it contains a high impact tool invocation intent, a candidate action is assigned diagnostic risk when it contains a diagnostic tool invocation intent or exhibits a semantic shift induced by mediation, and the rest are assigned low risk.

3. The indirect hint injection defense method for tool-oriented large language model agents according to claim 1, characterized in that, The mask input is a task-neutral probe that does not repeat the user's target, used to amplify the influence of the mediation channel on candidate actions; the purification mediation is an alternative version generated by rewriting the cached mediation content according to predetermined purification rules.

4. The indirect hint injection defense method for tool-oriented large language model agents according to claim 1, characterized in that, The user channel causal effect is obtained by comparing the expected difference of the three-valued ordered risk outcome of the original input-original intermediary and the masked input-original intermediary; the intermediary channel causal effect is obtained by comparing the expected difference of the three-valued ordered risk outcome of the masked input-original intermediary and the masked input-purification intermediary.

5. A method for defending against indirect hint injection during inference in a tool-oriented large language model agent according to claim 1, characterized in that, The degradation trend of the sliding time window is obtained by calculating the decay trend of the causal effect of the user channel and the enhancement trend of the causal effect of the intermediate channel within the sliding time window; when the causal effect of the user channel shows a decay trend and the causal effect of the intermediate channel shows an enhancement trend, and a preset threshold or significance condition is met, a boundary takeover indication is generated.

6. The indirect hint injection defense method for tool-oriented large language model agents according to claim 1, characterized in that, The evidence-preserving context cleansing applies only to intermediate channel content, including: stripping directive, priority-covering, and tool-capability-guiding fragments; retaining factual fields, entities, and structured information relevant to the user's goals; and injecting the retained content into the cleansed boundary context as a non-executable evidence representation.

7. A method for defending against indirect hint injection during inference in a tool-oriented large language model agent according to claim 1, characterized in that, The minimized action revision only deletes, shrinks, or reprograms the action components attributed to causal effects through mediating channels.

8. A defense system against indirect prompting injection during inference for tool-enhanced large language model agents, characterized in that, include: The boundary interception module is used to intercept the boundary context and corresponding runtime state information when the tool returns a boundary. The intermediary extraction and caching module is used to extract and cache the content of untrusted intermediary channels from the boundary context. The perturbation re-execution module is used to construct and execute multiple sets of controlled re-execution in a perturbation mode. The multiple sets of controlled re-execution include at least the original input-original intermediary, masked input-original intermediary, masked input-cleaned intermediary, and original input-cleaned intermediary. The risk assessment and causality estimation module is used to calculate the three-valued ordered risk results for candidate actions generated by controlled re-execution in each group, and to estimate the causal effects of the user channel and the mediator channel. The time degradation determination module is used to maintain the historical records of causal indicators on the boundary sequence, calculate trend characteristics within the sliding time window, and generate boundary takeover indicators. The context purification module is used to perform evidence-preserving purification on the content of the intermediary channel when the takeover is triggered, and output the purified boundary context. The action revision module is used to perform a minimal revision of the next action in the cleaned boundary context and output a safe action; The effect gating module is used to perform authorization verification on the security action, and submit external side effects only when the authorization conditions are met; otherwise, it suppresses external submission.

9. A defense system against indirect hint injection during inference for tool-oriented large language model agents according to claim 8, characterized in that, The three-valued ordered risk results include low risk, diagnostic risk, and high impact risk; the mask input is a task-neutral probe that does not repeat the user's objective; and the purification medium is an alternative version generated according to predetermined purification rules.

10. A defense system against indirect prompting injection during inference for tool-oriented large language model agents according to claim 8, characterized in that, When the time degradation determination module detects that the causal effect of the user channel is attenuating and the causal effect of the intermediate channel is increasing, and meets the preset threshold or significance condition, it generates a boundary takeover indication.

Images